Passing entrophy to virtual machines

Recently, when we were working on testing new images with Magnum, I found that the newest Fedora Atomic 29 images were taking a long time to boot up. A closer look using nova console-log revealed that they were getting stuck at boot with the following error.
[   12.220574] audit: type=1130 audit(1555723526.895:78): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-machine-id-commit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.248050] audit: type=1130 audit(1555723526.906:79): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journal-catalog-update comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1061.103725] random: crng init done
[ 1061.108094] random: 7 urandom warning(s) missed due to ratelimiting
[ 1063.306231] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
The number between the [] shows the number of seconds since boot, as you can see the VM was stuck for >1000 secs waiting for crng init.
It turns out that in some newer images, boot will block waiting for sufficient entropy. Entropy, or randomness, is used in operating systems for important things like random number generation. Traditionally machines generate their entropy by looking at inputs that are random, e.g. disk writes, mouse movement. (Fun experiment: On a Linux machine do a cat /dev/random, wait for the output to stop and move your mouse.)
A fresh VM has very little avenues to collect entropy, so unfortunately if doesn’t have enough entropy it may block. Luckily the smart people at QEMU has a solution called VirtIO RNG, which involves passing entropy from the host hypervisor to the VM. This allows the VM to seed it’s entropy pool and happily continue booting.
Openstack config docs points out that you need to set this in two places, both at flavor and at image. The flavor setting controls whether an image booted with this flavor is allowed to drain the host’s entropy, and what rates they are allowed to drain. Finding the correct rate is a bit of trial and error, as you want a high enough rate so that the VM will not block, but low enough that a malicious VM will not be able to totally drain a hypervisor’s entropy.
With a bit of testing, and advice from our fellow OpenStack Operators at Catalyst NZ, we have found the following values to work for us:
  • hw_rng:allowed='True'
  • hw_rng:rate_bytes='24'
  • hw_rng:rate_period='5000'
NOTE: as our friends from Catalyst points out, rate_period is specified in milliseconds and not seconds like what some documentation states.

Same rate, different periods?

One of the question that we had to answer was, what is the effect of the period setting? For example, if we know that we need 5 bytes/second of information, is it better to set
  1. 1 byte in 1 second, or
  2. 5 bytes in 5 seconds?
Both of this settings translate to the same effective rate, but their performance can be very different.
In this note, it was suggested that a smaller period is better because it means your process will block for a shorter time. However, a longer period can have some advantages in a cloud environment.
  1. in case of a benign VM, we want it to be able to burst as much as possible. If it needs 5 bytes it can get it in the first sec rather than wait till the 5th sec
  2. in case of a malicious VM, being able to block it for 5 secs means other VMs will be able to have a better chance to consume entropy
Hence, we have set a relatively large period on our environment.

Comments

Post a Comment

Popular posts from this blog

Getting capacity and usage information out of Openstack Placement

Image
One of the problems we have in Nectar is reporting on capacity and usage of nova. We have ~1000 compute nodes and these come and go. The Openstack Placement service seems like a good place to extract capacity and usage information from nova so I thought I'd see what it could do. First problem is there is no placement client. I found the osc-placment project but this isn't that useful as it is just an openstack client plugin and not a python library like the other clients. I've written a custom client before for an internal project so I created a very simple client that can get resource provider usage and capacity. In placement a resource provider maps directly to a compute node in nova. Now we have the means to query this information I wrote a custom Ceilometer poller to collect this information periodically. Once this was collecting data I needed to configure Ceilometer and Gnocchi to ingest the notifications that my new poller was generating. In Ceilometer I ne...
Read more

Understanding telemetry services in Nectar cloud

Intro Openstack telemetry service is to reliably collect data on the utilization of the physical and virtual resources comprising deployed clouds, persist these data for subsequent retrieval and analysis, and trigger actions when defined criteria are met.  Ceilometer project was the only piece of telemetry services in openstack, but since it was dumped into too many functions, it got split into several subprojects: Aodh (alarm evaluation functionality), Panko (events storage) and Ceilometer (data collection - more about ceilometer  history  from ex-PTL). The original storage and API function of ceilometer had been deprecated and moved to the other projects. The latest telemetry architecture could refer to openstack  system architecture . In Nectar cloud, we are using the Ceilometer for the metric data collection, Aodh for the alarm services, Gnocchi for the metric API services, and Influxdb as the time series database backend of Gnocchi. Influxdb backend drive...
Read more

Kubernetes now available on Nectar!

We’ve just deploy OpenStack Magnum (Container Infrastructure as a Service) on Nectar cloud. This allows a user to spin up a container cluster (kubernetes or docker swarm) on Nectar. We are in the process of coming out with official documentation, but in the meantime if you would like to test drive it, here are the steps to do so. First of all, you need the following Quotas for Floating IP Network Subnet If you have requested for floating IPs for your project, you will be fine. If not,  request for floating ip quota . Creating a Cluster You can create a cluster using either the Dashboard or CLI tools. Using Dashboard Log on to the  dashboard Navigate to  Container Infra . Click on  Clusters , then  Create Cluster . Give your cluster a name. Choose a cluster template. We have pre-defined global templates (in format  kubernetes-{az} ) to help you get started easier. Choose the template that you want your cluster to be in. ...
Read more