Do you know how Nova knows where to spawn a virtual machine? With the Nova scheduler service. This service allows to define a set of filters that should match with the host resources in order to define and spawn the virtual machines. Since Newton (a log time ago…) a new resource tracker was introduced: the Placement API. From this website we can read that “it is a separate REST API stack and data model used to track resource provider inventories and usages, along with different classes of resources“.
If you are not familiar with the Placement API, I really recommend you to, before reading the documentation (that is necessary), watch this fantastic video presented by Jay Pipes explaining how it works, the data model and how interacts with OpenStack. In a nutshell, in the Placement API you have:
- Resource classes, that are the countable resources that will be requires by the specific virtual machines (RAM, disk space, CPUs, network bandwidth, etc).
- Traits, that describe the provider of a resource class.
- Resource providers, that gives to Nova the requested providers. The resource providers could be nested; for example, a compute node is a parent resource provider and the SR-IOV agent is a nested resource provider of network bandwidth for these VFs allocated in a specific NIC (that will have associated an amount of bandwidth).
- Inventories, grouped by resources and providers (that means, the amount of “items” you need and who is providing you these “items”).
- And finally, the allocations, that are “chunks” of inventories (resources + providers), associated to a consumer; thus who is retrieving something, what (resource) and where (provider).
Because I’m not the brightest crayon in the box, it took me a while to understand this new data model…
Minimum bandwidth scheduling.
Or in other words, schedule a virtual machine depending on the available network bandwidth of a network card. In Neutron this is closely linked to the QoS extension but it is worth mentioning that:
- The minimum bandwidth scheduling is a filter for the Nova scheduler.
- The Neutron QoS extension enforces the QoS rules on the backend.
These are two completely different concepts. The first one (scheduling) won’t allow to spawn more virtual machines if the required bandwidth exceeds the available in the required network card. The second one will use the network capabilities (QoS in OVS, “ip-link vf” in SR-IOV) to enforce the minimum bandwidth on the interface. In this post we are referring to the first one only.
Following the data model briefly explained in the previous section, when a mechanism driver agent starts (ML2/OVS, ML2/SR-IOV), it creates a resource provider associated to this agent and nested under the node resource provider. For example:
Compute RP (name=hostname)
+
|
+-------+Network agent RP (for OVS agent), uuid = agent_uuid
| inventory:
| +
| |
| +------+Physical network interface RP,
| | uuid = uuid5(hostname:br0)
| | traits: CUSTOM_PHYSNET_1, CUSTOM_VNIC_TYPE_NORMAL
| | inventory:
| | {NET_BW_IGR_KILOBIT_PER_SEC: 10000,
| | NET_BW_EGR_KILOBIT_PER_SEC: 10000}
| |
| +------+Physical network interface RP,
| uuid = uuid5(hostname:br1)
| traits: CUSTOM_PHYSNET_2, CUSTOM_VNIC_TYPE_NORMAL
| inventory:
| {NET_BW_IGR_KILOBIT_PER_SEC: 10000,
| NET_BW_EGR_KILOBIT_PER_SEC: 10000}
This is much better described in the Neutron spec QoS minimum bandwidth allocation in Placement API, in the “Networking RP model”.
The resource classes that could be used are the ingress and the egress kbits/s, as defined in openstack/os-resource-classes. The trait (type of provider) is the physical network; to this physical network a interface is connected (network card, bond, VTEP, etc). The agent reads the bandwidth assigned to this interface and assigns the corresponding inventory for this resource provider.
ML2/OVN bandwidth reporting.
But in ML2/OVN the bandwidth reporting differs a bit. Do you remember that in ML2/OVN there are no agents? How the resource providers are created then? It is the Neutron server who reads the OVN Southband “Chassis” registers and reads the bandwidth assignation per network interface and host. It’s doing the same as in other mechanism drivers, but in this case the host configuration is defined in the local OVS Open vSwitch register, as any other local OVN configuration parameter, under the “ovn-cms-options” key, in the “external_ids” field. This information is retrieved by the local ovn-controller service and populated into the OVNSouthband “Chassis” register.
During the Neutron server initialization, a MaintenanceWorker thread will call OvnSbSynchronizer.do_sync, that will call OVNClientPlacementExtension.read_initial_chassis_config. This method lists all chassis and builds the resource provider information needed by Placement. This information is stored in the “Chassis” registers, in “external_ids:ovn-cms-options”, with the same format as retrieved from the local “Open_vSwitch” registers from each chassis.
The second method to update the Placement information is when a “Chassis” registers is updated. The OVNClientPlacementExtension extension registers an event handler that attends the OVN Southband “Chassis” bandwidth configuration changes. This event handler builds a PlacementState instance and sends it to the Placement API. If a new chassis is added or an existing one changes its resource provider configuration, this event updates it in the Placement database.
Just for reference, this is the list of patches that implemented the Placement API reporting in Neutron ML2/OVN:
- [OVN][Placement] Add a SB Chassis event to track changes in BW config
- [OVN][Placement] Add support for minimum bandwidth QoS rules
- [OVN][Placement] Make the Placement reporter compatible with OVN
- [OVN][Placement] Read the initial config in the maintenance worker
- [OVN][Placement] Drive binding by placement allocation
The next post of this blog will talk about the Placemenet API again, this time about the support for minimum bandwidth rules in tunnelled networks. Stay tuned!