Today’s post is… a bit dense, I will admit. And to make things a bit more complex, I’m going to talk about the external ports too but for a reason. Actually this post comes from a bug (no way!) that claims that the external ports are usually bound to chassis different to the router gateway ports. That’s correct: the method to decide what is the selected chassis for a router gateway port is different from the method used to bind the external ports.
External ports in OVN.
Since this patch, Neutron supports external ports. For example, SR-IOV ports connected to virtual machines. These ports are not directly handled by OVN nor the local OVS instances. However, OVN creates a logical port with type “external”. From the Neutron documentation, “OVN allows for setting up a port that lives externally to the instance and is reponsible for replying the ARP requests on its behalf”. That means OVN creates another port (and OVS port) in other host and this port receives the traffic sent by the physical one (the virtual machine port) and replies to the traffic. That gives OVN the ability to, for example, reply to DHCP requests and provide an IP for this port without needing an external DHCP server.
For that purpose, OVN uses a gateway chassis, that is a chassis with external connectivity. The logical port is bound to this gateway chassis and thus has connectivity with the physical port. To set a chassis as gateway we define it in the external-ids:ovn-cms-options of the Open vSwitch register of the local OVS service, with the flag enable-chassis-as-gw:
$ovs-vsctl set open . external-ids:ovn-cms-options=enable-chassis-as-gw
This chassis should also have the proper bridge mappings connecting to the corresponding provider network (OpenStack OVN manuall install).
If we have one single gateway chassis, we know where the port is being located. But when we have multiple chassis, where the external port is bound? That was solved in this series of patches in core OVN: using the HA chassis group.
HA chassis group.
Yeah… a new concept, but easy to understand. A HA chassis group is a register that links to one or many HA chassis registers. Each HA chassis has a 1:1 relationship with a gateway chassis (that means a chassis with external connectivity) plus a number that is the priority. So a HA chassis group is a list of chassis with external connectivity with a defined priority. The HA chassis register with the highest priority will be the master of this HA chassis group. If this master chassis fails, the responsibility is handled over the next highest HA chassis register.
Both the Logical Switch Ports and the Logical Router Ports have a field named ha_chassis_group. When set, OVN binds the port to the master HA chassis. This strategy is used by Neutron to schedule the external ports, levering the control of where to bind these ports to OVN.
How the router gateway ports are scheduled.
Now (just some weeks before the Antelope 2023.1 release), Neutron does not use the HA chassis group for that purpose with the router gateway ports. This is the cause of the previously mentioned bug: the SR-IOV port (external ports) is bound to a different chassis than the router gateway port. The external port has have very low performance when communicating through that router.
It’s worth mentioning, before continuing with this post, that routing SR-IOV port traffic though any Neutron router is not recommended. Any routing will be done at software level (in the iptables routing done in the L3 agent or in the OVS OpenFlow rules defined by OVN). Any performance improvement achieved by using SR-IOV will be lost.
The following link provides a very detailed description of how to manually create a router, two networks and how to route both. This link also describes how to create a provider network that will be used as gateway network. All this process is what Neutron does internally when ML2/OVN mechanism driver is used. Here is an example of how to create networks, ports and routers using the OpenStack CLI.
At that point, OVN can bind the gateway router port to one gateway chassis. In a non-HA environment, the gateway port is scheduled on a single chassis. But in HA mode Neutron uses a different strategy than the external ports one.
Neutron has a series of schedulers to calculate, per logical router port, the gateway chassis list and the priority. This list is assigned to the gateway_chassis field:
ovn-nbctl list logical_router_port lr0-public
_uuid : 0ced9cdb-fbc9-47f1-b2e2-97a49988d622
enabled : []
external_ids : {}
gateway_chassis : [6f2921d4-2555-4f81-9428-640cbf62151e, 745d7f84-0516-4a0f-9b3d-772e5cb58a48, 97595b29-139d-4a43-9973-8995ffe17c64]
ipv6_ra_configs : {}
mac : "00:00:20:20:12:13"
name : "lr0-public"
networks : ["172.168.0.200/24"]
options : {}
peer : []
The schedulers have different strategies to define the chassis priority, but the goal is the same: to list the available gateway chassis, provide a priority and create a Gateway Chassis register per chassis. This register is very similar to the HA chassis register; it has a 1:1 relationship with a chassis and a priority assigned. Once the Gateway Chassis registers are created, the scheduler writes this list into the gateway_chassis field of the Logical Router Port (see upper register).
Unlike when using HA chassis group, the redundancy is controller by Neutron, not OVN. Neutron is always listening to the Chassis events. If a chassis fails or is removed, Neutron rebuilds the gateway chassis list per router port.
Where is the problem and how to fix it.
The HA chassis group scheduling strategy will use always one master chassis. All ports will be bound to this chassis.
The L3 schedulers, for example OVNGatewayLeastLoadedScheduler, will try to balance the port binding between the existing gateway chassis. That is more efficient in terms of performance because the traffic will be distributed between different hosts.
But when external ports are used, the solution is to use the same scheduling strategy for both port types, external and router ports. Because the L3 service allows to configure the L3 scheduler to be used, this is where we are focusing our efforts.
This patch is implementing a new L3 scheduler, OVNGatewayHAChassisGroup, that will use HA chassis group for any new gateway port created. The scheduler to be used is configured in the config knob ovn_l3_scheduler, under the [ovn] section. If the wind blows and the code is accepted, if will be merged during the first weeks of the Bobcat 2023.2 release.