ovs, kvm, RHEL8 - guest to guest communication issues

Does anyone have experience using Red Hat 8, kvm and open vswitch?

I have a RHEL8 host using kvm with open vswitch but I am unable to get connectivity working between guest machines. The setup is as follows:

Physical Switch → trunk port → eno1 (host NIC) → ovs bridge0 → ovs port/interface (van 24) → guest

The management interface is setup on the second physical NIC and I am only testing with vlan24 at the moment so once this up and working then I will replicate/migrate the other vlans I need. All required vlans are allowed on the trunk port of the physical switch, connectivity from the guests to the physical network is working as expected, I can ping from guest in vlan 24 to a physical host in vlan 24.

The issue is that I cannot get traffic to pass directly between the guests in the same vlan (vlan24). I have both guests attached to the same ovs port/interface and the ovs port is configured with the correct vlan tag.

I checked the ovs arp table and I can see the MAC addresses of the physical host and the guest machines so ovs is correctly learning the MAC addresses:

Code:

# ovs-appctl fdb/show bridge0

 port  VLAN  MAC                Age

   12    24  3c:df:1e:97:ce:c2   11

   12     0  3c:df:1e:97:ce:94    5

   33    24  52:54:00:0b:19:1e    0

   33    24  52:54:00:3d:cb:9d    0

Below is the output of the port/interface setup from nmcli:

Code:

# nmcli conn show

NAME              UUID                                  TYPE           DEVICE           

eno2              40309b82-edec-449e-a703-975288f2e8e8  ethernet       eno2             

bridge0           b4811e6f-29da-4625-aa72-e05c310c07bb  ovs-bridge     bridge0          

bridge0-ovs-int   b2317363-ae1e-437e-88a2-c6ad59385719  ovs-interface  bridge0          

bridge0-ovs-port  05170645-df46-4426-88da-3550ec754b77  ovs-port       bridge0-ovs-port 

eno1              e2c3c790-34b3-403d-ab7e-e9ca6e30a6ab  ethernet       eno1             

vlan100           e3102987-ab5a-48fa-9977-4ce141c8cc48  vlan           vlan100          

vlan23            59f9dafc-3a7f-48d5-95bd-2ce8fd080b44  vlan           vlan23           

vlan24-ovs-int    25ae35eb-9d5f-4036-8244-f13a41306829  ovs-interface  vlan24           

vlan24-ovs-port   34cb47d3-9649-4123-97e7-1b36d4af70dc  ovs-port       vlan24           

Code:

# nmcli device status

DEVICE            TYPE           STATE      CONNECTION       

eno2              ethernet       connected  eno2             

eno1              ethernet       connected  eno1             

bridge0           ovs-bridge     connected  bridge0          

bridge0           ovs-interface  connected  bridge0-ovs-int  

vlan24            ovs-interface  connected  vlan24-ovs-int   

bridge0-ovs-port  ovs-port       connected  bridge0-ovs-port 

vlan24            ovs-port       connected  vlan24-ovs-port  

vlan100           vlan           connected  vlan100          

vlan23            vlan           connected  vlan23           

lo                loopback       unmanaged  --               

macvtap27         macvlan        unmanaged  --               

macvtap28         macvlan        unmanaged  --               

eno1              ovs-port       unmanaged  --               

The current ovs config:

Code:

# ovs-vsctl show

dd0de2f1-4f76-41ed-8a98-605021f9667f

    Bridge "bridge0"

        Port "vlan24"

            tag: 24

            Interface "vlan24"

                type: internal

        Port "bridge0-ovs-port"

            Interface "bridge0"

                type: internal

        Port "eno1"

            Interface "eno1"

    ovs_version: "2.12.0"

I understand that in general linux implements some arp filtering on the host itself, but from everything that I have read this does not seem apply to ovs? In any event I have not been able to find out how to check for or disable any said filtering (specifically for RHEL8).

I attempted to install ovs-tcpdump to see if this could tell me anything, but I have not been able to find a repo or rpm for RHEL8. I am using Red hat developer subscription, but it seems that this is limited when it comes to ovs/openstack repo access.

This is not a complex setup, I am simply setting up network segmentation for my home labs, vpns, wireless appliances etc however I have spent a huge amount of time on this over the past couple of weeks for what really should be a pretty simple job (and learning way more than I ever thought I would about linux bridging), keeping in mind this is also the first time I have used ovs.

I am just hoping that this issue is something simple I am overlooking. Appreciate if anyone has gotten a similar setup to work and could give me some hints. I have been reading a lot of ovs documentation but I have not found a reason for this problem, unless I am overlooking the obvious, to me this looks like it should be working.

 

For those following along at home, I found the fix for this and yes it was very obvious. In the same way a physical switch port cannot connect more than one end host, the same applies here. Simply creating a specific ovs port/interface pair, tagged in the same vlan for each guest solved the problem.

I guess I was used to this 'single' vlan interface from configuring VMware test environments in the past, where you can just create one interface per vlan and then attach it to multiple hosts.

Anyway, lesson learned, next time stop and think about the problem for a while instead of diving down rabbit holes. On the up side I now know a shed load more than I did before about linux networking. (not hard really as before I knew nothing )