2nd vmkernal is not reachable

Discussion in 'Networking, Telephony & Internet' started by Multiplexer, Mar 24, 2020.

  1. Multiplexer

    Multiplexer Member

    Joined:
    Feb 26, 2002
    Messages:
    2,093
    Location:
    Home
    I am building a vSAN lab using pfsense as a gateway.

    I have created another vmkernal adapter for vSAN connection. The issue I am facing is I cannot ping anything in the vSAN subnet, not even the gateway. I can confirm the pfsense gateway works fine using a Windows virtual machine.

    Any idea what I am doing wrong?

    Management
    IP: 192.168.11.201
    Subnet: 255.255.255.0
    Gateway: 192.168.11.99

    vSAN
    IP: 192.168.12.201
    Subnet: 255.255.255.0
    Gateway: 192.168.12.99

    Code:
    [root@esxi1:~] esxcli network ip interface ipv4 get
    Name  IPv4 Address    IPv4 Netmask   IPv4 Broadcast  Address Type  Gateway        DHCP DNS
    ----  --------------  -------------  --------------  ------------  -------------  --------
    vmk0  192.168.11.201  255.255.255.0  192.168.11.255  STATIC        192.168.11.99     false
    vmk1  192.168.12.201  255.255.255.0  192.168.12.255  STATIC        192.168.12.99     false
    vmk2  192.168.13.201  255.255.255.0  192.168.13.255  STATIC        192.168.13.99     false
    
    Code:
    [root@esxi1:~] esxcli network ip route ipv4 list
    Network       Netmask        Gateway        Interface  Source
    ------------  -------------  -------------  ---------  ------
    default       0.0.0.0        192.168.11.99  vmk0       MANUAL
    192.168.11.0  255.255.255.0  0.0.0.0        vmk0       MANUAL
    192.168.12.0  255.255.255.0  0.0.0.0        vmk1       MANUAL
    192.168.13.0  255.255.255.0  0.0.0.0        vmk2       MANUAL
    
    Code:
    [root@esxi1:~] esxcli network ip interface list |grep -E 'vmk|Netstack'
    vmk0
       Name: vmk0
       Netstack Instance: defaultTcpipStack
    vmk1
       Name: vmk1
       Netstack Instance: defaultTcpipStack
    vmk2
       Name: vmk2
       Netstack Instance: defaultTcpipStack
    
     
    Last edited: Mar 24, 2020
  2. DRAGONKZ

    DRAGONKZ Member

    Joined:
    Jul 3, 2001
    Messages:
    1,801
    Location:
    Bankstown, Sydney
    Can your hosts ping each other on their vSAN respective vmk?

    It’s would also be a good idea to put some more info about your setup, eg host many hosts, if direct connected, vlans...
     
  3. fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,365
    Location:
    City, Canberra, Australia
    I would use a different stack for iscsi/vsan and vmotion.

    What VLANs are configured? What switch are you using?


    Edit: It has been a while since I shutdown my vSAN cluster.
     
    Last edited: Mar 25, 2020
  4. DRAGONKZ

    DRAGONKZ Member

    Joined:
    Jul 3, 2001
    Messages:
    1,801
    Location:
    Bankstown, Sydney
  5. OP
    OP
    Multiplexer

    Multiplexer Member

    Joined:
    Feb 26, 2002
    Messages:
    2,093
    Location:
    Home
    So I have to use the default tcp/ip stack for all the vmkernal adapter (vmk1-management, vmk2-vSAN and vmk3-vMotion) where each adapter is in a different subnet, I will need to setup routing?
     
  6. DRAGONKZ

    DRAGONKZ Member

    Joined:
    Jul 3, 2001
    Messages:
    1,801
    Location:
    Bankstown, Sydney
    vMotion can use the vMotion stack if you need to route the traffic across sites (Eg via long distance vMotion), vSAN and management traffic should use the default.

    vSAN hosts in the same cluster are all normally in the same layer 2 network with no need to route vSAN traffic unless if you’re running a stretched cluster, but that also introduces more rules/complexities to the picture.

    It’s hard to give any more recommendations without knowing more about your setup.
     
  7. OP
    OP
    Multiplexer

    Multiplexer Member

    Joined:
    Feb 26, 2002
    Messages:
    2,093
    Location:
    Home
    I want each vmkernal to have its own subnet. The issue I am experiencing is vmk1 and vmk2 is not reachable. What is the normal implementation?
    Code:
    vmk0: Management     - 192.168.11.201/24, gateway=192.168.11.99
    vmk1: vSAN             - 192.168.12.201/24, gateway=192.168.12.99
    vmk2: vMotion         - 192.168.13.201/24, gateway=192.168.13.99
    
    [​IMG]
     
    Last edited: Mar 27, 2020
  8. DRAGONKZ

    DRAGONKZ Member

    Joined:
    Jul 3, 2001
    Messages:
    1,801
    Location:
    Bankstown, Sydney
    Ok, so you have 3 x vDS, with a single uplink, and no tagged VLANs are configured on the vmware side.

    What’s the physical cabling and switch setup look like?

    Are there VLANs on the physical switch side?

    If you were to ignore the cabling/switch for a moment, even though the config is not ideal and gives no fault tolerance, it should technically work as long as all hosts have the same config with different individual IP addresses.

    You can SSH to a host and use vmkping whilst specifying a vmk and should be able to ping the other addresses in the respective subnet.

    Don’t try to ping addresses in other subnets as your config won’t allow it.
     
  9. DRAGONKZ

    DRAGONKZ Member

    Joined:
    Jul 3, 2001
    Messages:
    1,801
    Location:
    Bankstown, Sydney
    In terms of a “normal” config it’s case dependant, but a very basic setup might look like:

    3 x hosts, identical config, with 2 x 10Gb NIC ports

    Each NIC is plumbed in to a physical switch port (normally 2 physical switches and 1 NIC port per switch for redundancy)

    VLAN tagging on the physical switch is used for all ports (Ports potentially in trunked mode)

    1 x vDS with 4 port groups, each on seperate tagged VLAN.

    Both physical NICs are assigned as uplinks, and the order can be changed per port group.

    eg, VLAN 1 = management, 192.168.1.x, vmk0
    VLAN 2 = vMotion, 192.168.2.X, vmk1
    VLAN 3 = vSAN, 192.168.3.X, vmk2
    VLAN 4 = VMs, 192.168.4.x, no vmkernel

    Hope that helps somewhat.
     
  10. OP
    OP
    Multiplexer

    Multiplexer Member

    Joined:
    Feb 26, 2002
    Messages:
    2,093
    Location:
    Home
    I am building a nested vmware lab.

    So based on the reply, I should have 1 vDS, with 2 uplink. multiple port group with its own VLAN. I think this is how my work is implemented.
     
  11. whysmell

    whysmell Member

    Joined:
    Aug 16, 2006
    Messages:
    28
    Location:
    Kilsyth
    From your earlier screenshot it doesn't appear that you have a vlan id on those adapter, though I don't believe that should stop you from pinging.
    I would be interested to see the vmk adapters page, and the configuration of the vsan D's port group.
    Have you made any changes to the MTU?
     
  12. OP
    OP
    Multiplexer

    Multiplexer Member

    Joined:
    Feb 26, 2002
    Messages:
    2,093
    Location:
    Home
    No VLAN configured and no changes to MTU. I just want to implement a typical standard corporate approach/design. Not having done VMware cert or have a guide, I just thought what I have is correct. One thing worth mentioning is the ESXi is able to ping itself. I have not tried wkping.
     
  13. fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,365
    Location:
    City, Canberra, Australia
    If it is nested theb put them on on the same subnet.

    There was a great powershell script to deploy this scenario. Would have it running from 0 in 34mins.
     

Share This Page

Advertisement: