Cisco entry point for multigig QinQ termination

Discussion in 'Business & Enterprise Computing' started by 7nothing, Nov 29, 2017.

  1. 7nothing

    7nothing Member

    Joined:
    Feb 15, 2002
    Messages:
    1,394
    Location:
    Brisbane
    So, first time configuring ExpressRoute for Azure, looking at 2x 1G expressroute over a 10G physical from megaport (accepting we don't have diverse path or redundant routers anyway). This is where I realise an IP Services L3 switch supports .1q tunnel but not QinQ termination.... damn.

    What's a multi gig router worth? $40k for an ASR 1001-X limited to 2.5G CEF? Fuck that.

    Megaport into .1q tunnel, out of another .1q tunnel switchport access vlan X to remove outer tag, into regular trunk port. Duplicate for outer vlan Y.

    $20k worth of feature improvement for the price of 2 patch leads. Seems to be working, kinda proud of my hack.
     
  2. tensop

    tensop Member

    Joined:
    Mar 26, 2002
    Messages:
    1,190
    This... almost needs pics :)
     
  3. ir0nhide

    ir0nhide Member

    Joined:
    Oct 24, 2003
    Messages:
    3,791
    Location:
    Adelaide
    I suppose since ExpressRoute supports that many subnetworks, you're not likely to run out of physical port capacity any time soon. Interesting idea.
     
  4. OP
    OP
    7nothing

    7nothing Member

    Joined:
    Feb 15, 2002
    Messages:
    1,394
    Location:
    Brisbane
    Never thought I'd find a legit reason (other then fucking a network with broadcast storm) to plug both ends of a cable into the same switch, but here we are


    Click to view full size!


    ...I'll get some shorter leads. Don't think of them as ethernet cables, think of them as gigabit routing feature keys :)

    If we ever surpass 1G on single ExpressRoute, I'll just have to stack with a 10G switch and get some twinax feature keys...

    Obvious drawback is the inner VLANs are all common, but that's not an issue for me, looks like it'll be a perfectly serviceable solution for this scenario.

    I told Cisco I'd found a workaround and they could close the case, guy asked what my workaround was... fuck that, my intellectual property, do your own R&D Cisco :)
     
  5. wintermute000

    wintermute000 Member

    Joined:
    Jan 23, 2011
    Messages:
    948
    erm I've done Q-in-Q on 3850s for example, not cheap but not exactly ASR prices even with IP services equivalent. In fact I've Q-in-Qed to Azure via Megaport with 3850s. Damn sight cheaper than ASRs... on paper a 3750X could do it too. There's a thousand google hits on "3750 Q-in-Q".

    I'm not sure why you think L3 switches can't do q-in-q? Though I haven't explicitly checked e.g. NXOS for example.

    I've always been confused as to why Megaport do it that way, saves them a Q-in-Q switch at one end I suppose ROFL its just that when I did SP stuff we would never ever trust the customer to tag correctly LOL

    cabling to same switch, nice dirty hack. Glad you found a creative way.

    EDIT your pic looks suspiciously like a 3750X. Here's proof... on ancient 12.2 IOS no less...

    https://www.cisco.com/c/en/us/td/do...se/configuration/guide/3750xscg/swtunnel.html
     
    Last edited: Dec 3, 2017
  6. OP
    OP
    7nothing

    7nothing Member

    Joined:
    Feb 15, 2002
    Messages:
    1,394
    Location:
    Brisbane
    Switches most definitely support QinQ tunneling, I don't believe they have any support for termination of QinQ frames without my hack.
     
  7. wintermute000

    wintermute000 Member

    Joined:
    Jan 23, 2011
    Messages:
    948
    oh OK in my case I was terminating on an aggregation router behind and the cat was purely L2
     
  8. freaky_beeky

    freaky_beeky Member

    Joined:
    Dec 2, 2004
    Messages:
    968
    Location:
    Brisbane
    I've been following this with a bit of interest (despite my incredible lack of cisco & networking config skills). It just seemed to me that having to do this was less than an ideal solution.

    I was having a bit of a chat with the principal network guy at work and while he demystified this a bit for me, he also pointed me in the direction of this cisco support post which, other than being related to a 4500-X pretty much seems, on point for what you have accomplished. Unfortunately the OP in the linked thread, doesn't reply that it worked, but I have a feeling that it might put you on a potentially better track?
     
  9. Dre_

    Dre_ Member

    Joined:
    May 25, 2014
    Messages:
    693
    They never do.
    Assume that it worked, otherwise they would have come back and bitched some more...
     
  10. freaky_beeky

    freaky_beeky Member

    Joined:
    Dec 2, 2004
    Messages:
    968
    Location:
    Brisbane
    My thoughts precisely! I posted it here, in the hope of getting a confirmation from our very own OP!
     
  11. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    2,852
    Location:
    Canberra
  12. OP
    OP
    7nothing

    7nothing Member

    Joined:
    Feb 15, 2002
    Messages:
    1,394
    Location:
    Brisbane
    I know my way around basic switching and routing, first time I've touched QinQ. My initial lack of understanding difference in feature set comes down to the fact that while it's a simple feature (how hard is it to push/pop an outer tag?), seemingly QinQ tunnel is a common provider edge feature, QinQ termination is more provider aggregation or core. I came across that same Cisco post, after confirming "switchport vlan mapping" wasn't an available command on C3850, wasn't any more relevance for me there.

    I did check feature navigator beforehand, there's a feature for '802.1Q-in-Q VLAN Tag Termination' (by the way, click "View Desc" on Cisco feature navigator for that, and it takes you to the QinQ tunneling feature page), but only listed platform are Nexus switches, there's another feature just called QinQ, only lists 2960-X, so I'm thinking, course it'll be fine on the 3850.

    So I had no real hope from the start. Found a hack I'm pleased with, is it ideal given all possible scenarios? nope. Is it ideal given the situation I was in? Well, you'd have to ask the boss. He was pissed that we were already "over budget" (maybe don't set arbitrary budget before scope is formed?) and really wasn't going to spend $40K on a 2 port ASR (which was Megaport's only suggestion), but when I came in the morning after telling him I was gonna try something crafty, and he'd got the emails about megaport service activation the night before, he was pretty stoked.

    Good to know, I did consider going cheaper-than-cisco router, honestly didn't think it'd be available in software.

    Still, does anyone else have a good reason to plug both ends of a network cable into a switch? (aside from aformentioned broadcast storm, and those old model HPs which would factory reset if powered on with ports 1&2 patched together)?
     
    Last edited: Dec 6, 2017
  13. Dre_

    Dre_ Member

    Joined:
    May 25, 2014
    Messages:
    693
    Keep dust out of the ports
     
  14. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    2,852
    Location:
    Canberra
    back in the day before software variable latency / packet loss tools existed, lab testing a simulated packet loss situation by deliberately terminating 8p8c to generate NEXT / FEXT.
     
  15. wintermute000

    wintermute000 Member

    Joined:
    Jan 23, 2011
    Messages:
    948
    Its actually expensive in silicon to do this kind of operation. Why do you think MPLS capable switches are expensive? Any why don't the merchant silicon DC switches that dominate these days do MPLS properly? The silicon logic to deal with variable length labels is not cheap. I've heard from more than 1 vendor SE that this is one of the big reasons VXLAN dominated over MPLS in the DC space despite MPLS being technically capable of doing the job of encapsulating MAC in L3 - I mean shoot, with labels I can do whatever I want - but the fixed VXLAN header was cheap in silicon. Another example, again from the VXLAN world - first gen VXLAN capable merchant silicon couldn't do IRB, and then they started hacking the capability via recirculating the packet - again, the ASICs just weren't built to double handle the encap.

    PFsense is purely in software so different rules, and again the whole software vs hardware thing is a different discussion :) I'm impressed though.

    FWIW you could have just used any router on a stick and stuck it on to the switch taking the inside VLAN, didn't have to be a 40k ASR, if you were OK with two devices doing the job not just one. Both times I've done megaport to cloud I've just terminated a L3 subinterface into the Q-in-Q encapsulating switch.
     
    Last edited: Dec 7, 2017
  16. OP
    OP
    7nothing

    7nothing Member

    Joined:
    Feb 15, 2002
    Messages:
    1,394
    Location:
    Brisbane
    Yea, but any real router was going to need gig throughput per interface, and I didn't need anything beyond what an ip services switch can do, there were 45 ports sitting there vacant...
     
  17. itsmydamnation

    itsmydamnation Member

    Joined:
    Apr 30, 2003
    Messages:
    9,837
    Location:
    Canberra
    Do this all the time with Vdc at way higher cost per port.

    I've seen it as a hack for things that don't support vlan mutation. I've done it heaps on lab equipment when using one switch and vrfs to simulate many devices.

    That's about all I can think of at the moment.
     

Share This Page