BGP full table question

So I'm talking to vendors for a solution for a project, one of them is suggesting the switch I was looking at, even though it list all the normal BGP features (4byte ASN, comunities, etc etc) might not be able to handle the full BGP table of 300k. (we're doing multi-homed + peering stuff, so definitely won't just be 1)

As such they're talking about just putting a router above that can handle. Now, I've done some research, back with 48k routes 512MB was enough, 1GB recommended.

I can't see what the switch I'm looking at has, but surely it'd have ~8GB and equivalently better CPU than the cisco 3500 suggested back then.

Oh, Dell S4810 and similar is the perspective model.

Alternatively, someone that isn't a marketting guy trying to make commission

 

The global v4 routing table is something like 410k routes now, tech sheet for that switch says it supports 16k. Not enough.

Suggestions on hardware that will do it really depends on whether you need to route out to your provider at >1G. If you can do it on 1G port it's cheap and easy. If you need to take a full table on a 10G port then the cost effectiveness depends on the scale of what you're doing and whether you need something that'll do line rate or not.

 

BGP table version is 3558942, main routing table version 3558942
420219 network entries using 50846499 bytes of memory
866802 path entries using 45073704 bytes of memory
137973/68368 BGP path/bestpath attribute entries using 10485948 bytes of memory
91 BGP rrinfo entries using 2184 bytes of memory
60315 BGP AS-PATH entries using 1792960 bytes of memory
51 BGP community entries using 1224 bytes of memory
24 BGP route-map cache entries using 768 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 108203287 total bytes of memory
416254 received paths for inbound soft reconfiguration
BGP activity 505603/85381 prefixes, 2287979/1421177 paths, scan interval 60 secs

Thats the current BGP table as of this morning

This router is just running a single BGP peer and the actual memory usage on the router is 400M, our route reflectors on the other hand that have multiple copies from our other 6 peers are running at 600M, We did try it with 512Mb and it fell over (That was only because we didn't notice it was 512Mb)

 

Heh, yea we need a crap load more than 10GE. Think hundreds of Gbit. Think datacenter/peering exchange levels of traffic.

I just can't see how a switch with 48x10GE + 4x40GE has BGP yet can't do full routing :/ Or maybe its just got good switching chips, but crappy routing? (thinks back to Cisco chassis, where you can add routing engines, switching, firewall blah blah).

Anyway, really need to sit down with someone from somewhere like Dell who can consult on those levels of BW I guess :/

 

what makes you think you NEED a full routing table?

 

Nothing per se, but I'm pretty sure we'd go over the 16k the S4810 can handle. Although, thinking again, if it is just he.net transit and AMSIX peering, perhaps that wouldn't be more than 16k :/

Eventually though we'll have way more transit and peering, surely it would go that large?

Also, why would it only be able to handle 16k routes? It can handle 1.28Tbps of switching, but not routing? Is it because that switching is only via a small number of ports, ie not that much in the arp table, ie nowhere near 400k in the table when you do the lookup to find out where to send shit?

 

Routing is harder to do than switching. Also, it allows the vendor to charge more for more features

I don't know much about that Dell switch, but perhaps a Nexus 5k with the layer 3 module installed might do what you're after? Hrm on second thought it doesn't look like it will handle a full BGP table either. I agree with ltd73 in that you probably don't really need a full routing table...

 

The route servers at AMS-IX are currently carrying around 50k IPv4 prefixes, the Force10 4810 won't even take that.

Might be better if you told us what you're trying to achieve..

 

Hundreds of Gbit via he.net and AMSIX, then moving onto other hundreds of Gbit via whatever other peers/transit.

Seems to me we're forced into getting some massive routing device like the e600 or e1200i, or pay a monthly premium

 

I was more referring to what function the network will be performing. If you're putting a network of seedboxes behind it then ltd is on the money and you don't need a full table. You probably don't even need the AMS-IX ports either, just get more from HE and call it done. They're cheap enough.

If you're selling BGP feeds to customers then you do need to carry a routing table. In that scenario then yeah, buy some proper routers. Last time I looked at Force10 the E series would only take ~500k IPv4 prefixes which won't be enough long term. It already might not be enough depending on their CAM partitioning. I think they had some cards coming to increase that but not sure.

 

don't take this the wrong way but i strongly suggest you get someone in who knows what they are doing.

you can have multiple bgp sessions with just default-route outbound and/or some minimal number of prefixes accepted for traffic engineering.

the silicon in question inside that L3 switch is "Broadcom Trident+".
It has a 16K LPM (longest-prefix-match) table which is in the form of an algorithmic (m-trie) lookup table.

the number of memory gates required for that 16K LPM space is more than the memory space required for the 128K MAC table (L2).
Its also independent to that of the 16K ARP table too (which is a hash-table lookup not a LPM table as above).

Reality is that you need to use the right product in the right place.
Or learn where and how you run bgp without necessarily ingesting a full table.

Or talk to a clueful networking vendor.

its no harder to do 'routing' just that the table sizes are larger if you wish to hold a full routing table. e.g. a 2M prefix table is 128x larger than a 16K LPM prefix table and that eats up die space / logic / power and e.g. if you had all that built-in on chip, the chip in question would no longer be 64x10G but would probably at best be 8x10G.

i find this statement very very funny. Because the L3 module inside a Nexus 5K is actually the _same_ silicon as the OP was talking about on their Dell switch. Just Cisco only use it for L3 on an oversubscribed daughtercard...

I have plenty of customers peering to AMS-IX and similar using Arista 7050 switches - which are the same silicon as that of the Dell 4810 you mentioned above with no issues whatsoever.

 

Well not *that* cheap, he.net charge 6k USD vs the 2200 or so for 10GE of peering. And if that is where the traffic will go.. 3x cost right there

Trying to, harder than it sounds Have no answer from one guy at Dell, no answer yet from the tech guy at Dell.. so..

The Arista 7050, how many routes are they ingesting into that?

Happy to sit down with someone, have tried to describe what I want with a simple visio diagram, so far no one seems to have understood the simplicity. I guess they are used to dealing with hundred million dollar companies at this level of bandwidth requirement :/

PM me an email or something, perhaps you can assist? Or if in Melbourne I'll do you lunch whereever you want

Oh and thanks for explaining the routing issue, I figured as much but didn't know the details

 

I am read this forum trail correctly, I've only gathered these requirements so far:

• Multiple gigabit links (up to 10ge)
• BGP peered

First question. Is the traffic going to be mainly sourced or sunk into your network? That is, will your network be mainly download or uploading content to your peers?

Second question will you be allowing transit? (That is will you allow peer X to use your network to access Peer Y?)

Third, based on the first question do you need control over the outbound traffic? (Traffic exiting your network) That is, do you need the ability to determine which link to use or can you tolerate a simple dumb load balance over all links?

If what your trying to do is confidential, then are you able to provide something that might explain your goals?

Instead of trying to find a product that can hold the full BGP table (Which there are lots of with varying degrees of port options and performance characteristics), we should be identifying the requirements and the scope of what your trying to do. Then find the product or products to meet that.

 

Er yea, multiple 10GE of transit/peering, growing hopefully to hundreds of 10GE

BGP routing yes, transit, peering, and IX peering.

Outbound mostly.

Will definitely need control, no point using expensive transit if there is a peering route.

Have a look at this: http://i.imgur.com/ZhyKD.png

 

happy to help out, yes I'm in melbourne. drop me an email ltd@aristanetworks.com

capabilities of the arista switches are per the datasheets. Arista 7050S/T/Q are 16K IPv4 route prefixes + 16K ARP entries (/32 IPv4 prefixes).

you could take a bgp feed from N places and selectively just use a subset of that for some traffic engineering.
alternately advertise your prefix to whomever you peer/transit with and just see how much comes 'back' to you.

 

Ltd, you know that this is the realm of the 12000WB... please make that the first suggestion next time.


@op, do you need the whole table? most carriers will offer just a domestic table and then you can default the rest.

 

*kung pow*

 

careful! your powers-that-be are looking at our kit too....

 

I sense inside jokes. You guys must know each other outside OCAU

Uhm no don't need full table per se, just want the right traffic to go to the right place, cheap as possible.

Cheers will drop you a line.

 

Right traffic to the right places?