Consumer 10GbE discussion

As I'm typing this I'm copying around 3TB of data between my NAS boxes and I'm watching it sit at 100MB/s or so. The arrays themselves can do 300MB/s+ but because consumer networking hasn't really changed in the last 10 years, I'm stuck transferring data at one third of it's potential. Lets not get started on SSDs. "Surely there is a better option", I thought.

I thought I'd check up on what 10GbE is up to in this day and age, which from memory was introduced in 2006 - nope, still outrageously expensive and obviously targeted towards the enterprise sector. I want a basic cheap (<$200) switch with a couple of cheap and basic 10GbE NICs (<$100) to suit. Considering you can buy a brand new 1gbps NIC for $5 these days, it saddens me that affordable 10GbE hasn't taken off. I can see so much potential in the SOHO sector nowadays.

Is affordable 10GbE hiding from me or is it still a distant dream?

2021 edit: I am now using 10Gbe using SFP DAC into a reasonably affordable switch, but having to use SPF DAC limits my ability to use existing ethernet, so I only have it for a couple of machines. Can't believe how slow the progress has been and the fact that this thread is still active lol...

 

yeah its slow, but for most people 10g isnt required.

the introduction of SSDs is driving the change because of the faster access, datacentres are seeing a lot of 10g 40g and 100g as hardware gets replaced roughly 6-7 years for most.

there was aome conversation around that they were going to try and ratify 25g, 40g is 4x10g and 100g is 4x25g so it kinda makes sense to make the silicon have 2 uses to bring thay cost down

 

I recall an interview (can't find it, sorry) from some years ago where some senior person from Linksys was asked much the same thing about the gigabit home router. the response is that it wasn't commoditised yet, which is a management buzzword for "nobody will pay for one".

until the average home user has a need to shift terabytes of data around quickly (which means drive arrays with capacity and sustained bandwidth to suit) then I don't expect to see the technology on the shelf at JB HiFi prices. the dog will be wagging the tail on that one.

it's been the same with every technology that has filtered down to the SOHO desktop. I recall a few years ago paying $160 for a secondhand 24-port 10/100Mbp switch that had been reviewed not 18 months before in Atomic with a list price of $2400.

 

I guess we'll just have to wait until there is a push for 10GbE. In the meantime, looks like I'll have to deal with setting up 2x1gbps links via link aggregation.

 

Have a look at Infiniband gear.. I got cards for about $40 on eBay and cabling is around $30 or so as well

The switch gear is expensive though, but not needed with basic setups.

I've pulled around 500 MByte/sec with Samba on IPoIB. 900 MByte/sec with RDMA

 

It will be a very very long time till "consumer" 10GbE is released.. Most companies still sell 100Mbps (Fast Ethernet) switches..

The cheapest solution I've found is to run 2x Server 2012 systems, 2x Intel Gigabit ET Dual Port Adapter's & Netgear GS724Tv3. Nothing on 10GbE but at least I get a very consistent 220Mb/s+

 

Infiniband is only good for short distances and as you said the switch gear is expensive.

I run 10gbe at home (dell 8132f with Intel x520 cards, so all fibre SFP+ and I have a 40gb qsfp module for it) and it's really not worth it for home use, I have arrays that push well over 1000Mb/s as well, you need to be transferring terabytes of data a day for it to be even remotely justifiable.

 

Hey can you describe your RDMA mechanism? Thanks.

 

It won't come to consumer, there's no volume. I expect wifi will get reliably faster than 1Gbps before we see wired 10Gbps at sub $100 price points due to that being the area of focus for the consumer market.

 

Infiniband doesn't really suit my situation and looks like a bit of a pain and the switches are very pricey.

Either way, this has solidified my decision to convert one of my Windows 8.1 storage boxes to Windows Server 2012 and finally make use of the dual NICs on the board. I even have a smart switch to pair it with.

On the flip side, it's a real shame that LACP can't be used on Windows 7/8.1 Pro - not sure how I am going to get around that limitation for the CAD boxes. I will be horrified if Wi-Fi becomes the solution :/

 

One way would be to go Fiber/SFP+ stuff, makes it a bit cheaper as there is a lot of 2nd hand stuff from server kits - then look for gigabit switches with say 2 or 4 SFP+ modules on them. That way you can have 2 servers sitting on the 10GBit ports while your clients are on 1Gbit, this should also mean you can transfer between your 2 servers at full 10gbit.

One such cheapish example:

http://routerboard.com/CRS226-24G-2SplusIN

Or Rackmount:

http://routerboard.com/CRS226-24G-2SplusRM

You can buy these units from local distributor such as duxtel (shop.duxtel.com.au)

 

point to point 10Gbps using Thunderbolt?

EDIT: PC support of course is somewhat lacking (Intel has yet to deliver).....

 

If you want 10GBE.. pay the premium for it. If you transfer massiva data back forward to the server at home, make the upgrade

People pay around 1000+ bucks for monitors and GPU

 

Last time I tried it was a few years ago. SDP with RDMA. Mounted a filesystem on my Linux storage box on my Windows workstation.

Windows 2012 now supports SMB Direct which is native. Haven't tried it yet (don't want a Windows based storage box anyway).

 

I know most people snicker at netgear, but if you want CHEAP 10Gb go their bottom of the line switch XS708E <$900 at only a bit more then $100 per port, that is damn cheap.

 

Problem is with it that they aren't particularly reliable from my testing it's one of the reasons why I stuck to dell

 

Thats a shame, a sub 1k 10g switch looks tempting. There are $300 10g NICs out there too.

Realistically though, dual or quad NICs with FreeNAS/server 2012 will work out cheaper for current workloads.

 

there's no volume at the moment. that will change. I suspect the driver will be multimedia - wait until the first 8K holographic theatre system appears. (or, based on the same theory, media for it does!)

I still think that a relatively cheap, very stable, high performance hard line is going to be the mainstay for some time.

 

If your paying 300 for a Nic you are getting robbed, I can buy brand new x520's cheaper, and the Ol server pulls can be had for around the 100-150 mark.

 

whats the tx rate of consumer pci express slots? there was talk of gigabit being challenged by bandwith of the host mainboard. Ditto for usb3 - "USB 3.0 (also known as SuperSpeed USB) has a maximum bandwidth rate of 5 gbps "

i'd guess you'd need 4x pci e before it makes sense. "The PCIe 2.0 standard doubles the transfer rate compared with PCIe 1.0 to 5 GT/s and the per-lane throughput rises from 250 MB/s to 500 MB/s."

so standard pci-e is 250 MB/sec per lane 4x would be 1GB.