Flain's Journey to 10GBit infiniband on the cheap (Opensolaris ZFS and Win7) - Page 13

Hive · 24th October 2012, 3:08 PM

Quote:

Originally Posted by flain

nice, ebay i assume? Also fibre or copper sfps for the 4 10gs?

The 48 ports are RJ45, the 10G's are CX-4 copper

.

CM96 · 24th October 2012, 3:30 PM

Been watching this thread for a while now and enjoying it thoroughly. Good job Flain

---

Edit:
Item location:
Suwanee - Atlanta Area - Georgia, United States

Ships to:
United States

Maybe you can convince the seller to send it here or use a freight forwarding service.

Also searching around and just in case someone wants it, there's (one) 10GbE RJ-45 card for cheap on fleabay - http://www.ebay.com/itm/Dell-Intel-1...50877388517%26

I personally would've snapped it up in a second if I actually had a need for 10G

ewok85 · 24th October 2012, 8:32 PM

Quote:

Originally Posted by yanman

Interested to see the price of copper 10GbE adapters (second hand) by next year..

10GbE switches will still be far too expensive for my wallet, so I don't care

Unbreakable · 24th October 2012, 8:33 PM

Quote:

Originally Posted by ewok85

10GbE switches will still be far too expensive for my wallet, so I don't care

You can get SFP equipped switches for a fair price (seeing some 24 RJ45 w/ 4 SFP) for <250 a pop

Its a serious thought if your switch is reasonably close to your SFP equipped gear.

mogran · 25th October 2012, 4:56 PM

Hi guys,

Been looking into this and have been ordering a bunch of parts. All coming in now so should be able to let everyone know how I go connecting 4 machines up via 10gb links (10gbe from ZFS server to ESXi & windows hosts) & infiniband for connection to a poweredge 2850 & ZFS box. CX4 10gbe pci-x cards are EXPENSIVE so I went with infiniband. Shame I'll have to either recompile freebsd with OFED support but thinking about switching to CentOS with ZFSonLinux.

Anyhow here is the cheapets 10gbe switches and cards i've found:

$400 (delivered) 10GBE switch with 4 CX4 ports:

Woven Systems TRX 100 - 4x 10GBE CX4 + 48 port 1gbe ports. Woven doens't exist anymore. This switch supports openflow and is supported by the indigo project if you wanted up to date OS. Can't find manual for 'em though, only data sheet:

http://www.ebay.com.au/itm/WOVEN-FOR...ht_2655wt_1167

Cheapest 10gbe cards (driver for pretty much every OS) I have found:

$75 US per card & shipping @ $15:

http://www.ebay.com/itm/MYRICOM-10GB...:X:RTQ:US:1123

Note: I highly recommend this seller, very nice, one card was faulty and he was willing to cross ship.

Regards,

Morgan

Hive · 25th October 2012, 5:05 PM

Quote:

Originally Posted by mogran

Hi guys,

Been looking into this and have been ordering a bunch of parts. All coming in now so should be able to let everyone know how I go connecting 4 machines up via 10gb links (10gbe from ZFS server to ESXi & windows hosts) & infiniband for connection to a poweredge 2850 & ZFS box. CX4 10gbe pci-x cards are EXPENSIVE so I went with infiniband. Shame I'll have to either recompile freebsd with OFED support but thinking about switching to CentOS with ZFSonLinux.

Anyhow here is the cheapets 10gbe switches and cards i've found:

$400 (delivered) 10GBE switch with 4 CX4 ports:

Woven Systems TRX 100 - 4x 10GBE CX4 + 48 port 1gbe ports. Woven doens't exist anymore. This switch supports openflow and is supported by the indigo project if you wanted up to date OS. Can't find manual for 'em though, only data sheet:

http://www.ebay.com.au/itm/WOVEN-FOR...ht_2655wt_1167

Cheapest 10gbe cards (driver for pretty much every OS) I have found:

$75 US per card & shipping @ $15:

http://www.ebay.com/itm/MYRICOM-10GB...:X:RTQ:US:1123

Note: I highly recommend this seller, very nice, one card was faulty and he was willing to cross ship.

Regards,

Morgan

Exact same gear i used

mogran · 25th October 2012, 6:03 PM

Hi Hive,

I'm really interested in those switches. I don't have one but have considered it.

How easy is it to use (CLI & web interface)? Did you replace VxWorks? Any problems?

Regards,

Morgan

Hive · 25th October 2012, 6:10 PM

Quote:

Originally Posted by mogran

Hi Hive,

I'm really interested in those switches. I don't have one but have considered it.

How easy is it to use (CLI & web interface)? Did you replace VxWorks? Any problems?

Regards,

Morgan

It's fasttrack switching... brocade gear with broadcom chips (the switch)

I just reset it, did the standard CLI setup, setup ssh etc, en passwords and then you can access the web managment.

Only thing is the stock fans are LOUD, jigsaw a 120mm hole and put in a fan of your choice, much better.

I get an easy ~660MB/s sustained with windows transfers

young_einstein · 23rd December 2012, 11:16 AM

There's some (used) Infiniband switches going pretty cheaply on eBay at the moment.

http://www.ebay.com.au/itm/261131717543

Can anybody see a reason *NOT* to buy one of these?

Pantagruel · 27th January 2013, 10:38 PM

The good people at ZFSbuild.com have run some comparative benchmarks.

Zfsbuild2012-nexenta-vs-freenas-vs-zfsguru/

Seeing these results and the fact that setting up IB under Nexenta/etc is 'quite' easy, it seems like a very viable option.

Sadly ZFSGuru doesn't cut it at their test (most likely will need some tuning), but it will be nice to get the data on how they setup their IB connection using ZFSguru (FreeBSD)

davros123 · 28th January 2013, 12:41 AM

nice. will give that site a good read. Thanks.

I so want to try some fast network gear. time and money prevents the fun stuff !

MnM · 29th January 2013, 8:15 PM

Looking to get some of these... time to hit eBay and hopefully get some cheap stuff..

Snuffy22 · 8th March 2013, 8:47 AM

I will just leave this here :P

Broadcom W1GCR 57810S DP 10Gb BASE-T Server Adapter
http://www.graysonline.com/lot/0005-...server-adapter

davros123 · 8th March 2013, 6:30 PM

hmmm.so, what to pay?

How do they perform user Solaris? do you know if it's plug and play under S11E?

23rd December 2012, 11:16 AM	#189
young_einstein Member Join Date: Jan 2006 Location: Caroline Springs, VIC Posts: 2,526	There's some (used) Infiniband switches going pretty cheaply on eBay at the moment. http://www.ebay.com.au/itm/261131717543 Can anybody see a reason NOT to buy one of these? __________________ OCAU Camera Club Member #275 For Sale: Sigma 100-300 f/4 APO DG + TS-41 tripod mount.

27th January 2013, 10:38 PM	#190
Pantagruel Member Join Date: Mar 2012 Posts: 6	ZFSbuild.com benchmarks The good people at ZFSbuild.com have run some comparative benchmarks. Zfsbuild2012-nexenta-vs-freenas-vs-zfsguru/ Seeing these results and the fact that setting up IB under Nexenta/etc is 'quite' easy, it seems like a very viable option. Sadly ZFSGuru doesn't cut it at their test (most likely will need some tuning), but it will be nice to get the data on how they setup their IB connection using ZFSguru (FreeBSD) Last edited by Pantagruel; 28th January 2013 at 12:04 AM.

29th January 2013, 8:15 PM	#192
MnM Member Join Date: Jun 2001 Posts: 490	Looking to get some of these... time to hit eBay and hopefully get some cheap stuff.. __________________ -==MnM==-

25th October 2012, 4:56 PM	#185
mogran Member Join Date: Apr 2003 Location: Brisbane Posts: 35	Hi guys, Been looking into this and have been ordering a bunch of parts. All coming in now so should be able to let everyone know how I go connecting 4 machines up via 10gb links (10gbe from ZFS server to ESXi & windows hosts) & infiniband for connection to a poweredge 2850 & ZFS box. CX4 10gbe pci-x cards are EXPENSIVE so I went with infiniband. Shame I'll have to either recompile freebsd with OFED support but thinking about switching to CentOS with ZFSonLinux. Anyhow here is the cheapets 10gbe switches and cards i've found: $400 (delivered) 10GBE switch with 4 CX4 ports: Woven Systems TRX 100 - 4x 10GBE CX4 + 48 port 1gbe ports. Woven doens't exist anymore. This switch supports openflow and is supported by the indigo project if you wanted up to date OS. Can't find manual for 'em though, only data sheet: http://www.ebay.com.au/itm/WOVEN-FOR...ht_2655wt_1167 Cheapest 10gbe cards (driver for pretty much every OS) I have found: $75 US per card & shipping @ $15: http://www.ebay.com/itm/MYRICOM-10GB...:X:RTQ:US:1123 Note: I highly recommend this seller, very nice, one card was faulty and he was willing to cross ship. Regards, Morgan

25th October 2012, 6:03 PM	#187
mogran Member Join Date: Apr 2003 Location: Brisbane Posts: 35	Hi Hive, I'm really interested in those switches. I don't have one but have considered it. How easy is it to use (CLI & web interface)? Did you replace VxWorks? Any problems? Regards, Morgan

8th March 2013, 8:47 AM	#193
Snuffy22 Member Join Date: Sep 2004 Location: Brisbane Posts: 624	I will just leave this here :P Broadcom W1GCR 57810S DP 10Gb BASE-T Server Adapter http://www.graysonline.com/lot/0005-...server-adapter