Flain's Journey to 10GBit infiniband on the cheap (Opensolaris ZFS and Win7)

[LostBenji]

10GbE cards are getting cheaper and so is switching if you hunt around. One single connection beats the shit out of bonding smaller 1GbE's

[Superchargedftw]

Originally Posted by LostBenji

10GbE cards are getting cheaper and so is switching if you hunt around. One single connection beats the shit out of bonding smaller 1GbE's

Copper cards are still exxy, SPF, CX-4 and direct fibre cards are über cheap ATM.

[aza2001]

Where are they cheap? Or I guess the better question is what do you class as cheap?

[Superchargedftw]

Originally Posted by aza2001

Where are they cheap? Or I guess the better question is what do you class as cheap?

Copper seems to be going for 350-400 on fleabay, SPF anywhere from 180-300. A stack of 12 went on Greys for 239 the other day

[cvidler]

Originally Posted by Superchargedftw

Copper seems to be going for 350-400 on fleabay, SPF anywhere from 180-300. A stack of 12 went on Greys for 239 the other day

I got my two Dell branded Intel 'AT' (copper) cards, for US$200 each + shipping. That's what I consider cheap (enough) to jump in.

Switches are still pricey if you want more than the 1 or 2 you may get as uplink ports on a gig switch.

[elvis]

Originally Posted by cvidler

I picked up a pair of Intel 10G AT cards (10G-BASET).

Easily beat my 2x1G bonded connection (which would max out at ~160MB/s) with ~300MB/s (which is about what one of the arrays can do - old PCIX controller)

iperf testing has come up to ~7gbps with 4 concurrent threads. (note one end is a ESXi host, so that's 7gbps into a VM)

That's pretty horrendous. I can easily hit 9.6gbps with a single threaded iperf on KVM using the same sort of test.

[cvidler]

Originally Posted by elvis

That's pretty horrendous. I can easily hit 9.6gbps with a single threaded iperf on KVM using the same sort of test.

Tweak the settings a bit (bigger TCP window and 9000B MSS for example) and....

[328]  0.0-600.0 sec   638 GBytes  9.14 Gbits/sec

(10 minute sustained throughput)

[elvis]

Originally Posted by cvidler

Tweak the settings a bit (bigger TCP window and 9000B MSS for example) and....

[328]  0.0-600.0 sec   638 GBytes  9.14 Gbits/sec

(10 minute sustained throughput)

OK, that's better. Even with default TCP window and 1500b MTU/MSS you can get 9.6gbps sustained on KVM, but at least ESXi can be tweaked to get there eventually.

[shane41]

MELLANOX INFINIHOST MHEA28-XTC ??

I see these on ebay.com, but prices are like $100+

Is $50 each reasonable from the US & $40 4day shipping

[nowlan]

Try a search on completed listing to get a feel of current prices.

Also the XTC doesnt have ram, so had issues with solaris or something. Mentioned in one of the threads here.

[Stanza]

Just was looking up the specs on the HP NC522SFP Dual Port 10Gbe Adapters

http://h20000.www2.hp.com/bizsupport...ctID=c01720419

Specs say,

NOTE: A minimum of two gigabyte (2 GB) of server memory is required per each NC522SFP adapter.

Sheeesh.

Not that it matters to tho with bucket loads of RAM.

But still, What are the other cards people here are using..? requirements RAM wise?

.

[cvidler]

Originally Posted by Stanza

Just was looking up the specs on the HP NC522SFP Dual Port 10Gbe Adapters

http://h20000.www2.hp.com/bizsupport...ctID=c01720419

Specs say,

NOTE: A minimum of two gigabyte (2 GB) of server memory is required per each NC522SFP adapter.

Sheeesh.

Not that it matters to tho with bucket loads of RAM.

But still, What are the other cards people here are using..? requirements RAM wise?

.

They say that so you don't go sticking the card in a box with 512mb of RAM and then whinge that you can only get 100MBps through it.

The memory if for send/receive buffers. Think at full tilt (both ports) it'll do 2GB/sec, so you want some good sized buffers to feed it.

[ewok85]

After some head scratching as to why I can't use my shiny new Infiniband card in Solaris, I found this: http://www.oracle.com/technetwork/sy...11-392732.html

The tavor driver that supports older InfiniBand HCAs (Host Channel Adapters) will be removed and not supported. No functionality for InfiniBand on these HCAs will be supported in the future, including firmware upgrade by using the fwflash utility.

Wonderful

Well I feel as sharp as a bucket of jello. Opened the "device driver utility", and that's that - it works

Now to work out how to get the subnet manager thing working

Trying to decode this, wish the guy would be useful and actually just tell how he did it. Sounds like he just took the package included in Solaris and compiled opensm from what was already there.
http://syoyo.wordpress.com/2011/12/1...laris11-works/

Edit: Well I have to shelve this until I move my ZFS server over to something that will run OpenSM, or until I buy a switch. Ugh

[young_einstein]

I realise that I might be a little bit late to this party, but I'm looking to jump onto the infiniband-wagon as well (see what I did there ).

My original plan was to try and connect a ZFS box into my ESXi host and get some SCSI goodness happening between them, however my reading so far has revealed that there's still no SRP drivers/support for ESXi 5, and IPoIB seems to suffer from pretty woeful performance problems (by comparison).

There is talk that a driver package with SRP support is being worked on, and might even be available by the end of the year, although according some people earlier target dates for its release have been and gone several times already.

So ... onto plan B (for now).

I've got a second ZFS box that's not doing much at the moment, so I'm thinking that it might be a good learning exercise to just link them both up so that I can shoot zfs send/recv snapshots between them all day long. :P

Now, keeping in mind that I'm coming into this knowing (almost) nothing about Infiniband, I'm just wondering if anybody might have any specific tips/tricks/advice based on their own experiences that might make my learning curve a little bit less painful?

Both my ZFS boxes are running Solaris 11 Express, so I've picked up on the fact that I need IB HCA's that have onboard memory.

I know that I need to get a subnet manager up and running on one of the boxes, but the exact process of doing that in Solaris is still a bit of an unknown quantity at the moment (I haven't really made it that far yet).

What else am I going to need to do in order to make this little venture of mine successful?

[DJ_Datte]

Originally Posted by young_einstein

I realise that I might be a little bit late to this party, but I'm looking to jump onto the infiniband-wagon as well (see what I did there ).

My original plan was to try and connect a ZFS box into my ESXi host and get some SCSI goodness happening between them, however my reading so far has revealed that there's still no SRP drivers/support for ESXi 5, and IPoIB seems to suffer from pretty woeful performance problems (by comparison).

There is talk that a driver package with SRP support is being worked on, and might even be available by the end of the year, although according some people earlier target dates for its release have been and gone several times already.

So ... onto plan B (for now).

I've got a second ZFS box that's not doing much at the moment, so I'm thinking that it might be a good learning exercise to just link them both up so that I can shoot zfs send/recv snapshots between them all day long. :P

Now, keeping in mind that I'm coming into this knowing (almost) nothing about Infiniband, I'm just wondering if anybody might have any specific tips/tricks/advice based on their own experiences that might make my learning curve a little bit less painful?

Both my ZFS boxes are running Solaris 11 Express, so I've picked up on the fact that I need IB HCA's that have onboard memory.

I know that I need to get a subnet manager up and running on one of the boxes, but the exact process of doing that in Solaris is still a bit of an unknown quantity at the moment (I haven't really made it that far yet).

What else am I going to need to do in order to make this little venture of mine successful?

Hi,

I think that there are no subnet managers for Solaris. SUN had one in house at some point before being bought up by Oracle, but they decided that they won't release it. As a enterprise class solution, they expect you to use a infiniband switch, and that one should/would/could have a SM manager built into its firmware.