Flain's Journey to 10GBit infiniband on the cheap (Opensolaris ZFS and Win7) - Page 5

flain · 25th March 2011, 12:56 AM

Quote:

Originally Posted by DJ_Datte

Open the task manager in windows, if you are hitting 25% cpu usage while running the bench, you are capping it.

ah ok well i am - i'm hitting 35% consistently during the test

also heres the results of a one way iperf (rather than dual)

Code:

[  3] local 10.0.0.2 port 50439 connected with 10.0.0.1 port 5001
^C[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-431.5 sec  72.1 GBytes  1.43 Gbits/sec

Could the subnet manager be using the CPU? Its running on the windows box.

DJ_Datte · 25th March 2011, 7:38 AM

Nah, the SM should just be a service that hooks things up. After your cards get link, you can (to test) actually disable the SM service, and notice that everything still works.

But the ipoib protocol translating Ip to infiniband traffic most def uses cpu hardcore.

flain · 25th March 2011, 10:09 AM

Quote:

Originally Posted by DJ_Datte

On Linux and Solaris, Connected Mode is default (65520 MTU), on Windows, Connected Mode is disabled by default and needs to be enabled on the interface settings.

Also, what CPUs are you using (actually, how many cores) Because IPoIB is not threaded on the windows end for sure, and I know I capped it on a 3.5GHz i7 when doing some light benchmarking and it turned out to be the chokepoint.

So a weaker CPU would produce lower results. And a single core on a i7-970 would still look like only ~8 CPU usage in any measurement software.

Quote:

Originally Posted by flain

Just changed the windows end to connected mode, not a lot of difference. Both now have a MTU of 65520.

The windows machine only has a Q8200 (2.33ghz quad core). The server is a core i7-970.

How can i tell if i'm capping out due to CPU though? Surly the measurement software should show it?

Hmm interestingly on the openfabrics download page they list connected mode as comming in version 3. I was able to select it in the adapter properties in windows 7. Now i'm wondering if it actually did anything or not.

From the site:

Quote:

OFED for Windows (WinOFED) Roadmap
-----------------------------------

[ Last updated: 03-09-2011 ]

WinOFED 2.3 released December 28, 2010

OFED for Windows 3.0 release target Q3'11; freeze Q2'11
ROCE support
NetworkDirect v2 provider
OpenSM 3.3.9 (vendor umad)
IPoIB CM (Connected Mode) support.
DAT/DAPL 2.0.35

http://www.openfabrics.org/downloads...ws/Roadmap.txt

Also if i can't get better performance i wonder if these work in solais 11 - http://cgi.ebay.com/Intel-10-Gigabit...item4156edd04a $250 for 10gbase-t wow

DJ_Datte · 26th March 2011, 8:20 AM

They are likely to work. I am really surprised with Solaris H/W compatibility, today I stuck a brand new (and it's a fairly "new" product) Intel i340-T4 into the Solaris box and it just accepted it and added 4 new ports that I could aggregate into a 4GB Link (The i340-T4 is a bleeding edge Intel nic with all kind of fancy offloading, and has 4 1GB RJ45 ports).

flain · 1st April 2011, 2:53 PM

Update:

The ebay seller is getting me a quote on international postage. So far hes checked FedEx and UPS which is about $110-$130. Hes checking USPS for me too.

If anyone else is interested let me know and i can ask if he can post more. He has 7 left according to his auction.

These are intel 10GBase-T cards - 10gbit ethernet with rj45/cat6 as the media

. $250 USD each is cheeeap.

doodz · 1st April 2011, 3:10 PM

$110-$130 to send taht small card?

Im pretty sure you can send it in a USPS flat rate large box for $60?

flain · 1st April 2011, 3:16 PM

Quote:

Originally Posted by doodz

$110-$130 to send taht small card?

Im pretty sure you can send it in a USPS flat rate large box for $60?

Yeah hes going to go check USPS for me. Also i must put a disclaimer - i don't know if these work in solaris or not. I'm taking a gamble on them. Win7 is a pretty safe bet though.

doodz · 1st April 2011, 3:29 PM

Quote:

Originally Posted by flain

Yeah hes going to go check USPS for me. Also i must put a disclaimer - i don't know if these work in solaris or not. I'm taking a gamble on them. Win7 is a pretty safe bet though.

Im pretty sure, 99% sure that USPS will defaintley fit those cards, @ $60 delivery. I sent back a box with stuff in it and it was $60. Box is pretty large too.

If it's one card you might get away with fitting it into a Small Flat Rate Box and would cost like $30

http://www.usps.com/prices/priority-mail-prices.htm

davros123 · 1st April 2011, 5:39 PM

Muuust reeesist...can feel myself weakening...must say no to 10GB.

Usps should be no more than 30 for 4 of the cards.

flain · 4th April 2011, 7:29 AM

He updated the auction with postage to AU now

Just ordered my two!

akashra · 4th April 2011, 8:47 AM

Infiniband? For home storage?

WTF and why!? I'm struggling to see the need to transfer an entire 42minute 720p mkv in to memory in under a second - that's the equivalent bandwidth we're talking about here :P

elvis · 4th April 2011, 8:53 AM

Quote:

Originally Posted by akashra

Infiniband? For home storage?

WTF and why!? I'm struggling to see the need to transfer an entire 42minute 720p mkv in to memory in under a second - that's the equivalent bandwidth we're talking about here :P

You know that some people do crazy shit, like work from home, right?

This thread is of great interest to a number of small business user mates of mine, all of whom have very high bandwidth requirements, but with bugger all budget.

Sephir0th · 4th April 2011, 9:29 AM

Quote:

Originally Posted by akashra

Infiniband? For home storage?

WTF and why!? I'm struggling to see the need to transfer an entire 42minute 720p mkv in to memory in under a second - that's the equivalent bandwidth we're talking about here :P

as said before, why not? why do people buy a ferrari when a corolla does the same job?

also interested in results, play with some pretty high end gear at work for 10G would like to see it done on the cheap

davros123 · 4th April 2011, 9:36 AM

Fast access to a large amount of high speed (>600MB/s), highly redundant and integral storage ...who would want that at home? Crazy I tell you! Madness.

One might ask why you need a faster nas too

seriously though...I already have a (solaris 11 Express/zfs) server at home that has 20TB of very fast (>600MB/s) and highly redundant and resilient storage. I personally want to be able to access my nas at (close to) native speeds of 600MB/s from my main/games PC. Small/light main PC...beastly fast and massive storage array. Nice.

Snuffy22 · 4th April 2011, 9:41 AM

if i had more disposable income........ i would be purchasing 2 of those cards.... but i just cant justify it

1st April 2011, 3:10 PM	#66
doodz Member Join Date: Nov 2002 Location: Sydney Posts: 4,384	$110-$130 to send taht small card? Im pretty sure you can send it in a USPS flat rate large box for $60? __________________ KEEP LEFT!! \| My Japan 2008 Blog \| Greatest Dead RAM Thread \| My R35 GTR Build Blog My EOC Folding Stats Norco 4220 20 HDD - 28.4TB raidz2 ZFS Server

4th April 2011, 8:47 AM	#71
akashra Member Join Date: Apr 2003 Location: Melbourne, AU Posts: 3,324	Infiniband? For home storage? WTF and why!? I'm struggling to see the need to transfer an entire 42minute 720p mkv in to memory in under a second - that's the equivalent bandwidth we're talking about here :P __________________ Punctuation is the difference between "Let's eat, grandma!" and "Let's eat grandma!". Building a Metricon Delta 21 in Craigieburn: http://homeofzero.blogspot.com/

25th March 2011, 7:38 AM	#62
DJ_Datte Member Join Date: Mar 2011 Posts: 23	Nah, the SM should just be a service that hooks things up. After your cards get link, you can (to test) actually disable the SM service, and notice that everything still works. But the ipoib protocol translating Ip to infiniband traffic most def uses cpu hardcore.

26th March 2011, 8:20 AM	#64
DJ_Datte Member Join Date: Mar 2011 Posts: 23	They are likely to work. I am really surprised with Solaris H/W compatibility, today I stuck a brand new (and it's a fairly "new" product) Intel i340-T4 into the Solaris box and it just accepted it and added 4 new ports that I could aggregate into a 4GB Link (The i340-T4 is a bleeding edge Intel nic with all kind of fancy offloading, and has 4 1GB RJ45 ports).

1st April 2011, 2:53 PM	#65
flain Member Join Date: Oct 2005 Posts: 1,789	Update: The ebay seller is getting me a quote on international postage. So far hes checked FedEx and UPS which is about $110-$130. Hes checking USPS for me too. If anyone else is interested let me know and i can ask if he can post more. He has 7 left according to his auction. These are intel 10GBase-T cards - 10gbit ethernet with rj45/cat6 as the media . $250 USD each is cheeeap.

4th April 2011, 7:29 AM	#70
flain Member Join Date: Oct 2005 Posts: 1,789	He updated the auction with postage to AU now Just ordered my two!

4th April 2011, 9:41 AM	#75
Snuffy22 Member Join Date: Sep 2004 Location: Brisbane Posts: 624	if i had more disposable income........ i would be purchasing 2 of those cards.... but i just cant justify it