Cheap 10G local networking

[ltd73]

Originally Posted by Herballizard

I still think I might go 10GBe copper over IB though

that would be 8Gbps with Infiniband and its 8b/10 encoding.

[Luke212]

Originally Posted by Hive

Yeah, Woven no longer exist.

Good news though, got it up to ~700MB/s with 15% usage of an i7 950 @ 4.2. Pretty good for windows eh?

[SUM] 25.0-26.0 sec  0.67 GBytes  5.73 Gbits/sec

Jumbo frames work on a Host<--->Host connection , now to get the switch playing...

Although the other node was a little shithouse, so with a faster end node may yield better Jperf results... the end was only running a Xeon X5355, not latest by any definition.

can you please explain how you got the cpu % down? was it 100% on one core? didnt think it would be multi threaded! was it because without jumbo frames, it had to send too many packets, which meant the cpu limited the rate?

also interesting, to stream uncompressed 1080p 60Hz across network you need 373MB/s. So it might be soon possible to make a distributed 3d ray tracer. install on all office computers and crowd around one pc after hours. hmm still just a dream...

[ltd73]

Originally Posted by Luke212

can you please explain how you got the cpu % down? was it 100% on one core? didnt think it would be multi threaded! was it because without jumbo frames, it had to send too many packets, which meant the cpu limited the rate?

probably a few reasons, but jumbo frames would be a big part of it.
9K frames vs 1.5K frames means a 6x reduction in # of interrupts/second and the system overall is far more efficient at DMAing 9KB memory transfers across the PCI bus than 1.5KB memory transfers.

more modern network adapters are even better than that as they do both TSO (TCP segmentation offload) and RSO (receive segmentation offload), which for e.g. a TCP window size of 128K means that they're typically doing far more efficient 128KB DMA requests across the PCI bus and the adapter is slicing/dicing that up into individual packets/frames without CPU involvement.
I think the OP's NIC is too old to be doing that or some of the more modern tricks (multi-queue tied to multiple cpu cores/threads) and various other optimizations.

Originally Posted by Luke212

also interesting, to stream uncompressed 1080p 60Hz across network you need 373MB/s. So it might be soon possible to make a distributed 3d ray tracer. install on all office computers and crowd around one pc after hours. hmm still just a dream...

you could do this already, quite trivially actualy, since the 'crowd sourced' content (N PCs) would presumably be N PCs. lets say N is 60 and each is generating 1fps, then they are each generating 373MB/s / 60 = 6.2MB/s.
they could be fastethernet (100Mbps) attached and still achieve that.....

[Luke212]

Originally Posted by ltd73

probably a few reasons, but jumbo frames would be a big part of it.
9K frames vs 1.5K frames means a 6x reduction in # of interrupts/second and the system overall is far more efficient at DMAing 9KB memory transfers across the PCI bus than 1.5KB memory transfers.

more modern network adapters are even better than that as they do both TSO (TCP segmentation offload) and RSO (receive segmentation offload), which for e.g. a TCP window size of 128K means that they're typically doing far more efficient 128KB DMA requests across the PCI bus and the adapter is slicing/dicing that up into individual packets/frames without CPU involvement.
I think the OP's NIC is too old to be doing that or some of the more modern tricks (multi-queue tied to multiple cpu cores/threads) and various other optimizations.



you could do this already, quite trivially actualy, since the 'crowd sourced' content (N PCs) would presumably be N PCs. lets say N is 60 and each is generating 1fps, then they are each generating 373MB/s / 60 = 6.2MB/s.
they could be fastethernet (100Mbps) attached and still achieve that.....

tnx for the replies. re the renderer trick, you would still need to aggregate the separate images on to one pc. this would require a single pc with 10Gbs. but as you point out, each slave pc only has to be 100Mbs. hmmm. i have my own ray tracer engine. I can probably do this.

sorry for distraction.