When is consumer 10GbE going to happen?

Discussion in 'Networking, Telephony & Internet' started by Smokin Whale, Sep 21, 2014.

  1. wwwww

    wwwww Member

    Joined:
    Aug 22, 2005
    Messages:
    5,191
    Location:
    Melbourne
    Well you can get 10GbE SFP+ cards for ~$60 now and used switches with 4 SFP+ ports for $200. That's pretty much consumer level pricing now.
     
  2. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    37,454
    Location:
    Brisbane
    1) It's $7K new with lifetime warranty for the switch I linked to. That's almost 1/10th the price of what it was a couple of years ago. Certainly a fraction of the price of current competitors we're evaluating.

    2) It's L3 Ethernet, not Infiniband. Our networking needs are moving lots of big files around (petabytes of them per day). Infiniband doesn't work for us, as that's a different use case.
     
  3. koopz

    koopz Member

    Joined:
    Dec 27, 2001
    Messages:
    2,034
    Location:
    Qld

    we don't do 10Gb over copper.

    we install OS1 fiber instead.
     
  4. fredhoon

    fredhoon Member

    Joined:
    Jun 27, 2003
    Messages:
    2,643
    Location:
    Brisbane
    I'm looking to get a couple of 10GBe NICs for direct connection between my NAS and PC via a 3m Twinax and eventually 10-15m of Twinax or 850nm FO. After a bit of reading around I have few quick questions for those that have trail blazed Consumer 10G...


    1. Anyone tried the "China eBay" Mellanox ConnectX-2 cards? I've seen a few reports from "Serve the Home" that they appear to work without issue, no "Counterfeit" & "Beware" stories were popping up when googling (could have been my poor search terms).

    2. If I go a more conservative route and buy server pulls, do different brands still have compatibility issues with SFP+ Twinax modules? Am I best to find two NICs from the same brand and will I need similar branded SFP/cables?

    3. From what I've read Optical SFP+ modules are still vendor roulette, any brands to look out for that generally offer better compatibility? Googling thus far suggests I'm best to find a re-branded Mellanox card that can be cross-flashed with Mellanox firmware.


    [edit] Found some very cheap IBM Virtual Fabric 49Y4250 / Emulex OneConnect OCe10102 dual 10GBe cards on ebay, however they may be more trouble than they're worth:
    https://virtualhobbit.com/2015/09/06/fixing-emulex-10gbe-on-esxi-5-5/
    https://forums.servethehome.com/ind...l-fabric-adapter-ii-x8-pci-e-slot.3491/page-2
    (also is Stanza still active on OCAU?)
     
    Last edited: Mar 27, 2018
  5. HobartTas

    HobartTas Member

    Joined:
    Jun 22, 2006
    Messages:
    821
    AFAIK your probably better off going for Mellanox cards and if you get those you definitely need their cables as well, probably the best bang for the buck are the QDR Infiniband cards because they are both 40 Gb Infiniband and also 10 Gb Ethernet and this gives you the flexibility to use either protocol, ideally you probably want HP cards 593412-001 as they are re-branded Mellanox MHQH29B-XTR cards, so buy either whichever is available usually they are like $50 - $100, if they come with short brackets you can buy the long brackets from the Mellanox store. The QSFP+ cables come in either fibre or copper and you can use either although the maximum length of the copper ones are like 7? metres in length and they usually pop up on Ebay cheap and also in bulk (like lots of 5/10/20) and as far as the 10-15M fibre one you also want its probably best to wait until a bargain also comes up on Ebay for that one, all of them have the optical transceiver unit already built in to both ends of the cable so its as simple as just plugging it directly into the card. If your feeling adventurous and want to also use Infiniband their switches are IS5022,IS5023,IS5024,IS5025 and I've seen them as low as USD $200, they are unmanaged so you will have to run the free OFED subnet manager software on one of your PC's to activate the IP addresses on the network.
     
    Last edited: Mar 27, 2018
    fredhoon likes this.
  6. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    3,346
    Location:
    Canberra
    I came back to this, to figure out the VPC capabilities and who's silicon it was inside the tin.

    I took a look at the Q&A section..... Fuck Me, the FS verified purchases all read chinglish. Astroturf is obvious, staying the hell away now.
     
  7. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,675
    Location:
    Canberra
    At 7k? given a comparable is probably 60 odd. Why not?
     
  8. waltermitty

    waltermitty Member

    Joined:
    Feb 19, 2016
    Messages:
    1,019
    Location:
    BRISBANE
    I've got 2 directly connected mellanox connectx 2 cards, mainline 4.15 linux kernel, 1gbps with iperf, I can't remember the aliexpress seller but I got 2 x cards and 2 x 1m DACs for < AUD$200, this was 6 months ago, I think there was a flood of mellanox cards on eBay at the time.
     
  9. Luke212

    Luke212 Member

    Joined:
    Feb 26, 2003
    Messages:
    9,653
    Location:
    Sydney
    Amazingly affordable 32x 40gbe! I’d be excited about that.

    And flexible for the 10gbe to workstations
     
  10. fredhoon

    fredhoon Member

    Joined:
    Jun 27, 2003
    Messages:
    2,643
    Location:
    Brisbane
    Were they new cards or server pulls?

    Is the "1gbps with iperf" a typo or are you saying these cards are counterfeit and won't perform at 10GBit speeds?
     
  11. waltermitty

    waltermitty Member

    Joined:
    Feb 19, 2016
    Messages:
    1,019
    Location:
    BRISBANE
    Brand spankers. 10gbit == 1 gbps
     
  12. fredhoon

    fredhoon Member

    Joined:
    Jun 27, 2003
    Messages:
    2,643
    Location:
    Brisbane
    Cheers, I assumed b = bit & B = Byte sorry.

    Upon reflection my assumption isn't even consistent with my own post referring to 10GBe as bit.
     
  13. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    12,311
    Location:
    Canberra
    people improperly/interchangeably using the SI suffixes make things difficult to understand.
     
  14. Hive

    Hive Member

    Joined:
    Jul 8, 2010
    Messages:
    5,406
    Location:
    Some place far away
  15. HobartTas

    HobartTas Member

    Joined:
    Jun 22, 2006
    Messages:
    821
    Hive likes this.
  16. Hive

    Hive Member

    Joined:
    Jul 8, 2010
    Messages:
    5,406
    Location:
    Some place far away
    Last edited: Apr 13, 2018
  17. fredhoon

    fredhoon Member

    Joined:
    Jun 27, 2003
    Messages:
    2,643
    Location:
    Brisbane
    Thanks for the detailed advice. I picked up 1x ConnectX-2 IB card from eBay and 2x ConnectX-3 cards from OCAU (for the same unit price as the eBay card!). Now to play around with cross-flashing the HP cards and sourcing compatible DAC cables for a reasonable price, although I'll probably just buy something new to save time and hassle.
     
  18. HobartTas

    HobartTas Member

    Joined:
    Jun 22, 2006
    Messages:
    821
    Yes, that's usually the setup you use although all the TCP/IP work is done by the CPU but with the modern processors of today the workload is not an issue like it was when people used to run Q6600/Q9450's on the old 10 Gb's CX4 copper cable Infiniband where the IPOIB CPU loading was reported to be around 80% on all cores when running at full 10 Gb's speed.

    If you want to use those QDR cards in Ethernet mode instead then its only 10Gb speed however you can turn on RDMA for those cards to offload the CPU work which is called RoCE (RDMA over Ethernet), however, those infiniband switches then become useless for this purpose because they don't do Ethernet and your choices are either a simple connection between PC and NAS or actually getting a 10GbE switch and since I'm not greatly familiar with fibre (so get someone else to confirm) I believe they are all SFP+ so you would need an adapter cable like this and try to use as much Melanox gear as you can as I've heard the Infiniband stuff doesn't play too well with other brands but since all their switches, cards and cables can be bought cheap this shouldn't be much of an issue. I recently bought an 18 port switch and I look forward to setting all this up next month at home when I'm on leave.

    Your welcome, however, perhaps also consider picking up a 36 port Infiniband switch for $200 as per my previous post.

    I've got a couple of weird questions for you if you would be kind enough to answer. I bought a Fibre Channel LTO6 tape drive real cheap probably because it was FC and not SAS and to use it I also got a Qlogic QLE2462 card for $25 and an LC to LC cable for $10 to back up my Solaris ZFS NAS, anyway my questions are unlike Linux since Solaris still retains IPFC would it be viable to just get a second QLE2462 card and use another Solaris workstation and network the two together with IPFC as apparently all I have to do hardware wise is reverse the two fibre cables at one card? My second question is if I didn't want to run Solaris as the base OS could I use the IPFC networking capabilities by having Solaris in a VM and what virtualization software would I use that would allow FC pass through? I note that 4 Gb FC switches are relatively inexpensive as well. So for effectively $60 you would have a 4 Gb connection yielding 500MB's (perhaps 400MB's if using 8b/10b encoding) unless people want to buy faster 8 Gb cards for around a hundred dollars each or even faster 16 Gb cards for several hundred.

    I wonder why Linux deprecated IPFC a while back as FC technology is old and and the software running it would have been well mature given its mainframe technology so its not like they would have to keep throwing intensive development at it.
     
  19. waltermitty

    waltermitty Member

    Joined:
    Feb 19, 2016
    Messages:
    1,019
    Location:
    BRISBANE
  20. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    12,311
    Location:
    Canberra
    from those photos, the card and even the packaging look 100% spot on with the real Intel one I have in front of me. Probably comes from the same factory that makes the Intel ones. Only thing that isn't right is the driver CD.

    OR

    they used a genuine card/package for the photos.

    for that money, if I needed some, I'd try a couple.
     

Share This Page

Advertisement: