When is consumer 10GbE going to happen?

Discussion in 'Networking, Telephony & Internet' started by Smokin Whale, Sep 21, 2014.

  1. Aetherone

    Aetherone Member

    Joined:
    Jan 15, 2002
    Messages:
    8,591
    Location:
    Adelaide, SA
  2. DanglingPointer

    DanglingPointer Member

    Joined:
    Feb 19, 2007
    Messages:
    153
    Bought two of them from MSY to install in two old Ivy-Bridge and Sandy-Bridge non-pro-servers for direct crossover.
    Unfortunately it only worked in the Ivy-Bridge which has a Z68XP mobo.
    The Sandy-Bridge (File backup server) has an old H61M mobo with a single PCIEx16 which doesn't work with the NIC (x4). It looks like the single PCIEx16 can only take graphics cards which sux. All the other slots are PCIEx1 and too small for the NIC. Nothing in the BIOS seems to work.

    So basically a lot of money spent on the two cards for a setup which can't work until I replace the very old H61M mobo with a newer but still old sandy-era/ivy-era mobo.
     
  3. Hater

    Hater Member

    Joined:
    Nov 19, 2012
    Messages:
    3,708
    Location:
    Canberra
    new mac mini comes with 10GB Ethernet

    that's fairly consumer
     
  4. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,930
    Hmmm...10G googness on a budget!

    [​IMG] [​IMG] [​IMG]

    TP-Link T1700G-28TQ JetStream 24-Port Gigabit Stackable Smart Switch 4 10GE SFP+ 28 port Switch for $341AUD delivered.
    SFP+ modules from here ($15USD offer accepted - you can also get them from Aliexpress for $14USD inc. postage)
    Edit: I needed some Intel module for Intel X520 cards and for them for $22 from FS.copm http://www.fs.com/products/36431.html
    3m LC-LC Optical cables from 4-Cabling @$18ea (could get cheaper from aliexpress...but I like 4 Cabling)
    I already have the dual port NIC's, but these babies are pretty damn cheap @39USDpp

    All in all a damn good way to link up my 3 servers and games PC with 10G and still keep 24 ports. Like.


    I guess my faithful HP 1810-24G is about to be for sale :)
     
    Last edited: Feb 3, 2019
    markpunti and MrSnuffy like this.
  5. g00nster

    g00nster Member

    Joined:
    Sep 10, 2004
    Messages:
    348
    Location:
    Melbourne
  6. davros123

    davros123 Member

    Joined:
    Jun 18, 2008
    Messages:
    2,930
    Nice...though, if the following is true, may be some issues bridging it across to 1Gig network...
     
  7. Hive

    Hive Member

    Joined:
    Jul 8, 2010
    Messages:
    5,406
    Location:
    Some place far away
    It will be fixed in future releases of rOS
     
  8. ae00711

    ae00711 Member

    Joined:
    Apr 9, 2013
    Messages:
    1,396
    I actually connected my first ever 10GbE today:- PC 2 PC direct, under Windows Server 2019 (both rigs)... no fiddling with settings, just OOB as WS19 default... did a quick, dirty transfer 'test', a VM, SSD 2 SSD. Speeds fluctuated wildly, probably due to different file sizes. Max I saw was 800MB/s, lowest 80MB/s

    I'm already very pleased with that! :D
     
  9. DanglingPointer

    DanglingPointer Member

    Joined:
    Feb 19, 2007
    Messages:
    153
    That's great, my primary use case is file-transfer (back-up and restore) and NFS async read between the same two for thousands of large files (2GB to 20GB).
    All the files are in 18TB BTRFS RAID0 (in one) and 14TB BTRFS RAID5 (in another). So 1Gbps is super slow! Can't wait till the I get another compatible mobo for the Asus XG-C100C!
     
    Last edited: Jan 25, 2019
  10. ae00711

    ae00711 Member

    Joined:
    Apr 9, 2013
    Messages:
    1,396
    ? compat mobo for the asus? is that thing picky?
    cheap mellanox won't do the trick?
     
  11. ex4n

    ex4n Member

    Joined:
    Oct 5, 2011
    Messages:
    2,184
    Location:
    Perth
    It's getting pretty affordable these days, i just started using a mikrotik CRS (CRS317-1G-16S+RM) with 16 sfp+ and 1 gb ethernet port and got it last month for about $450 I think, so far pretty happy with it.
    sfp+ cards are getting cheap too, single's about $35-40 and duals about $70 on ebay, probably cheaper if you find a good deal.
     
  12. DanglingPointer

    DanglingPointer Member

    Joined:
    Feb 19, 2007
    Messages:
    153
    More detail on post 324 above.

    I don't think it is the card but the really old H61 mobo. It has a single slot that can accommodate a x4 length card which is PCIEx16 graphics slot. Unfortunately the mobo only seems to accept graphics cards on that slot. All the other slots are PCIEx1. So I have to swap out the old mobo with a newer albeit still old "used" mobo from somewhere. I'm not looking at building a whole new rig when everything just works except for the 3rd party NIC. So a newer used mobo will do.
     
  13. ae00711

    ae00711 Member

    Joined:
    Apr 9, 2013
    Messages:
    1,396
    oo dang that sucks
     
  14. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    37,454
    Location:
    Brisbane
    You're brave.
     
    NSanity, Hive, ae00711 and 1 other person like this.
  15. ae00711

    ae00711 Member

    Joined:
    Apr 9, 2013
    Messages:
    1,396
    sweet mother of abraham lincoln

    my previous test was flawed.. I did it again, it was more a steady ~250-300 MB/s than anything. That makes sense, as both SSD in each rig are attached thru SATA2 (3Gbps) controllers, so obviously that's the max I'll get

    Just set up a 24GB RAMdisk on main rig, DAC'd to another rig with a Samsung XP941 (PCIE 256GB SSD)... again, no settings tuning, same OS's (Windows Server 2019)... the majority of the transfer was done @ 1.09GB/s! :D I transferred a single 11GB video file. There was a dip down to 0, small pause, at about..mm.. 60%, then it picked right up again.
     
  16. DanglingPointer

    DanglingPointer Member

    Joined:
    Feb 19, 2007
    Messages:
    153
    Been using BTRFS RAID5 since 2014 and has worked fantastically!
    In fact it saved my bacon! I had a disk go bad and BTRFS RAID5 kept on running until I replaced it and rebalanced!
    I've also grown the array by another disk.

    In all those 5 years, I've suffered blackouts crashes, you name it! I just did a BTRFS scrub and everything was sweet!

    If you compare mdadm stable RAID5 features compared to BTRFS RAID5 you'll see objectively that BTRFS RAID5 has more stable features! The warnings given in the BTRFS wikis for RAID5/6 not being fully ready are for the envisioned functionality that mdadm will never have. The only thing close to the envisioned state of BTRFS is ZFS.
     
  17. evilasdeath

    evilasdeath Member

    Joined:
    Jul 24, 2004
    Messages:
    4,861
    brave indeed, this should probably go to the storage thread, indeed btfrs is likely closely aligned to standard raid5 which has write hole issues, the problem is you dont know they exist and your scrub isnt going to know they are there either, thats the whole issue BTFRS should know!

    Unless you independantly checksum all your data how do you know you dont have data rot?
     
  18. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    37,454
    Location:
    Brisbane
    BtrFS RAID5 is currently marked as unstable by its own developers. If you want "objectivity", start with the official documentation.

    Everyone else reading this: BtrFS RAID5 (and RAID6) is being improved, but is still not production ready. Until the developers mark it stable, please don't use it outside of test data you don't care about.

    BtrFS RAID 0/1/10 are all considered stable.

    Correct. Even with recent code improvements, BtrFS RAID5/6 still suffers the write hole problem. To date, only ZFS has solved this.
     
    Last edited: Jan 26, 2019
  19. DanglingPointer

    DanglingPointer Member

    Joined:
    Feb 19, 2007
    Messages:
    153
    I'm using it for a bunch of servers in a office lab. Not production.

    What is always failed to be mentioned, like in your statement above is that BTRFS RAID5/6 has a write hole problem just like all other software RAID5s out there like mdadm RAID5, and only ZFS has it sorted.

    So if you're using mdadm RAID5/6 it is not stable for production.

    As already stated previously the conversation belongs in a different thread, best not here to have these flame-fanboy wars.
     
  20. DanglingPointer

    DanglingPointer Member

    Joined:
    Feb 19, 2007
    Messages:
    153
    I'm well aware of if mate, and it is good that you mentioned that that standard RAID5 has the same issue, which many times fanboys out there fail to realise when they go RAID5/6 with mdadm or a cheap controller card or onboard motherboard RAID5.

    The data is also on another rig hence why I bought the two Asus 10gbe cards, to speed up backups and copies. These are just lab servers for tinkering around, dev work, research.
     

Share This Page

Advertisement: