[Worklog] Openfiler/ZFS/iSCSI 2.2 2Tb (iSCSI,SMB,ZFS) Project

Discussion in 'Storage & Backup' started by fad, Nov 12, 2007.

  1. fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,441
    Location:
    City, Canberra, Australia
    I have been doing alot of research on the different types of storage that can be built using various linux projects. I was thinking I would start a work log of the project. I am aiming for 2-4Tb of usable space.

    Here are a few of them.


    I have tested the above dists, and they are all quite good. They have different features and problems. The biggest problem I will have is getting HBA's to suit the different Linux operating systems.

    Openfiler

    This has really good iSCSI support.

    Supported SATA cards.

    SIL3114
    Onboard NV SATA ports NF4
    BC4852 (Doesnt seem to detect all drives, might be raid firmware like 3114)

    Freenas

    This has been around for a while. It does samba support really well. However it would be nice to have a bit more.

    ZFS on Solaris

    The "best" filesystem, that I can see. It features alot of really smart code and ideas. Would make building a machine cheaper, and the files more reliable, mabey faster


    Suggested Hardware Platforms

    Intel i845, Pentium 4, Onboard SATA,PATA
    Intel P965, C2D, SATA cards (ICH support seems lacking)
    AMD X2 NF4, DFI SLI, 4 onboard nv ports, 4 onboard SIL3114 ports.


    Suggested Raid/SATA HBA Cards

    • 3Ware/ACC
    • Acera
    • Promise
    • LSI Megaraid
    • Adaptec
    • SIL3114
      Syba 3114. Flashed to non raid.
    • Marvell


    Suggested Cases


    • External SATA Case
      Not really looking at this, but there is a multilane external enclosure
      http://cgi.ebay.com.au/4x-MULTILANE...oryZ3757QQrdZ1QQssPageNameZWD2VQQcmdZViewItem
    • Internal SATA Cage
      There was a few server cages that would fit a normal case, and provide hotswap rails.
      link
      There is the Coolermaster 4in3 device, which I already have a few of. link
      3Ware cage. This is a SAF-TE, which allows status lights. Unlike any of the above.
    • SATA Server Case
      Chenbro RM4146 4RU 16 Hotswap SATA Rackmount Case

    Here are a few things to look out for.



    Here is a sneek preview of a Openfiler I just got up and running.

    [​IMG]

    [​IMG]
     
    Last edited: Nov 13, 2007
  2. kbekus

    kbekus Member

    Joined:
    Aug 28, 2001
    Messages:
    1,206
    Location:
    Pascoe Vale, Melbourne
    this is great thanks Fad. I want to build a similar box and am looking at OpenFiler for it's iSCSI support. Let's see how you go.
     
  3. evilhomey

    evilhomey Member

    Joined:
    Jan 15, 2002
    Messages:
    208
    Location:
    Melbourne
    Looks good, have you tried using the iscsi feature on freenas? how does it compare to openfiler?

    I've recently resurrected a maxtor max attach 4300, running freenas. It's old hardware so it stays in the testing environment.
     
  4. HeXa

    HeXa Member

    Joined:
    Jul 7, 2001
    Messages:
    10,216
    Location:
    Canberra, ACT
    mmm... explains where the Samsungs are likely to be used :p

    what you using all the space for?
     
  5. OP
    OP
    fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,441
    Location:
    City, Canberra, Australia

    iSCSI storage for, ESX cluster, windows fs.

    Also testing for a bigger windows store. I am thinking that Windows 2003 is not really the best for a >2Tb array. The 250's will be for testing, before I buy some 750's 1000's. I will be looking to consolidate most of my storage from 200-300.

    iSCSI on Openfiler 2.2. Waiting for 2.3

    AMD X2 NF4, DFI SLI, 4 onboard nv ports, 4 onboard SIL3114 ports.

    Using 4x 320Gb Seagates SATA on SIL, 1x 80Gb Seagate SATA boot.

    NV raid off, SIL3114 off, All sata ports on.

    Current speeds on single Gbe interfaces are, 9-12Mb/sec.

    I configured three Gbe Intel cards on the iSCSI. They do redundant failover well. Need some MPIO software to increase speeds. Might try with 2003 Server.
     
    Last edited: Nov 13, 2007
  6. OP
    OP
    fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,441
    Location:
    City, Canberra, Australia
    Update:


    Just bought 4x 250Gb SATA disks. So now the test array will be 4x 320Gb 4x 250Gb 6x 200Gb. I have also ordered a Supermicro AOC-SAT2-MV8 card, Marvell chipset SATA II 8x SATA ports in PCI-X card.


    I also just ordered the Chenbro case, 3ware 16 port ML pci-e, 16x 750Gb SATA, with a C2D setup. Should be interesting to see the speed of this thing.

    Also those SIL3114 cards seem to be a bit flaky. Seems you can not buy good SIL3114 cards, I have it onboard on my DFI SLI board and its really good. I suppose I will have to buy some more.
     
  7. OP
    OP
    fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,441
    Location:
    City, Canberra, Australia
    Finally got all the disks into the machine. The hda drive is to boot the OS. To keep the raid drives free.
    At the moment I am running them in raid 0, total size 2.1Tb. Speed seems better than with the SIL3114 cards.

    At this stage for ease of use, the DFI SLI NF4 board seems really good. They work with the ide ports, which the Intel ICH 8-9 have trouble with, like the Asus P5B which has a onboard PATA chip which is not support, jmicron?

    I also have make a drive over the 2Tb limit, all I did was install SP1 on Windows 2003 server. Then shared it via samba. Openfiler however supports samba access. So mabey instead of a iscsi storage box and a Windows 2003 box, the openfiler is a better option in one. It seems pretty stable. I was testing iscsi r3, it however had issues booting 100% of the time.


    Edit Disk Type Description Size Label type Partitions
    /dev/hda IDE WDC WD2000JB-00DUA0 186.31 GB msdos 3 (view)
    /dev/sda SCSI ATA ST3320620AS 298.09 GB msdos 1 (view)
    /dev/sdb SCSI ATA ST3320620AS 298.09 GB msdos 1 (view)
    /dev/sdc SCSI ATA ST3320620AS 298.09 GB msdos 1 (view)
    /dev/sdd SCSI ATA ST3320620AS 298.09 GB msdos 1 (view)
    /dev/sde SCSI ATA SAMSUNG SP2504C 232.88 GB msdos 1 (view)
    /dev/sdf SCSI ATA SAMSUNG SP2504C 232.88 GB msdos 1 (view)
    /dev/sdg SCSI ATA SAMSUNG SP2504C 232.88 GB msdos 1 (view)
    /dev/sdh SCSI ATA SAMSUNG SP2504C 232.88 GB msdos 1 (view)
     
    Last edited: Nov 21, 2007
  8. aza2001

    aza2001 Member

    Joined:
    Sep 14, 2002
    Messages:
    2,016
    Location:
    Northmead
    Just so you are aware...the onboard silicon image is crap with raid5 cause it is pure software raid. if you want to create a disk larger that 2TB you will have to use what is called a GPT disk. This will allow over 2TB in windows (windows 2003 server SP1 and above)

    i have a 3TB array sitting in my server at the moment....

    by the way which chenbro case did you buy out of curiosity?
     
  9. OP
    OP
    fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,441
    Location:
    City, Canberra, Australia
    Just to clear some things up.

    I am running this with linux software raid. The idea is to use the SATA controller as just a controller, with the OS seeing every drive. If the idea was to use raid controllers, I would just buy a 3ware etc card.

    Currently running in iSCSI at 2.1Tb, loaded with openfiler 2.2 64bit. Installing on the DFI was a breeze. I did try on the Asus P5B, it was alot harder, and I had to use a SATA port to boot off.

    This is a proof of concept for a bigger array, say 8 1Tb WDC drives.

    Now to try test ZFS on the same mobo. I had a ZFS install on a intel P4, with 4 PATA 200gb disks. Was ok, but the speed wasnt the best.


    [​IMG]
    Click to view full-sized image!
    Hosted by UGBox Image Store
     
  10. aza2001

    aza2001 Member

    Joined:
    Sep 14, 2002
    Messages:
    2,016
    Location:
    Northmead

    Sorry i didnt mean to bag out software raid in total i just meant that the silicon image software raid is shit... if you setup linux raid 5 correctly it works wonders and can be as fast as alot of hardware raid controllers
     
  11. OP
    OP
    fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,441
    Location:
    City, Canberra, Australia
    I am still working on this. I was however very busy with work, setting up some esx hosts, and a san.

    I have since upgraded this machine to 8x 500gb Seagate drives.

    I went out and bought a Supermicro AOC-SAT2-MV8 8 port card. It is a clone of the broadcom 4852 card, without the raid firmware on it. Pitty. I was able to get ZFS to install on this, however this machine doesnt have 64bit pci slots, so the performance wasnt that great. I might have another go with a 64bit pci system.
     
  12. pace282

    pace282 Member

    Joined:
    Jun 26, 2001
    Messages:
    171
    fad interested to see your results with the linux and bds solutions.
    I'm not confident enough with them so I used Server2003.
    Mine is a:
    DFI NF4
    3ware 9650SE-16ML
    8x 1TB WD GP Raid6
    7x 500GB Seagate Raid5
    OS is Win 2003sp2.

    Question with Openfiler how well does the Sil3114 work. We have had issues with corruption with that chipset on Freenas. Is it any more stable on Openfiler?
     

Share This Page

Advertisement: