Restoring backups for 1,000 servers from compressed tape. OUCH!

Discussion in 'Storage & Backup' started by rowan194, Oct 19, 2010.

  1. rowan194

    rowan194 Member

    Joined:
    Jan 5, 2009
    Messages:
    2,031
    Reality Check Networks, a web host in the USA, has apparently had an ex employee hack into their management system and from there corrupt the start of the arrays in most (all?) of their servers, effectively nuking their file system and killing the server.

    As a further hurdle, all of their backups are on tape, and compressed. This system was only designed for the occasional restore to a toasted server, not a wholesale reinstall to every single server. They estimate that to restore everything (at the current rate) could take up to a month.

    So far it's been about 2 days. Possibly one of the most significant storage events/losses in the hosting industry in recent times.

    This is when clients come out of the woodwork and complain they're losing thousands of dollars, yet they have a single server at a single webhost... redundancy doesn't just mean sticking a second drive in a RAID1 array...
     
  2. Ravennoir

    Ravennoir Member

    Joined:
    May 1, 2007
    Messages:
    6,389
    Location:
    Melbourne
    Have you got any links about this ?

    How did you hear about it ?

    Other than backup tape, what would be your options for redundancy ?
     
  3. Herballizard

    Herballizard Member

    Joined:
    Oct 9, 2002
    Messages:
    1,533
    Location:
    .
    it was on torrent-freak but can't find a more reputable source
     
  4. spludgey

    spludgey Member

    Joined:
    Jun 30, 2001
    Messages:
    5,451
    Location:
    Sydney
    Ironical name!
     
  5. Thalfura

    Thalfura Member

    Joined:
    Sep 11, 2006
    Messages:
    355
    Location:
    Wait awhile state 6112
    i think this says it all... Clicky

    Sure as hell wouldn't want to be the one doing all the restores.
     
  6. OP
    OP
    rowan194

    rowan194 Member

    Joined:
    Jan 5, 2009
    Messages:
    2,031
    Just off the top of my head...

    0) RAID with hotswap - a failed drive won't down your server (straight backups take time to restore, and will lose any data changed since the last backup)
    1) Host managed backups
    2) Extra HD(s) in your server for backups
    3) Offsite backups, so you can restore data to a new server at another host within hours
    4) Secondary server at another host, regularly synced to primary, so you can switch your site over in the event of an extended outage. Like RAID1, but for servers rather than HDs. :leet:

    Ideally you'd implement more than one of these solutions: you don't want to find out that the host "forgot" to back you up (it happens :( ), or that a backup HD has failed, when you most need it. These solutions cost extra money of course, but so does any insurance.
     
  7. aXis

    aXis Member

    Joined:
    Jun 27, 2001
    Messages:
    5,164
    Location:
    Kalgoorlie, WA
    Nasty. Damn I hate tape, easier these days to buy cheap consumer 1TB drrives.

    For my servers I have regular hardware redundancy (RAID, PSU's, fans) and replace the hardware every few years when the warranties run out. Config files and database get backup up to a NAS.

    I then use the previous hardware to host VM images of the same servers, and restore the config/databases from NAS back to the VM's daily. This tests both the effectiveness of the backups and gives me failover servers (albeit at lower performance levels).
     
    Last edited: Oct 19, 2010
  8. Ravennoir

    Ravennoir Member

    Joined:
    May 1, 2007
    Messages:
    6,389
    Location:
    Melbourne
    I think though, if he has "nuked" the arrays, if they had redundant servers, SAN Disk Storage or Extra HDDs, he would have just nuked those too

    Best option would have been to ensure he had no access to the Servers\Data etc


    EDIT : I found a News Article

    http://www.theinquirer.net/inquirer/news/1790673/-employee-brings-bittorrent-trackers
     
  9. mr_wrxman

    mr_wrxman Member

    Joined:
    May 1, 2005
    Messages:
    2,381
    Location:
    Sydney 2141
    Wow, I hope he gets reamed for this, surely reality check networks could sue him for something like this and send him bankrupt?
     
  10. Quoccus

    Quoccus Member

    Joined:
    Mar 23, 2009
    Messages:
    387
    Location:
    Perth
    it wouldn't matter what setup you had, he still had access and knew the network setup.

    Just poor security management on their end.
     
  11. Silvion

    Silvion Member

    Joined:
    Jan 18, 2008
    Messages:
    475
    Location:
    Melbourne
    He could of been a sys admin, therefore knowing a lot more than most..?
    Not that it specifies though, so no one knows
     
  12. OP
    OP
    rowan194

    rowan194 Member

    Joined:
    Jan 5, 2009
    Messages:
    2,031
    Since there's 1,000 servers I presume he used some sort of script or other automation; all he had to do to kill them was corrupt the OS drive/array, which is pretty likely to be the first storage device on each server.

    Hopefully that means that if people do have offsite backups, or secondary servers, that they were not touched. Enough damage would have been done by disabling 1k servers...

    I've read that it was an ex employee so he had intimate knowledge of their management system.

    Bit of a worry if he was able to get in from outside their NOC.

    [​IMG]
     
    Last edited by a moderator: Oct 19, 2010
  13. cvidler

    cvidler Member

    Joined:
    Jun 29, 2001
    Messages:
    11,664
    Location:
    Canberra
    There's nowhere to put cheap consumer 1TB drives in a modern server.
    They all take 2.5" SAS drives.

    Tape is still a very important and common backup solution. But as stated RCN's problem stemmed from poor security management. The ex-exmployees accounts shouldve been disabled promptly, and all root/admin passwords changed as a precaution whether he knew them or not - again promptly.
     
  14. aXis

    aXis Member

    Joined:
    Jun 27, 2001
    Messages:
    5,164
    Location:
    Kalgoorlie, WA
    Personally I stick them in a cheap gigabit NAS and then rotate a few through for offsite storage, but then again I only have to deal with 10 to 20 servers.
     
  15. MetalPhreak

    MetalPhreak Member

    Joined:
    Apr 8, 2007
    Messages:
    1,763
    Location:
    Perth
  16. DeVo

    DeVo Member

    Joined:
    Jan 3, 2002
    Messages:
    344
    Location:
    Bendigo
    RAID is not a form of backup, it's a High Availability technology. This is important to remember and no form of RAID will protect you against corruption of data or a malicious attack.

    A data protection strategy for service providers hosting thousands of servers is not a simple task. Using cheap consumer HDD's or NAS's is not appropriate for an environment of that scale.

    That said, I'm not defending this company. Relying on tape as your primary replica of your production data is asking for trouble. Tape is cheap, service providers using tape as their primary data replica are trying to save money and unfortunately for RCN, this has come back to bite them in the end.

    From a technology perspective, in an ideal world, they would be replicating their SANs to alternate data centres. This would give them a thick replica of their data on spinning disk for rapid restoration.

    That is a technology solution however and in most cases, your storage guys have access to all of your storage. It still only takes one disgruntled employee to bring the empire down. This is very difficult to mitigate for any organisation.

    P.S. RAID is not a form of backup!!

    Cheers
    Matt
     
  17. darth_wolf

    darth_wolf Member

    Joined:
    Jul 20, 2007
    Messages:
    1,663
    "They say they had an Ex-Employee break into their system and corrupt their servers"

    "Well i guess that was,"

    "a reality check"

    YEAHHHHHHHHHHHHH!
     
  18. DeVo

    DeVo Member

    Joined:
    Jan 3, 2002
    Messages:
    344
    Location:
    Bendigo
    HAhahahahaa wow what a sensational call.

    ahahaha.
     
  19. RyoSaeba

    RyoSaeba Member

    Joined:
    Sep 11, 2001
    Messages:
    12,279
    Location:
    Perth
    Maybe it's that company that works for RIAA that goes around hacking into BT tracker servers etc.
     
  20. Herballizard

    Herballizard Member

    Joined:
    Oct 9, 2002
    Messages:
    1,533
    Location:
    .
    If an actual chain of evidence could be produced and there are more than just BT trackers hosted there consider them sunk
     

Share This Page