1. OCAU Merchandise now available! Check out our 20th Anniversary Mugs, Classic Logo Shirts and much more! Discussion here.
    Dismiss Notice

Virgin Blue computer outage

Discussion in 'Business & Enterprise Computing' started by IntelInside, Oct 1, 2010.

  1. Nyarghnia

    Nyarghnia (Taking a Break)

    Joined:
    Aug 5, 2008
    Messages:
    1,274
    Sorry to sound ignorant of the issue here... but was Virgin knocked out by something as simple as a few hard drives falling over?!

    -NyarghNia
     
  2. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    3,379
    Location:
    Canberra
    Ref:http://www.theregister.co.uk/2010/09/29/netapp_virgin_blue/
    Ref:http://en.wikipedia.org/wiki/RAID

    whether netapp was addressing those units on an individual basis or as members of a RAID-DP aggregate I'm uncertain, but either way one of the units took a hit and shit went pair shaped.
     
  3. Nyarghnia

    Nyarghnia (Taking a Break)

    Joined:
    Aug 5, 2008
    Messages:
    1,274
    Thanks for the info..

    Frankly I'm amazed that anyone is deploying SSD's in production at the moment, from where I sit they're just 'too new'.

    I'd rather go for more traditional disk storages systems, Surely you don't need 100,000+ sustained IOPS to handle booking for Virgin Blue!?

    So, we're talking about 2TB of SSD disk storage?

    Sounds to me like they just went overboard with tech and forgot the basics...

    -NyarghNia
     
  4. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    3,379
    Location:
    Canberra
    yep: http://www.ramsan.com/images/products/500.jpg

    • The World's Fastest StorageĀ®
    • Cached Flash RAID for the Enterprise
    • 1TB to 2TB Flash RAID
    • 16GB to 64GB DDR Cache
    • 100,000 random IOPS sustained (reads from Flash)
    • 2 GB/s sustained bandwidth (to Flash)
    • Full array of hardware redundancy to ensure availability.

    Ref: http://viroptics.pancamo.com/2007/09/ramsan-500-2tb-solid-state-raid.html

    Ref: http://www.texmemsys.com/files/f000233.pdf
     
    Last edited: Oct 7, 2010
  5. DeejW

    DeejW Member

    Joined:
    May 18, 2008
    Messages:
    1,950
    Location:
    Margaret River WA
    I lold... :p
     
  6. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    41,770
    Location:
    Brisbane
    I loathe this sort of thinking.

    "IT" is so diverse, it's not anyone's core business. Nobody does IT for the sake of itself. IT is merely the digital representation of business concepts.

    Outsourcing it to an "IT specialist" is flawed thinking. No third party is going to know your business as well as you do. They're not going to empathise with your customers like you do. Sure, there's a time and a place where some things can be outsourced, but this concept of outsourcing everything that takes an electric current is just madness.

    And that's an important point, too. I'm not a racist by any stretch of the imagination, but Indian culture/business is very, VERY different to that of the west. And it's something that's more often than not given no consideration until it's too late. Instead, dollar costs are always given higher priorities, and the real world risks ignored just to make budget.

    It sounds chillingly like the mob I'm working for at the moment. I've tried leaving a few times, only to be convinced by the top brass to stay each time I hand in a resignation. They tell me they need people like me to stop the place imploding and fix all the issues, but these days I wonder if just a handful of people can really make a difference in a place that has literally hundreds of people who range from incompetent to apathetic.

    If only the bastards didn't pay so well...
     
  7. lavi

    lavi Member

    Joined:
    Dec 20, 2002
    Messages:
    4,004
    Location:
    Brisbane
    the way i look at it is this .... they saved a few million by outsourcing and getting rid of staff and lost just as much with this failure and to top it of they are the laugh of town and will spend more millions in getting back on track ....they practically shoot themselves in the foot not once but twice! and now they're ready to reload and do it again ... can somebody from here go there and take rounds and throw them away and put them out of their missery ?
     
  8. s.Neo

    s.Neo Member

    Joined:
    Oct 23, 2002
    Messages:
    398
    Location:
    Darwin, NT, Australia
    SSD's arn't new. They've been used in their modern 'flash' based format for around 15 years. The fact is that like any hardware (solid state or mechanical), drives fail. We manage this risk by building redundancy into our systems, but for what ever reason, Navitaire's system(s) failed to deliver.
     
  9. VorknHull

    VorknHull Member

    Joined:
    Jun 27, 2001
    Messages:
    665
    Location:
    Riverhills, QLD
    'Sorry, we're a LCC so we are unable to pay you at market rate' ... still remember that :)
     
  10. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    Not hard to read between the lines on this one, NetApp jumped up straight away to deny any involvement with the problems. Nothing sounds more guilty than instant denial :)

    There's nothing wrong with "outsourcing" services as long as you're doing it right. It's like servicing cars, we "outsource" the job to a mechanic because they're generally better trained and better equipped to do it. There would be a lot of wasted effort to train yourself to the same level, yet only service a car once a year.

    I don't see IT as any different, it makes sense to outsource some items to experts as they can potentially do a better job. I think the main problem is that the outsourcing decision is made to save money, not because it makes sense.

    Nyarghnia: There's a lot of SSD's out there in the enterprise world, it's just not possible to get the same performance with out massive SAN's. I've used a few of the FusionIO cards and the speed is amazing when it comes to SQL performance. As for needing 100,000 IOPS I definitely see it as possible. You'd be amazed at how complex some of those systems can actually be (and need to be).
     
  11. exodus_68

    exodus_68 Member

    Joined:
    Jun 10, 2003
    Messages:
    354
    Location:
    Perth
    Slightly OT:

    See your still thinking of storage and IOPS in the 'old' way, start thinking more about virtualisation and SSD starts to make a lot of sense. Remember that most SANs were (and a few still are) designed for RDM between a few hosts with 1:1 operating system ratios, not consolidated virtual environments.

    Keep in mind the vendors offerings SSD are setting them up with storage virtualisation so you are not bound to a static LUN, the storage group exists accross multiple tiers of storage, and will move data blocks onto high performance disk as required. Enterprise class SSD is also quoted by vendors has having a lower failure rate that traditional spindles as there are no moving parts.

    Anyway that's my two cents. ;)
     
  12. daffy

    daffy Member

    Joined:
    Jun 29, 2001
    Messages:
    1,280
    Location:
    Brisbane
    I agree with your whole post but this part especially, given the company I work for now this ever so obvious:sick:
     
  13. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    41,770
    Location:
    Brisbane
    I'm witnessing it more and more where managers think that good process means being able to replace intelligent people with monkeys (or to the same result: not replacing intelligent people when they get fed up and leave).

    There seems to be a complete void in their logic where they fail to realise the need for good people to come up with new processes in the first place.

    While phrases like "innovation" are thrown around with glorious abandon, the actual practice of being innovative seems to petrify most managers. Even worse are the places that pride themselves in being innovative on paper, but don't do anything unless it's been industry proven first. (Surely they should see the fact that you can't innovate if you merely follow the crowd?).

    It seems some of these individuals are actually quite afraid of having technical people with freedom of thought. By their logic, if it's not on a giant checklist, then it must be wrong on principle.

    I have a bookmark folder in my browser called "Corporate Boneheads". In it I store links to IT disaster stories like this recent Virgin Blue episode, and dig them out 2-3 years after when some idiot manager has a brilliant idea to try the same thing all over again (clearly these individuals have the memory spans of guppies).
     
    Last edited: Oct 8, 2010
  14. Nyarghnia

    Nyarghnia (Taking a Break)

    Joined:
    Aug 5, 2008
    Messages:
    1,274
    Umm... was in a meeting some time ago in a previous job...you'll love this..

    CIO: "We dont have any technical people involved in I.T"
    Me: "That doesn't seem to make much sense..."
    CIO: "I'd rather get a business person and have them learn I.T fundamentals than get an I.T person to understand business"

    Me: ???

    Being 'Non Technical' is almost a badge of honour, even amoungst those people who are involved with information technology...

    Or my favorite...

    "I don't believe a project manager needs any I.T knowledge to manage an I.T project"...

    That was from a senior person at a very large public coorporation who blew $65m on a 3 year failed project that cost about 250 people their jobs.. of course they got promoted after that... i'll say nothing more.


    -NyarghNia
     
  15. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    41,770
    Location:
    Brisbane
    Yeah, I definitely feel that these days.

    It's almost as if a certain subset of these managers like to hold on to the archaic idea that techs are anti-social nerds with no business acumen. The very idea of business-savvy technical people seems to almost scare them to their core.

    I think when it comes down to it, they want to believe as hard as they can that an MBA is the worthwhile qualification, and that technical qualifications and experience in the techy trenches is merely some sort of black magic voodoo that can be outsourced.
     
  16. Doc-of-FC

    Doc-of-FC Member

    Joined:
    Aug 30, 2001
    Messages:
    3,379
    Location:
    Canberra
    in 2035, technology researchers will scour the old interwebs on optical media and one day stumble upon this thread.

    it will be frustrating for the researchers because they probably won't be able to find who we are in 25 years time.

    I'm sure they'll have a plethora of questions though.

    vmware, xen, kvm = appletalk, netbeui, ipx/spx, tcp/ip

    technologies will come and go, only the best will live on.
     
    Last edited: Oct 8, 2010
  17. CordlezToaster

    CordlezToaster Member

    Joined:
    Nov 3, 2006
    Messages:
    4,078
    Location:
    Melbourne
    Thats touching doc
     
  18. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    41,770
    Location:
    Brisbane
    Those that are based on open standards have a much better chance of living longer.

    You only need to look at the oldest existing software and protocols today to see that's true. If longevity is your goal, open standards and systems are what you need to specialise in.
     
  19. ewok85

    ewok85 Member

    Joined:
    Jul 4, 2002
    Messages:
    8,104
    Location:
    Tokyo, Japan
    I've heard lots of good things about these SAN's - this isn't off the shelf stuff. They've had this stuff on the market for 3 years and have been rolling out more and more models over the years with alot of success.

    I remember the 2TB Ramsan-500 was about US$300k in 2007 when announced, today its probably closer to $50-60k.
     
    Last edited: Oct 9, 2010
  20. round

    round (Banned or Deleted)

    Joined:
    Apr 7, 2007
    Messages:
    15,474
    Location:
    /pol/
    well to be fair, i outsource everything that i cant do as cheap as others can, or as well.

    i cant pull my own teeth, so i outsource

    i cant do my own hair....etc etc
     

Share This Page

Advertisement: