Ok who broke rsync at the ATO?

Discussion in 'Business & Enterprise Computing' started by link1896, Dec 14, 2016.

  1. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,784
    Location:
    Canberra
    5 times in 6 months with the same root cause they didn't.

    5 times in 6 months with the same root cause they didn't.

    They bought a fucking lemon config'd san.
    They let idiots leave all their recovery tools on said san.
    They left said SAN configured such that something like as little as 12 disks brought down an array with 800 drives in it.
    They let idiots re-wire their fabric, and dislodge fibre cards from the controllers, causing more outage.

    They've had 4-5 outages in 6 months. Its absolutely unacceptable.
     
  2. hosh0

    hosh0 Member

    Joined:
    May 28, 2007
    Messages:
    8,969
    Location:
    Sydney N.S.W
    Jesus. NAB had three outages in a week! Also it's three incidents not 5 and one was not related to the other two. But sure let's just keep shitting away without knowing the full problem or root cause.

    I hope you apply yourself better in your work because you've sure jumped in half assed here.
     
  3. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,784
    Location:
    Canberra
    The ultimate root cause is people signing off on things they don't understand - but just seeing dollars - and attempting to outsource risk.

    But after multiple, org-wide outages that resulted in having to send thousands of staff home and seriously delay tens of thousands of businesses that have a reporting requirement with them - you pull your head in and do what needs to be done to get it sorted - which in the ultra short term means opening up your wallet (good, fast, cheap - pick two).

    :rolleyes:

    There is literally a published document about the causes of the failures here - and its straight up amateur hour.

    https://www.theregister.co.uk/2017/06/08/ato_hpe_outage_report/

    https://www.ato.gov.au/uploadedFiles/Content/CR/downloads/js39322_ATO-systems-report_w.pdf

    We discussed this earlier in the month - and everyone who isn't a shill for HPe agrees that all of these problems should have been avoided in a deployment of this size.
     
  4. hosh0

    hosh0 Member

    Joined:
    May 28, 2007
    Messages:
    8,969
    Location:
    Sydney N.S.W
    That's all beside my point. Which was your comment that if they were a bank they wouldn't exist was total shit.

    These are massive cock ups no doubt and people responsibly should definitely be held accountable. But it's not as if the ATO is the only one to ever suffer outages and that banks never do.

    That attitude is just fundamentally wrong.
     
  5. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    17,784
    Location:
    Canberra
    The only bank outage genuinely comparable is the CBA HPe where they pushed a SCCM refresh deployment meant for their dev env across prod - flatting ~600 servers and 9000 odd desktops. As an individual incident, its absolutely worse - but we're talking repeated org-wide outages - both internal and client facing.
     
    Last edited: Jun 26, 2017
  6. chip

    chip Member

    Joined:
    Dec 24, 2001
    Messages:
    3,772
    Location:
    Pooraka Maccas drivethrough
    We (the public) will probably never hear the full story, thanks to wonder of "commercial in confidence". This is when we need a strong 4th estate, and they have always relied on confidential sources...
     
  7. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    13,818
    Get yourself behind 7 Boxees and go deep throat style on Reddit or Wikileaks.

    If the wrong information is being published, its because the wrong information is being provided somewhere.

    Someone at the ATO decided what the "official line" was going to be, thats what gets fed to the press, so that's what they publish.

    They can't really do any investigation on it, because everyone in your position values their public service gravy train to much to blow the whistle on it.
     
  8. hosh0

    hosh0 Member

    Joined:
    May 28, 2007
    Messages:
    8,969
    Location:
    Sydney N.S.W
    NAB's was far far worse, three outages over a week, one of which went for two or so days where people couldn't get paid, access their money or even log into internet banking to make payments. All of these were nation wide as well so across all their systems.

    http://www.news.com.au/finance/busi...g/news-story/a310b0b146e55f56a53bf77a6051c247

    That was three times last year, and then was out again earlier this year

    http://www.smh.com.au/technology/te...s-customers-in-the-lurch-20170531-gwh3kk.html


    Again none of these are okay and everyone who is responsible should be held accountable. But I just can't stand this notion that ATO is alone in failing and if it was private industry this wouldn't happen.

    It's bullshit, it happens in private industry just as much and we need to stop acting like IT is in a good state in any segment of the market.
     
  9. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,469
    Location:
    qld.au
    Outside looking in, you're looking more the ass than NSanity....

    ^^^ This. Anyone with any level of competency I've talked to dumped HPe storage like a hot potato. The only people I know still using it couldn't afford to replace, so they're weathering it out. None of them ran a config like the ATO, nor did they ignore the blatant deployment best practices.

    I wouldn't trust bank systems either, but they do generally have better planning and redundancy. The outages have been far, far more complex than a trivial storage failure and certainly wasn't as long as the ATO's issues.

    The ATO's failures are akin to an airline forgetting to refuel a plane before taking flight, then repeating again because they didn't learn from the last time. The third outage will be 1 part bad luck, 9 parts poor planning.

    Please, name the entities you know who have had 3 massive storage failures in the last 6 months? If it wouldn't happen in the private world, then there must be examples :)
     
  10. hosh0

    hosh0 Member

    Joined:
    May 28, 2007
    Messages:
    8,969
    Location:
    Sydney N.S.W
    I know it's been a few months since nab went down so I'll go you the benefit of the doubt and just say you might have forgotten. But their first outage was caused by someone deleting the primary production domain name.

    I can't name any other major company that's had the storage outages. But does the fact it's a storage vs some other equally avoidable method matter?

    I'm getting the sense that you think I work for HPE or care what happens to them.

    I don't I couldn't give a shit to be honest. It's the attitude that the private sector is somehow more competent than the Gov and this would never happen there that I've got a massive problem with, because it's wrong. If this was a private company there wouldn't be a report published for us to read.
     
  11. BAK

    BAK Member

    Joined:
    Jan 7, 2005
    Messages:
    1,168
    Location:
    MornPen, VIC
    The stereotype that government tends to pay more for less competence than private sector exists for a reason. It's still a stereotype, and doesn't fit every situation, but having worked in both sectors the stereotype fits a lot more than I'd like it to (being a taxpayer etc).

    Private companies do sometimes publish incident reports. Gitlab is a fairly major one that occurred recently: https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/
     
  12. hosh0

    hosh0 Member

    Joined:
    May 28, 2007
    Messages:
    8,969
    Location:
    Sydney N.S.W
    Couldn't agree more. Thankfully I'm finally starting to see a change in attitude and more large scale projects are looking to be done internally now.

    The main problem is a pay scale to acknowledge that there are technical people worth a senior executives salary.
     
  13. scissorspaper

    scissorspaper Member

    Joined:
    Apr 10, 2016
    Messages:
    16
    Location:
    Canberra
    Yeah too much bias with HPE Being the vendor pushing HP Crap.
    They cut back heaps of staff, to make the books look better
    Sold to CSC (DSC now)
     
  14. trevor68

    trevor68 Member

    Joined:
    Jun 28, 2001
    Messages:
    4,297
    Location:
    Canberra
    Nope, it's because nobody fact checks shit anymore, they clearly just publish whatever they feel like, or are just idiots, one or the other.

    BTW, I'm not refering to the two well documented and entirely unrelated incidents from Dec/Jan, only the utter tripe that got rehashed for the third and entirely unrelated incident. ;-)
     
  15. Diode

    Diode Member

    Joined:
    Jun 17, 2011
    Messages:
    1,736
    Location:
    Melbourne
  16. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    13,818
    Why not both?

    How can you fact check when the official spokesperson have NFI, and those that do, don't actually provide any real information, and just poo-poo your story from afar?
     
  17. looktall

    looktall Working Class Doughnut

    Joined:
    Sep 17, 2001
    Messages:
    25,018
    i see the ATO is sending out emails encouraging people to wait until mid-august to lodge their returns.

    they claim it's to allow for all of the pre-filled info to be ready.
     

Share This Page

Advertisement: