Overclockers Australia Forums

OCAU News - Wiki - QuickLinks - Pix - Sponsors  

Go Back   Overclockers Australia Forums > Specific Hardware Topics > Business & Enterprise Computing

Notices


Sign up for a free OCAU account and this ad will go away!
Search our forums with Google:
Reply
 
Thread Tools
Old 14th December 2016, 10:06 AM   #1
link1896 Thread Starter
Member
 
link1896's Avatar
 
Join Date: Jul 2005
Location: Melbourne
Posts: 351
Default Ok who broke rsync at the ATO?

http://www.zdnet.com/article/ato-out...storage-issue/

Petabyte of data lost. Very few details yet. Could make the census debacle seem like a teddy bears picnic.
link1896 is offline   Reply With Quote

Join OCAU to remove this ad!
Old 14th December 2016, 10:22 AM   #2
NSanity
Member
 
NSanity's Avatar
 
Join Date: Mar 2002
Location: Canberra
Posts: 15,952
Default

I mean, its back up now (well parts of it).

This is why you need a backup.

Last edited by NSanity; 14th December 2016 at 10:31 AM.
NSanity is online now   Reply With Quote
Old 14th December 2016, 11:50 AM   #3
^catalyst
Member
 
^catalyst's Avatar
 
Join Date: Jun 2001
Location: melbourne
Posts: 11,498
Default

RAID is not backup
RAID is not backup
^catalyst is offline   Reply With Quote
Old 14th December 2016, 11:51 AM   #4
NSanity
Member
 
NSanity's Avatar
 
Join Date: Mar 2002
Location: Canberra
Posts: 15,952
Default

Quote:
Originally Posted by ^catalyst View Post
RAID is not backup
RAID is not backup
BUT HP SED IF I RSYNC TO DAT ONE WE GUD?
NSanity is online now   Reply With Quote
Old 14th December 2016, 12:14 PM   #5
shredder
Member
 
shredder's Avatar
 
Join Date: Dec 2001
Location: Dec 1991
Posts: 8,966
Default

Quote:
Originally Posted by NSanity View Post
BUT HP SED IF I RSYNC TO DAT ONE WE GUD?
shredder is online now   Reply With Quote
Old 14th December 2016, 12:21 PM   #6
NSanity
Member
 
NSanity's Avatar
 
Join Date: Mar 2002
Location: Canberra
Posts: 15,952
Default

Quote:
Originally Posted by shredder View Post
https://youtu.be/Gnjb1WVkhmU?t=112 (skip to 1 min 52 sec).
NSanity is online now   Reply With Quote
Old 14th December 2016, 12:35 PM   #7
elvis
Old school old fool
 
elvis's Avatar
 
Join Date: Jun 2001
Location: Brisbane
Posts: 28,504
Default

Boss: "Realtime synchronise all of the things!"

Grunt: "But sir, what if there's an error and we don't have an offline copy of..."

Boss: "I SAID ALL OF THE THINGS!!!"

Days later, corruption. Days later, investigation. Days later, result:

Boss: "Heads will roll! We've sacked all our grunts, and told the new ones never to make this mistake again".
__________________
Play old games with me!
elvis is offline   Reply With Quote
Old 14th December 2016, 12:42 PM   #8
Sphinx2000
Member
 
Sphinx2000's Avatar
 
Join Date: Sep 2001
Location: Brisbane
Posts: 4,650
Default

HPE Engineer: "How we tell them it actually failed?"
HPE Boss: "Tell them this is first time this has ever happened in the world, that always makes them feel better"

Quote:
According to ATO acting chief information officer Steve Hamilton, "specialist engineers" have been working through the night with ATO staff to rectify the outages.
"We understand this is the first time this problem has been encountered anywhere in the world and we are working with HPE to determine the underlying cause."
Sphinx2000 is offline   Reply With Quote
Old 14th December 2016, 12:47 PM   #9
PabloEscobar
Member
 
Join Date: Jan 2008
Posts: 9,124
Default

All the press is coming from the "Acting CIO"...
Perhaps heads already have.
PabloEscobar is offline   Reply With Quote
Old 14th December 2016, 12:50 PM   #10
chip
Member
 
Join Date: Dec 2001
Location: Perth
Posts: 3,301
Default

Quote:
Originally Posted by Sphinx2000 View Post
HPE Engineer: "How we tell them it actually failed?"
HPE Boss: "Tell them this is first time this has ever happened in the world, that always makes them feel better"
I don't have specifics on either case, but it sounds similar (ie catastrophic data loss) to this one: http://www.theregister.co.uk/2016/11...ake_their_own/

I've also seen smaller HP SANs with firmware faults that replicated and killed production data a couple of times over the years.
chip is offline   Reply With Quote
Old 14th December 2016, 12:51 PM   #11
power
Member
 
power's Avatar
 
Join Date: Apr 2002
Location: brisbane
Posts: 48,893
Default

Quote:
Originally Posted by Sphinx2000 View Post
HPE Engineer: "How we tell them it actually failed?"
HPE Boss: "Tell them this is first time this has ever happened in the world, that always makes them feel better"



HP Appreciates your participation in this public beta.
__________________
this is who we are.
power is offline   Reply With Quote
Old 14th December 2016, 12:53 PM   #12
cesario
Member
 
cesario's Avatar
 
Join Date: Jun 2009
Posts: 256
Default

This story has been the hot topic around my workplace, id be very keen to find out exactly what happened.
Sounds like one of those 'oops' moments
__________________
Carbon Antec Super Darkboy
cesario is offline   Reply With Quote
Old 14th December 2016, 12:58 PM   #13
NSanity
Member
 
NSanity's Avatar
 
Join Date: Mar 2002
Location: Canberra
Posts: 15,952
Default

Quote:
Originally Posted by cesario View Post
This story has been the hot topic around my workplace, id be very keen to find out exactly what happened.
Sounds like one of those 'oops' moments
Hardly.

SAN's flake out from time to time. If its filesystem level, it typically replicates at the speed of light (i mean, thats what you wanted right?) - but by the time you notice the damage is done.

If you have snapshots pre-corruption, and they aren't affected, you can roll back.

If you don't well you're rolling back to backup pre-corruption.

elvis and I keep saying this, but large datasets need filesystem level integrity checks - and well that's a next-gen filesystem - e.g ZFS, BTRFS and ReFS (although the latter doesn't work for VM workloads - because you turn integrity streams off).
NSanity is online now   Reply With Quote
Old 14th December 2016, 1:05 PM   #14
PabloEscobar
Member
 
Join Date: Jan 2008
Posts: 9,124
Default

Quote:
Originally Posted by cesario View Post
This story has been the hot topic around my workplace, id be very keen to find out exactly what happened.
The first problem was buying HPE kit...
PabloEscobar is offline   Reply With Quote
Old 14th December 2016, 1:16 PM   #15
cesario
Member
 
cesario's Avatar
 
Join Date: Jun 2009
Posts: 256
Default

Kinda have to feel bad for Steve Hamilton though
"Acting CIO 28 November to 16 December"
All this 2 days before his role ended, now thats some terrible timing
__________________
Carbon Antec Super Darkboy
cesario is offline   Reply With Quote
Reply

Bookmarks

Sign up for a free OCAU account and this ad will go away!

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time now is 7:31 PM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
OCAU is not responsible for the content of individual messages posted by others.
Other content copyright Overclockers Australia.
OCAU is hosted by Micron21!