Distribute.IT's CEO's story of the '11 hack that destroyed Distribute.IT

Discussion in 'Business & Enterprise Computing' started by Cubix, May 12, 2014.

  1. Cubix

    Cubix Member

    Joined:
    Apr 15, 2011
    Messages:
    110
    Bit of an interesting explanation of events on how the attack unfolded.


    Article has since been removed by CIO.com.au



    As per cache grab from ShadowBurger of before article was removed.



    -------------------- Page 1 --------------------
    Distribute.IT: When a hacker destroys your business
    Byron Connolly (CIO) on 12 May, 2014 11:38

    It’s been almost three years since business owners Carl Woerndle and his brother Alex were caught up in a cyber attack so damaging, it destroyed their once prospering technology business, Distribute.IT.

    At this year’s CIO Summit in Perth, Woerndle – now a cyber security advisor at Deloitte – gave a warts and all account of how he and other staff at his former company dealt with the crisis and the fallout.

    The brothers founded Distribute.IT in 2002, a typical startup working out of a spare room with, as Carl puts it, “no money, lots of good ideas and lots of enthusiasm".

    At the time, the fledgling firm was granted one of five domain registrar licences available in the marketplace following moves by the government to place more control around domain name management, and remove Melbourne IT’s monopoly position.

    The company adopted a channel sales strategy, appointing resellers to on-sell its services unlike its competitors, which were focusing on retail business.

    Over the next nine years, the firm branched off into Web server hosting, in 2006 building its infrastructure around virtual server technology. By 2011, it was operating three data centres and was an Australian distributor for Verisign.

    Distribute.IT had secured a 10 per cent market share for Australian domain names with 250,000 under management, and more than 200,000 clients on its books.

    “We were growing at 4 per cent a month,” says Woerndle. “We also had over 30,000 hosting clients on that infrastructure ... and 3,000 active resellers on our database at that time.”

    The company had more than 50 global domain name accreditations and opened an office in Jakarta, Indonesia as a launching pad to sell through Asia. It was looking at a couple of acquisitions and a possible IPO in 2014.

    The initial breach – week one

    At 5pm on Friday June 3, 2011, Woerndle received a call from his CIO alerting him to a breach in the company’s network.

    “Being a hosted platform, we had 16,000 IP addresses in our network, we had ports all over the place for people to transact their business in and out, it [was] a very open network,” says Woerndle.

    “We were very used to hacking attacks, and attempts to infiltrate our network were actually a daily occurrence for us.”

    Distribute.IT had experienced six major distributed denial of service (DDoS) attacks up until June that year and had written its own proprietary software to detect and “shift out” those IP addresses when they were hacked.

    “We had about 30,000 clients and a minimum of two per day were targeted on our network. Most of those were limited to the websites themselves such as a vulnerability on the site or a particular server.”

    But this attack was different. The hacker had managed to bypass the company’s entire security protocol, get behind its firewall, and gain access to its master user access information.

    “We had somebody inside our network; I’m watching them, they are not doing anything, moving very freely around our network structure not stealing anything, just a bit of a fishing exercise,” he says.

    Distribute.IT had a secondary problem. Under its agreements with the regulator – .au Domain Administration (auDA) – it was mandatory for the company to report breaches of this nature.

    That evening, the company’s team of 12 IT staff gathered at its Melbourne data centre to create a plan to deal with the breach.

    “We took a proactive and aggressive approach to dealing with the issue – we had a lot spare equipment lying around and some plans around building a new network structure and thought it was a good opportunity to kill two birds with one stone.

    “Over the next 48 hours, we pulled out entire network off the rack – three data centres – at one point everything was sitting on the ground.”

    This created a complete blackout for its customer base before the network was rebuilt and back online on the Monday.

    “We contacted the regulator and advised them of the breach and they made us reset all 250,000 passwords and user information, and advise every single one of those customers of the new details before they would let us connect back to the registry again,” says Woerndle.

    To do this, the team spent the next couple of days writing proprietary software to distribute the information to its clients. Distribute.IT was able to connect back to AusRegistry by Thursday that week, six days since the breach.

    “We were very cautious about the way we opened up access back into our network again. We made a lot of our customers send through requests for IP addresses, names of people for access, ports numbers that they wanted access to – we opened these up very slowly one by one,” he says.

    Carl and his brother and most of Distribute.IT’s staff had worked back-to-back 72 hour shifts during the week following the breach with less than four hours sleep in between. They were exhausted but felt comfortable with the company’s position by the following Friday night.

    -------------------- Page 2 --------------------

    At 4:30pm on Saturday June 11, Distribute.IT’s network monitoring system went crazy. The IT team was watching servers go offline every few seconds.

    The hacker had regained access to the company’s network, meaning that all the work completed the week prior was a total waste of time. The front door was wide open and this time the attack was completely malicious.

    “The first thing they did was replace our primary website with a blank page that says ‘you have been hacked’ using the moniker ‘evil’, put all the passwords from our network down on that page – all the passwords we had reset two hours before the attack,” says Woerndle.

    The hacker accessed the company’s firewall, changed the settings, and locked staff out of the network.

    The hacker also embedded a program inside its network and ran a command called 'rm-rf', which removes the file names and leaves scrambled data on the drives, making them useless.

    “They had a cascading program that ran through our network to destroy as many servers as they could until they got control of it,” says Woerndle.

    This attack targeted Distribute.IT’s primary trading platforms and hosting systems, shared Web servers, and all the corresponding backup systems, removing its ability to trade.

    “They went as far through the system destroying what they could until we could get control of it. They only way we could get control of our network was to go to the data centre, pull the plug out of the wall and turn the power off.”

    The 12-man IT team were called back in an attempt to recover services. The company also contacted the Australian Federal Police (AFP) for assistance.

    “What we’d done the week before, we did again – we rebuilt our entire infrastructure from the ground up. We were into our third 72-hour block [working on the problem] and by this time, we were completely and utterly exhausted.”

    The network was switched on again on the evening of Monday June 13. But with its primary websites and VoIP systems down and client databases compromised, Distribute.IT had no way of communicating with its clients.

    “Even if we did, we didn’t know who they were anyway,” says Woerndle.

    The company had no choice but to create a blog site and post regular updates for customers as the recovery progressed. The site included an email address and Distribute.IT’s small IT team were dealing with 20,000 emails each day. The company was fielding an average of 300 emails daily before the attack.

    Earlier that week competitors were running fire sale offers for Distribute.IT clients.

    “We were pretty happy with that as you can imagine,” says Woerndle in a sarcastic tone.

    Again, the company completed its regulatory reporting, went through the same domain reset process, and met with the AFP for the first time.

    “By the 14th (Tuesday), I am through three shifts of 72 hours, I’m completely exhausted. Around the period, even though it was closer to 90 per cent of our servers [and] we were able to get back up and running again, we still had critical infrastructure that was down.

    “All our core trading platforms and backups, a large number of shared Web servers and those backups were still completely compromised.”

    Distribute.IT started to lose clients as people came to the company’s data centres to pick up their equipment. Trust and brand equity that had been built up over nine years was starting to erode.

    By Wednesday, three IT staff had resigned, and two others went AWOL. Distribute.IT’s CIO suffered particularly badly.

    “Everything fell onto this poor fellow. The AFP wanted him, we wanted to know what we could do to help. He had to come up with a game plan and was under immense pressure,” says Woerndle.

    The CIO was under so much strain that on Wednesday, he collapsed on the floor of the data centre and was taken home by other staff for medical treatment.

    “The game plan changed for us again on the 16th (Thursday) because it was the first day we hit the mainstream press. The customer churn during this period was starting to accelerate … there was a lot of pressure and misinformation running around the place,” he says.

    The media coverage hit its peak on the 18th and 19th of June and journalists were camped outside Distribute.IT’s door.

    “Every time you wanted to go out for a toilet break, they would stick a microphone under your face for an interview. I say this with most respect for IT guys but our IT guys weren’t used to dealing with anybody at all; they sat in the backroom very quietly, they certainly weren’t equipped to deal with the media.

    “One guy was under so much pressure he didn’t want to go outside – he had a bucket under his desk, he was taking toilet breaks under his desk.”

    Knowledge of the hack became so widespread that the company had an email from hacking group Anonymous saying ‘it wasn’t us’.

    “We had phone calls from every major bank in Australia concerned about credit card leakage, PCI compliance, we had the privacy commissioner on the phone – people we hadn’t even heard of who wanted to get a piece of the action so to speak. We even had a call from [Julia] Gillard’s office asking how they could help.”

    Distribute.IT had to announce that it had lost four of its major Web servers used by 4,000 hosted clients. Around 3,000 resellers were also affected.

    “We are talking about our primary servers and up to four levels of backups completely destroyed by the hacker. When we were having our board meetings, we were starting to think about litigation, are we covered for insurances?”

    The recovery effort was slow and by the weekend, 12 IT staff became 6. The CIO turned up 48 hours after his breakdown, which Woerndle described as a ‘godsend'.

    “It shows how heavily reliant we were on that particular individual. During the weekend we had good support from the regulators, but felt we were two to three weeks out from recovering our core trading platform.”

    On the Monday morning (June 20), Distribute.IT received a breach notice from the regulators indicating that they felt the company was no longer able to perform its duties as a domain name registrar.

    “They give you a timeframe in which to rectify the situation or they enact a contingency plan. A contingency plan is pretty easy in the domain name industry. They take all 250,000 domains off you and give them to one of your competitors.

    “We had a secondary problem with this – our resellers – which weren’t recognised in the domain industry. So if we had of lost our domains, our resellers would have lost their domains to our competitors, wiping out another 3,000 domain businesses.”

    And the timeframe to restore services? 24 hours.

    “My brother and I knew at this point that our business was gone.”

    The men contacted NetRegistry, which had expressed interest in buying the business before the disaster. A full acquisition of its assets by NetRegistry was negotiated within 24 hours.

    “Next morning, the people from NetRegistry came down to Melbourne, we finished signing off contracts and they handed us a cheque for the business which was significantly less than the offer they had made us three months prior.

    “We handed over the key to the business. My brother and I took our little cheque and handed it over to the bank. It unfortunately wasn’t enough to cover our obligations so we had to go through a full liquidation and we were out of business.

    “Between Alex and I, we had $6 in our pocket – we went down to the local coffee shop – bought a couple of coffees, sat down, didn’t say anything to each other for two hours and thought: 'What did we do wrong?'”

    -------------------- Page 3 --------------------

    The aftermath

    During the attack, Distribute.IT received an anonymous email from someone claiming to know who was responsible for the attack, including the person’s name, address and phone number.

    The email was sent to the AFP. Three weeks later the AFP arrested an unemployed truck driver from NSW, who it had put under surveillance after receiving the email.

    “While he was trying to cover his tracks with our hack, he was also preparing to do something else. They raided him on a Friday night, and within an hour of him pushing the button and doing what he did to us to another 360 networks across Australia.

    “They arrested him for that and he ended up doing two and a half years [in gaol] for that hack. But he was never formally charged with our [hack]. He was an IT nutter, sat at a computer for 20 hours a day, hated the world, applied for IT jobs; not with us, we were his random victim.

    "He learnt his trade from [videos] on YouTube and from [other hackers] on chat sites."

    So how did he get in? He hacked Distribute.IT’s VoIP system through a simple software vulnerability, and its website was taken down through a vulnerability in the PHP [programming language].

    But it was the third breach that was key to this hack.

    “We focused on efforts on the network itself, rebuilding the network, putting the security around it. What we missed during this period was what came from outside.

    “The [hacker] had been in contact with the company and made a couple of relationships internally – a bit of social engineering. He somehow managed to isolate a vulnerable staff member in IT.

    “He had conversations with that staff member and they exchanged emails. That staff member was second in charge in our IT area. As part of that exchange, he managed to put some malware on his [the staff member’s] personal laptop at home.

    “During the second week [of the hack], he left himself a keyhole into the laptop and used that keyhole access to get control of the laptop and then the secure VPN back to the network.”

    Woerndle says the way in which you manage the early stages of a hacking incident will have a big bearing on the outcome. Distribute.IT’s decision to take down its network after the first breach alerted the hacker.

    “So whatever he had in mind, he had to initiate it at that point or he would have run out of time. In retrospect, what I should have done was the complete opposite; it’s one of the hardest human emotions to watch what goes on.

    “That’s the point in time where you get forensics involved, have a look around the network, see where those entry points were and build up a real case against the perpetrator,” said Woerndle.

    So how have the brothers recovered personally and professionally from this incident?

    “It was a perfect storm of events because we had plans to open up in China and were preparing for an acquisition. So after nine years, in early 2011, was the first time we put our homes up against our business.

    “When this event took place, Alex and I lost our homes. The single hardest thing to this day was me going back home a day after the end of it – I hadn’t seen my family in three weeks – and telling them we had lost everything,” he says.

    “We spent nine years on that business, Alex and I worked seven days a week on that business. We took four weeks holiday during that whole period, we put our heart and soul into that.”

    “It took Alex and I six to 12 months to get over it. To my brother’s credit, it was him who came along one day and said, ‘You know what, nobody ever talks about this stuff. There are a lot of lessons there for other people, maybe we should talk about it.’

    “I remember looking at him thinking, ‘Are you mad?’”

    Carl has recovered and still has an entrepreneurial spirit. He has a few “software plays in the background” that he is trying to develop.

    “It’s a long journey back,” he says.

    Speaking to others about his experience has certainly been a positive step.
    __________________
     
    Last edited: May 14, 2014
  2. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    12,885
    Kudos on him for talking about it, But I'd have liked more details, such as what he would have done differently. Hopefully there is more to come.
     
  3. RyoSaeba

    RyoSaeba Member

    Joined:
    Sep 11, 2001
    Messages:
    12,313
    Location:
    Perth
    Sickening to read people who does these things maliciously. All the lives completely screwed up by this one person, for no good reason either.
     
  4. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,408
    Location:
    qld.au
    Wow, so the hacker had also done some social engineering as well.

    I felt sorry for them at the time and feel even more so now. Sure there was probably some bad things they did security wise (I bet they're not the only ones) but nobody deserves this.

    The badgering of the IT people by media isn't kind either and well outside of the job description for most sysadmins. I can understand why they had some of them quit.
     
  5. chip

    chip Member

    Joined:
    Dec 24, 2001
    Messages:
    3,582
    Location:
    Pooraka Maccas drivethrough
    Well they do get people messed up by drugs to talk to school kids about how drugs can mess you up.
     
  6. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    12,885
    But the anti-drug foundations don't pay a 6 figure salary to the ex-junkies :).

    I don't like how little responsibility he seems to accept from it. He's not blameless in the whole saga.
     
  7. QuakeDude

    QuakeDude ooooh weeee ooooh

    Joined:
    Aug 4, 2004
    Messages:
    8,425
    Location:
    Melbourne
    6 Figures? Where did you read that?
     
  8. Dr Evil

    Dr Evil Member

    Joined:
    Oct 20, 2010
    Messages:
    2,855
    Location:
    Perth
    From what they know it was mostly compromised from the basic method as 'Social Engineering', which was then used to secretly manipulate the other end user through an email (as I could only assume, since it states 'they exchanged emails').. seems a bit..

    Good enough reason for some. It's no different to someone 'skimming' CC's or general theft to small/large companies that deal with electronic funds.
     
  9. zero_velocity

    zero_velocity Member

    Joined:
    Sep 16, 2010
    Messages:
    2,536
    Location:
    QLD
    Wow what a read.

    Im actually surprised that non-critical infrastructures were re-opened after the network rebuilds (i.e. the VPN's for PERSONAL computers).

    Its a tough lesson to learn but IMO they should have only rebuilt the necessities to keep the business running.
     
  10. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    12,885
    Educated guess at what a Cyber Security Advisor at Deloitte would expect to earn.
     
  11. zero_velocity

    zero_velocity Member

    Joined:
    Sep 16, 2010
    Messages:
    2,536
    Location:
    QLD
    Actually, they do. Who do you thinks runs and makes the majority of AA meeting groups, sponsors and other drug support systems?

    Recovering addicts....

    Simply because they have BEEN there and they've DONE that, earning them experience.

    Would I hire a security guard thats been beaten up a bit (and therefore EXPERIENCED) or the green guys on the block who've got no idea what happens when shit hits the fan.

    Reality is, with his experience he would make an ideal leader in a reproduction of the same situation, he would know how to handle a team under pressure and guide the company to an ideal solution.

    He doesn't need to know the solution off the top of his head, he just needs to be able to listen to the person in the team that does.... :thumbup:
     
  12. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    12,885
    Given how much companies I've worked for have paid Deloitte for in the past, It would surprise me greatly if their Dunny cleaners were on less than 100K.
     
  13. chip

    chip Member

    Joined:
    Dec 24, 2001
    Messages:
    3,582
    Location:
    Pooraka Maccas drivethrough
    Barry Dunning-Kruger: "Hi, I'm Barry, and my systems haven't been compromised for 43 days...as far as I know"
    Group: "Hi Barry!"
     
  14. ir0nhide

    ir0nhide Member

    Joined:
    Oct 24, 2003
    Messages:
    4,118
    Location:
    Adelaide
    Wow, just unbelievable how fragile their IT team was (CIO was basically the only guy holding it together) and how badly they reacted (burning out most of their staff on pointless exercises). Wonder what 'security measures' they put in place?

    Much as I would hate for this to happen to a business and their employees, it kinda sounds like it would have happened anyway at some point.
     
  15. idiot_child

    idiot_child Member

    Joined:
    Jan 11, 2003
    Messages:
    1,349
    Location:
    Sydney
    "The hacker also embedded a program inside its network and ran a command called 'rm-rf', which removes the file names and leaves scrambled data on the drives, making them useless."

    It's so cute when journalists talk technical. And yet oddly accurate.

    If I was Deloitte and there was a former C*O available for hire for $99k or less (the disbelief that they're on a six figure salary), bargain! That's like grunt/senior grunt pay.
     
  16. cbb1935

    cbb1935 Guest

    To me it reads like it was inexperience and reliance on "their own software", instead of tried and trusted solutions which boned them. Absolute amateur hour effort by all involved.

    Surely they didn't own 3 data centres, but rather racks or space IN a data centre.

    So (at least from how it reads), they knew someone was in their network, allowed them to F around, before finally acting on it, did NOTHING to ensure the integrity of the network before putting it back online, and then wonder why they completely had their pants pulled down again?

    If I'm getting the gist correctly, it was a disaster waiting to happen and was a matter of when, not if they got hammered.

    Probably had everything sitting behind a cheap router/firewall with default passwords.

    and I loooove the excuse about having ports open too LOL.

    But the icing on the cake, is someone now employing the bloke as a cyber security advisor.

    Crikey!

    Maybe they should start a new company up and call it Upthesh.IT



    ..............

    On another note ... David Cecil might be out of prison end of this year. Was facing up to 12 years. Idiot judge gives him 2 year and 6 months, with non parole period of 12 months. That was around June 2012.
     
    Last edited by a moderator: May 13, 2014
  17. power

    power Member

    Joined:
    Apr 20, 2002
    Messages:
    57,072
    Location:
    brisbane
    They sound like a startup that didn't know the appropriate time to cash in their chips.
     
  18. sjp770

    sjp770 Member

    Joined:
    Apr 23, 2009
    Messages:
    1,717
    Location:
    Sticks, NSW
    The part that said it destroyed 4 levels of backups.... wouldn't you take a full backup of the network offline so it was not vulnerable? Scan it thoroughly to make sure there were no scripts /hacks etc waiting to come back?

    I assume that 72 hr days may have included some work from home over VPN hence the thought to keep it open but really? Check every system with full access before giving it access again.

    The idea that he should have let the guy poke around... I would think that only reason to do that is if you can only see how he is accessing the network while he was actively connected so you could then block him effectively. But who know what other access he could have opened or learnt of while he was there...
     
  19. ir0nhide

    ir0nhide Member

    Joined:
    Oct 24, 2003
    Messages:
    4,118
    Location:
    Adelaide
    They had no forensics and thought firewalling could save them. 1990 called wanting its security methods back :lol:
     
  20. Luke212

    Luke212 Member

    Joined:
    Feb 26, 2003
    Messages:
    9,466
    Location:
    Sydney
    like mt gox

    unlikely thats what they meant. 4 separate backups more like it.. eg database, files, image etc. probably all stored on a network drive or attached usb.

    upon first detection, they should have immediately taken detached backups and kept them detached.

    but either way they were fucked because it sounded like the guy had a program running that would sit inside the network through the employees laptop VPN.

    it is definitely understandable though, i think they did not realise the severity of the issue. hindsight is hindsight.
     
    Last edited: May 13, 2014

Share This Page