Another Cryptolocker

Discussion in 'Troubleshooting Help' started by th3_hawk, Oct 19, 2019.

  1. th3_hawk

    th3_hawk Member

    Joined:
    Jun 4, 2005
    Messages:
    1,926
    Location:
    [VIC] Eastern Suburbs
    So it was almost inevitable that I'd get hit eventually. Seems my QNAP NAS has been hit with what looks like an 'eCh0raix' variant or something along those lines. Lots of files encrypted with a .locker16 extension. Google hasn't turned up anything that has provided anything useful, but it shouldn't matter too much anyway.

    The good news is that all the important stuff was last backed up on Sunday, including all the photos, the backup has been tested and it's all there and working nicely for when I'm ready to reload it all to the NAS. For now it's sitting quietly waiting.


    But what is odd is that it seems that only some files were encrypted during the evening of the 17th then nothing more. It seems I've lost a couple of Windows VMs, but none of the Centos ones (all sitting in the same folder). (even though they are all the same format).
    An old iTunes backup seems to have all its album art encrypted as was some random photos buried in a backup of a photo book. But then the folders labelled "photos and videos" is untouched.
    The entire media library also appears to be fine.

    Despite me turning off all port forwarding there appears to be something dodgy on the NAS as it's opening up some ports via UPNP then getting hammered with log in attempts (which are all failing since I've disabled all accounts and reset the passwords for good measure). I've now firewalled off that device so it can't access the internet anymore. I've also disabled all services while I attempt to suck some of that media off the drives.

    It's almost like the encryption was done file by file and only while the remote link was up. No idea why all the issues appear to be at 8:30pm on the 17th then nothing either since I didn't discover it for about 24 hours.

    I'm in the midst of pulling off the media library since it's not backed up, most of it could be re-ripped from disc or re-acquired I'm sure, but when it's all just sitting there I sort of want to save it if I can, the downside being that I had to run out and purchase another 8TB drive which I didn't really want to spend money on. I'd really like to be able to just clean up the infection, but then I don't know if I would trust that machine anymore until I nuke it from orbit then start again from scratch.

    On a side note, copying multiple TB of data takes so so long, but while it will take a couple of days to restore everything, I should have zero loss of anything important and very minimal loss of anything else (although that last bit is pure luck).

    On a positive note, this will give me the opportunity to re structure the way things are stored on the NAS, something which has been 5 years in the making. The plan is to have two arrays instead of one (maybe one separate and single disk) so that I can put the IPCams and live services on that one drive and hopefully let the rest of the drives spin down when they are not actively being used.
     
    Last edited: Oct 19, 2019
  2. havabeer

    havabeer Member

    Joined:
    Dec 12, 2010
    Messages:
    5,236
    well this is a good reminder for me to go back up atleast the photos I have on my nas
     
  3. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    39,246
    Location:
    Brisbane
    Why "inevitable"? What about your setup makes it a given?
     
  4. demiurge3141

    demiurge3141 Member

    Joined:
    Aug 19, 2005
    Messages:
    1,644
    Location:
    Melbourne 3073
    Probably should've run zfs on your Nas.
     
  5. argent

    argent Member

    Joined:
    Mar 20, 2003
    Messages:
    2,355
    Location:
    adl.sa.au:5063
    What was the infection vector?
     
  6. fad

    fad Member

    Joined:
    Jun 26, 2001
    Messages:
    2,312
    Location:
    City, Canberra, Australia
    I just lost my entire array too, but it’s the raid controllers fault. I have a full from a week ago.

    Backups on seperate systems for the win.

    Was it via a public port?
     
  7. OP
    OP
    th3_hawk

    th3_hawk Member

    Joined:
    Jun 4, 2005
    Messages:
    1,926
    Location:
    [VIC] Eastern Suburbs
    I figure as careful as I might be and as up to date as I keep things there is always a chance that eventually someone, somewhere will manage to sneak through somewhere. If anything this is the cautionary tale of keeping good backups, one I've already used with one family member who (on my advice) got a portable HDD to backup her photos from her computer for safe keeping a few months back... which is still in its box :( Hopefully after todays conversation she will actually use it (then unplug it).

    I'm not sure, but here has been some noise lately about vulnerabilities in the QNAP systems and as it turns out I was about a month behind on updates (assuming that was the cause anyway).

    I have to assume it was via a public port, there was a number of ports open for different services that have been in place for years and years with strong passwords and limited access, but apparently something got through somewhere.

    I have the old NAS in another part of the house that still powers on once per week and does an incremental and versioned backup then shuts down again. That one has zero services or need to connect to the internet so has far less potential for intrusion. I'm thinking I might firewall it off so it just can't access the internet at all (since it doesn't need to).

    I do have one incoming service which transfers a daily backup of a website I manage, I might have to look into how that is working and see if there is a more secure way for that too. At the moment it's split into two parts, a cron job which rsyncs the latest files from the remote server to me and a second job which pushes the sql database via ftp. I could use S3 or something else entirely to get my server completely out of the loop, but I like having a copy of my stuff locally. In writing this I suppose I could go the other direction and pull the information rather than send it which should remove the need for any ports to be open for this... something to look into when settings things back up.

    I'm also looking at another Raspberry Pi for other services that can be migrated off the NAS, while they will still need to connect to the NAS I can limit the access for that account and therefore the damage that can be done should that ever be compromised. (It also means there is more chance of the NAS getting to power down when not doing things).

    That should remove *most* of the port forwards, and accessing the network via VPN is already in place, but there is still those couple of requirements for external access to things that are likely to need to keep some ports open. But something I will review again as I rebuild over the next week.
     
  8. de_overfiend

    de_overfiend Member

    Joined:
    Jul 12, 2001
    Messages:
    2,319
    Location:
    Gold Coast
  9. OP
    OP
    th3_hawk

    th3_hawk Member

    Joined:
    Jun 4, 2005
    Messages:
    1,926
    Location:
    [VIC] Eastern Suburbs
    I pulled a bunch of the clean files off that I wanted to keep, wiped the whole NAS and rebuilt the array. I'm about 80% through restoring files and have tightened security (something I probably should have done long ago).

    The firewall is currently locked down so only a single fixed IP can get through from my webserver back home for automated backups, I will evaluate if any other ports need to be opened or not, otherwise any other external access will require a VPN.

    I've also ordered another raspberry pi to setup some of the things I was running on the NAS. This is partly to remove another potential security hole from the NAS but mostly to separate the requirements so if there are ever issues with the NAS (or it gets upgraded/replaced) I don't have to screw about re-setting those things. There are backups of configuration for these things already, but stand alone appliances sound like a good plan and will provide some redundancy (I will end up with at least two Pis).

    Next steps beyond that are to reconfigure the individual machines to point at the newly configured NAS and also sort the backup NAS to continue it's backup regime.
     
    Last edited: Oct 22, 2019
  10. ruffdayz

    ruffdayz Member

    Joined:
    May 27, 2017
    Messages:
    1,157
    Tis why I have 2 NAS drives. One for TV/Movies/etc, and another for personal data.

    Personal data one is on a completely different subnet and only accessible internally.
     
  11. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    39,246
    Location:
    Brisbane
    I have to wonder how. Are you allowing remote access to this thing? Is it doing some sort of uPnP to open up ports?

    Or is the cryptolocker initiating from a workstation inside your network, in which case it's not the NAS, but rather end point security that's the problem?

    I don't consider cryptolocker "inevitable" at all. It's entirely preventable, and a few standard safety practices, and any network should be safe.
     
  12. dakiller

    dakiller (Oscillating & Impeding)

    Joined:
    Jun 27, 2001
    Messages:
    7,958
    Location:
    Gippsland
    This is what I do, 12 hourly snapshots are kept going back 6 months.
     
  13. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    39,246
    Location:
    Brisbane
    I do decreasing resolution.

    * Every 30 minutes for 24 hours
    * Daily for a week
    * Weekly for "a month" (i.e.: 5 weeks)
    * Monthly for a year

    I don't generally need something from exactly 8573 hours ago.
     
  14. dakiller

    dakiller (Oscillating & Impeding)

    Joined:
    Jun 27, 2001
    Messages:
    7,958
    Location:
    Gippsland
    Freenas UI doesn't have that much control, so that's why mine is like that. 90% of the diffs are <1MB, and I have yet to even have a need for going back to get something in over 3 years now.
     
    elvis likes this.
  15. demiurge3141

    demiurge3141 Member

    Joined:
    Aug 19, 2005
    Messages:
    1,644
    Location:
    Melbourne 3073
    You can set up multiple automatic snapshots that would do the trick.
     
  16. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    39,246
    Location:
    Brisbane
    This is how I've done it on larger enterprise ZFS arrays. 4 separate schedules, as there's no way to set up complex rules for a single schedule.

    But all good either way. If your deltas are low, no harm in lots of snapshots. The only time that's an issue is when you need to delete stuff to reclaim space, and you don't want to have to delete a million snapshots.
     
  17. OP
    OP
    th3_hawk

    th3_hawk Member

    Joined:
    Jun 4, 2005
    Messages:
    1,926
    Location:
    [VIC] Eastern Suburbs
    Yep, remote access to things that was clearly too open + uPNP is on (and the NAS was opening ports itself after I found it was compromised). It is locked down far more tightly now.

    I do not believe it was an internal breach from another machine since there were affected files outside the shared folder (eg the VM HDD images), that and all other users on the network do not have access to all the shares either (and my machine isn't compromised).

    Inevitable maybe is not quite the right word, but with a family full of users, each on their own machines there is always a possibility it is going to happen to one of them eventually despite all the guidance over the years.
    In this case, it appears to be my own fault with one too many open ports due to a little complacency on my part, a discovered exploit and a firmware that was ~30 days out of date. I will hopefully not make the same mistakes again.
     

Share This Page

Advertisement: