What is your backup and editing solution?

Discussion in 'Photography & Video' started by TwinII, Sep 24, 2020.

  1. TwinII

    TwinII Member

    Joined:
    Feb 8, 2005
    Messages:
    1,490
    Location:
    Brisbane
    Hi OCAU,

    After my computer failed a week or two ago, and another Windows re-install to fix it I want to look into setting up a better backup solution for my images - I probably have 12 years worth. Most are backed up but my process is not robust, timely or automated.

    The workflow I would like is:
    - Take photos
    - Transfer photos to Network Attached Storage Device (+/- some sort of local backup on my PC?? - DO I need this)
    - Edit my photos using Lightroom from any of my devices (PC, Laptop, Phone)
    - Export Photos to relevant online sites (Facebook, Flickr, website etc)

    What can you recommend? Can I also include a workflow where I also hae a local copy on one of my computers for redundancy?

    Currently I Take photos, Upload to my main PC, Edit on my main PC, Do random backup to USB external HDD's. On occasions when I am feeling lazy I will upload to my laptop to edit, but then I have to transfer to my PC and merge the LR catalog -and its annoying.
    I upload large images to my website (hosted with Zenfolio). I am thinking of uploading my high res large images to OneDrive also for backup purposes.

    I have no issues with the way I structure my photos - have had the same process for 10 years or so.

    Any thoughts or ideas on how your workflow is would be much appreciated.

    Thanks,

    Martin
     
  2. lonewolf1983

    lonewolf1983 Member

    Joined:
    Jun 12, 2002
    Messages:
    4,648
    Location:
    Perth, WA
    Sounds like a pretty standard workflow - personally i dont edit photos on network attached storage (if it had fast disks and 10gb/s link may be passable) but my current images are stored/worked on SSD/NVME storage for speed - as is my scratch disk/catalog.
    but if you want to share images/catalog with a laptop then you may have to do it that way.

    I use DAS (drobo) for my non-current storage - e.g. once processed they go there.
    Then an automated backup to my server (NAS basically) which hosts an automatic link to Carbonite for my offsite backup storage.

    Software wise check out free file sync, for either manual/scheduled backups, or it has a realtime sync option which monitors directory and copies by itself.
     
  3. Athiril

    Athiril Member

    Joined:
    Jun 11, 2003
    Messages:
    4,115
    Location:
    Tsumagoi-Mura, Japan
    I have a Lightroom 1tb subscription, I have about 400gb left. All my raws are backed up to the cloud, an internal drive, and an external drive. Also it means I can pull up and access anything from my catalogue on my phone, tablet, etc where ever I am. Lightroom with cloud storage will then always have an up to date backup, not just when I decide to make backups. I leave stuff on SD cards until its in multiple locations and on the cloud.

    Windows 10 has an automatic backup feature you can activate for specific times or specific locations too iirc. Also I should point out that RAID1 is not a backup, just disk redundancy, and whatever happens that isnt hardware failure to the first disk happens to the second disk as well.
     
  4. Frozen_Hell

    Frozen_Hell Member

    Joined:
    Sep 11, 2002
    Messages:
    3,007
    Location:
    Cairns
    Its also important to understand what failures you're trying to prevent against with your backups. Single disk failure or accidental deletion, then external backup drive will protect that.. ransomware'd though, it'd target any attached drives, so all done. Also if you had a break in, power surge or fire etc. then you'd also lose everything. Doing backups to the internet will be dependent on how good your internet connection is, I've been doing periodic backups to an Amazon S3 bucket - but I do have 100/40 HFC NBN which does actually reach those speeds and unlimited data.
     
  5. Xang

    Xang Member

    Joined:
    Apr 16, 2007
    Messages:
    1,080
    Edit on NVMe

    Backup to NAS

    Offsite to cloud (Backblaze + Amazon Glacier)
     
  6. MR CHILLED

    MR CHILLED D'oh!

    Joined:
    Jan 2, 2002
    Messages:
    156,222
    Location:
    Omicron Persei 8
    Main rig has primary copy of everything. Synology Nas has backup of everything I can't afford to lose. Still working on off site, perhaps work, location. Backed up monthly as core files that need keeping don't get created often enough to warrant anything more.
     
  7. lonewolf1983

    lonewolf1983 Member

    Joined:
    Jun 12, 2002
    Messages:
    4,648
    Location:
    Perth, WA
    Yep, RAID aint a backup, neither is anything like a mapped drive on a NAS (unless it supports versioning)
    Preferably you have your backup device not able to be accessed by your normal user account (e.g. it backs up with a seperate user account)

    The benefit of cloud based stuff is they tend to have good versioning support so should you get crypto you can get your files back easily
     
    BurningFeetMan likes this.
  8. OP
    OP
    TwinII

    TwinII Member

    Joined:
    Feb 8, 2005
    Messages:
    1,490
    Location:
    Brisbane
    Thanks for the advice everyone!
    One issue I noticed with my proposed workflow is you need your LR catalog on you local machine.
    I am thinking I will have an LR catalog on each of my computers (desktop and MacBook) for each year. I will merge them at the end of the year and then store on my main PC and a copy to a NAS. I will store my current USB HDD's off-site I think.
    I will have to look into cloud storage too. More and more I think this is the way to go. I will need at least 4 Terabytes I expect 0 at least!

    What cloud storage solution would you recommend? I find its a bit frustrating how certain software locks you into their product - e.g. Apple and iCloud; Windows and OneDrive; LR and Adobe's cloud subscription. I wish I could choose 1 cloud storage service and link to each of Apple, Microsoft Office/Windows, LR/PS etc easily. Is there a solution out there?

    Thanks,

    Martin
     
  9. Athiril

    Athiril Member

    Joined:
    Jun 11, 2003
    Messages:
    4,115
    Location:
    Tsumagoi-Mura, Japan
    Lightroom's cloud storage has major advantages, because the cloud side has to also run Lightroom to for that to work. Eg; sync your original images to the cloud and want to share or proof them via a link? Cloud side renders them (or already rendered), when you edit something only the edit metadata is uploaded, and the image will be rendered cloud side, so you dont need to keep reuploading images.

    Other cloud storage is just storage. Upload your files, and your catalogue file, redownload it if it goes ass up. But then you have to download the entire thing. In Lightroom, if you have a local copy of originals and lose everything, you really only need to download the smart previews you want to look at and edit, and not for the entire catalogue, let alone the originals. You can download the whole thing of course (internet permitting), but you can get back up and working right away before having to restore the original data.

    I will often plug my SD card into my ipad and transfer the originals into Lightroom, and let it sync in my bag. Pull it out and do some quick edits, or give people a share link to preview or select images - links are generated from cloud storage so you dont have to render, export, and upload somewhere to send a proofing or preview link to people, so I can do this often before I was even home to plug an SD card into a computer. This is handy if youre shooting a lot of portraits or people etc. Their selections can be filtered and made into an album without manually picking file names, and the edits you make dont need to be rendered, exported and uploaded to somewhere to be downloaded, just a link generated from Lightroom (only the edit metadata needs to be uploaded, edits reflect online almost instantly).

    I imagine 4tb would be quite costly though. I'm considering what to do once I hit 1tb. Its not so bad for me since my raw files arent huge, but that will change if I upgrade my camera.
    This is the rough cost here "How do I add more storage to my plan?
    Upgrade a 20 GB plan to 1 TB or boost your total storage to 2 TB, 5 TB or 10 TB, starting at A$14.29/month per terabyte."

    I think Google Drive is $125/month for 10tb, I couldnt find a 5tb or 4tb plan, only 2tb which is $12.49/month.
     
  10. Deftone2k

    Deftone2k In the Darkroom

    Joined:
    Sep 3, 2002
    Messages:
    15,800
    Location:
    Sydney
    2TB Working SSD Drive
    Working Drive Syncs to two local drives
    RAW Dump Drive (8TB) connected to my working machine. Stores up to around 1 years worth of ALL raws from shoots/travel etc)
    Finished edits/catalogs are kept locally on a yearly External Drive
    Finished edits/catalogs are kept offsite on a rotating yearly External Drive (have two I rotate when I visit my parents)
    Each year has 2 x External Drive backups of all delivered files/raws/catalogs (one onsite/one offsite)
    Dropbox Business Unlimited Cloud Backup
    Enough memory cards to shoot on fresh cards until jobs are delivered (Slot 1 is taken out of camera and used to copy/backup onto local machines and goes back into rotation. Slot 2 goes into a card wallet that I keep in the car until job is delivered)

    No cloud service does what you want AFAIK Martin. If you want file cloud your best bets are;

    Google Drive
    MS One Drive
    Dropbox
    Jottacloud
    Amazon (whatever form meets your needs).

    Cost is considerable but 5TB is manageable. Good to use in conjunction with local backups.

    You have other options like Backblaze, Carbonite etc etc but they all serve different models and methods of backup/syncing.

    I have about 17TB in the cloud at the moment, switching over to Dropbox Business.
     
    Last edited: Sep 26, 2020
  11. lonewolf1983

    lonewolf1983 Member

    Joined:
    Jun 12, 2002
    Messages:
    4,648
    Location:
    Perth, WA
    i personally chose carbonite unlimited as it offers what i need
    Unlimited space, automatic backup of anything i dump into monitored folders, and for me i wanted a private encryption key - and only ~$90 a year

    Plenty of options these days for cloud storage - just make sure you get something with versioning if you want it to act as a backup
     
  12. Athiril

    Athiril Member

    Joined:
    Jun 11, 2003
    Messages:
    4,115
    Location:
    Tsumagoi-Mura, Japan
    How much does this cost you? :Paranoid:
     
  13. Deftone2k

    Deftone2k In the Darkroom

    Joined:
    Sep 3, 2002
    Messages:
    15,800
    Location:
    Sydney
    About $1000 per year. Cost of doing business.
     
  14. Frozen_Hell

    Frozen_Hell Member

    Joined:
    Sep 11, 2002
    Messages:
    3,007
    Location:
    Cairns
    Probably one of your cheapest costs in the scheme of things too.
     
  15. Deftone2k

    Deftone2k In the Darkroom

    Joined:
    Sep 3, 2002
    Messages:
    15,800
    Location:
    Sydney
    Haha don't you know it. There are soooooooooo many things that add up each year (especially with everything going subscription base).

    I mean always going to do things properly, just all adds up $$$
     
  16. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    18,184
    Location:
    Canberra
    Paging elvis

    But typically I've set this up with either:
    • NAS-based shares w/ a cloud-sync from the NAS to cloud, all mobile devices access the cloud, local desktops access the NAS
    • Pure Cloud w/ Onedrive Files on Demand - all devices leverage the cloud/local cache (this alleviates performance issues for large datasets - as you cache your active jobs locally, and the sync client moves things in the background as you save/release the file lock)
    In both instances you are going to live/die by your upload/wan connectivity, and I would *highly* suggest you invest in at least 250/100 NBN connectivity to dramatically improve your workflow, as well as effective RPO/RTO.

    Not really sure how you'd achieve this effectively on OSX.
     
  17. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    43,136
    Location:
    Brisbane
    I'm glad you said you backed up locally too, because Adobe have once already deleted people's content by accident, and given their ineptitude will absolutely do it again:
    https://www.theverge.com/2020/8/20/...os-app-update-pictures-photos-presets-deleted

    After 20 years of dealing with a mix of large and small media companies, my patience for Adobe's uselessness as reached an all time low.

    This is a pretty good setup for a one-person business. Backblaze (and others) also allow for accidental deletion protection - you can set retention of varying times on files that are modified or deleted. Not that this will eat into your usage and thus cost, but it's great for peace of mind.

    Yup, all this too.

    My background is sysadmin work for media shops since forever. One thing that I consistently find is that the tech out there for media folks excels, but never gets used because "digital house keeping" is considered too much effort, right up until all the data is lost.

    The biggest thing you can do to help yourself is automation. Whatever can be scheduled can turn from "I need to do my backups" to "I need to check the backups", the latter of which is a fraction of the time. The least functional backup, regardless of technology, is the backup that is never done (which includes the automated backup that failed silently).

    Second to that, snapshots. A always see media people argue brand names of DAS and NAS endlessly, but the single biggest feature that seems to be missing from most recommended devices are snapshots. The slip of a mouse/wacom or an accidental delete of the raws instead of the compressed files, and you've lost enormous amounts of revenue. Let us not forget either that ransomware is by far the single biggest Internet based threat of 2020, and can reduce your primary and secondary backups to garbage in seconds (scrambling just enough of every file to make them garbage in seconds across TB of media, and then syncing immediately to your cloud storage). Read-only snapshots are one of the easiest mitigations for that today.

    And of course, keep a set offsite for the fire/flood/theft problem. I used to recommend external drives in rotation, but today cloud providers come in almost on parity for cost once you factor in your time and effort as well.

    The brands really don't matter. As above, get the features right - automation and snapshots/retention being the big "must haves" for anyone, big or small.


    The transfers to NAS/cloud are the sorts of things I do myself via tools like Rclone:
    https://rclone.org/

    The downside being that they're a bit nerdy and require some command line stuff to set up properly. The upside being that they're $0 and work flawlessly as either a script you can double-click and execute on demand, or something you can set via task scheduler. Rclone is especially useful in that you can point it at many TB of data, and it will copy only changed files, either to your NAS or to the cloud. Point it at the very top of your file system, and you know you get everything every time.

    Here's all the stuff Rclone supports:
    https://rclone.org/docs/

    Which includes an enormous list of cloud vendors as well as local SMB/network/direct-attach storage. So the nice thing there is you can set up several "remotes", and treat the all the same. Write yourself a simple script that copies local -> NAS, and local -> Cloud, and that's half the battle done.

    If you need help with such a script, sing out. Plenty of folks use the tool, and if you tag me I can show you some examples. However the docs do a pretty good job of showing you how to set up the remotes, and then the script becomes "rclone copy from:/path to:/path" each time, nice and simple.

    Exporting to Facebook/Flickr, I can't help you with. They're constantly moving targets and don't support simple APIs, so tools like rclone don't work. Often there's pay-for software that handles mass-media stuff (where you want to post to all your socials in one hit), but that's outside of my expertise.
     
    3Toed, 2SHY and NSanity like this.
  18. NismoR31

    NismoR31 Member

    Joined:
    Aug 3, 2006
    Messages:
    4,217
    Location:
    Sydney
    I import to pc, backup to nas which backs up to its own usb hdd (only accessible by a single user/process on the nas) and also to backblaze b2

    Don't need to worry about catalogues anymore. I use Capture One sessions which stay with the raws. Makes it easy to move to/from laptop if i need to.
     
  19. Athiril

    Athiril Member

    Joined:
    Jun 11, 2003
    Messages:
    4,115
    Location:
    Tsumagoi-Mura, Japan
    Just want to point out this is misinformation. Anything stored in the cloud was unaffected. This only affected people who stored all their originals on their iPhone (or iPad) and nowhere else - not SD cards, not their PC, and also didn’t sync any of their originals to the cloud - ie people using the free version of Lightroom (and thus didnt have cloud access) on their iPhone as a camera app etc.

    Anybody with a subscription who had synced all their images to the cloud or the originals on their PC - ie all photographers would be unaffected, none of us would solely store all our raw files inside of Lightroom (and not in iOS gallery) on an iPhone and erase all other copies. This would only affect you if losing your phone meant you lost all your images and couldn’t get them back.

    I didn’t open that version of Lightroom on my iPad, but if I did, it wouldn’t have affected me - I would have lost locally stored cached smart previews, that would be it.

    It would affect you if you were an iPhone photographer that decided to use the free version of Lightroom as a camera app and store all the images inside of - and never backed anything up to your pc. Etc.
     
    Last edited: Sep 30, 2020
  20. elvis

    elvis Old school old fool

    Joined:
    Jun 27, 2001
    Messages:
    43,136
    Location:
    Brisbane
    That Adobe are rubbish? Yeah nah, go check the email you use for AdobeCC on https://haveibeenpwned.com/ .

    If they're not deleting things be accident, they're leaking your private data. Buyer beware.
     

Share This Page

Advertisement: