Comment by sho
2 months ago
Wow. This is a cautionary tale. I don't think I'd be as devastated as this poor chap, but as it grew I realize I've allowed my iCloud photo library to become a single copy.
How are people handling this these days? If i wanted to ensure a full backup of everything on my iCloud to a NAS, what's the best way these days? Seems like they make it difficult by design..
I self host an Immich [1] instance to backup photos on my iPhone. It’s OSS and has a level of polish I’ve rarely seen in free software. Really, it’s shockingly good. The iOS app whisks my photo off to my home server several times per day.
What I’m not sure about is how to backup things like iMessages, Notes, and my Contacts. Every time I’ve looked, it appears the only options are random GitHub scripts that have reverse engineered the iMessage database.
1. https://immich.app/
The imessage db is literally just a sqlite db. If you have a Mac you can read the entire thing with an applescript. It’s really easy from what I remember from years ago
I run a nextcloud [1] instance and use it for contacts, calendars, and reminders
1. https://nextcloud.com
I use Nextcloud for files/contacts/calendar/etc. as well, but for photos I use PhotopPrism [1].
The reason is simple: photos require much more processing and focus on performance. In addition, photos take up much more space, so while my Nextcloud instance runs on an SSD, the photos reside on an HDD, mostly in sleep mode.
[1] https://www.photoprism.app
What's wrong with `imessage-exporter`?
https://github.com/ReagentX/imessage-exporter
One rather counter intuitive way to “backup” your photos is to install Google Photos and One Drive on your iPhone!
Google and MS don’t charge as much as Apple for storage, and you probably need you need to pay beyond the free limits, but it’s not a huge expense.
Once your installed Google Photos and One Drive on your iPhone, just tell the apps to sync all your photos all the time!
Now I appreciate that isn’t for everyone.
But it works, is reliable, and requires no technical knowledge of running your own service.
The other thing to do is setup a Mac that synchs all your iCloud data, One Drive documents and Google Drive.
Then back up that device with Backblaze.
This gets expensive as a Mac with decent levels of storage isn’t cheap!
I live in fear everyday or my primary Apple and Google accounts getting locked!
I’ve had accounts since day one of iTools and very shortly after Gmail launched….
The issue with OneDrive is that it doesn’t store metadata like the photo location, its damn near useless. But I do pay for storage for Google Photos and iCloud.
If you take all of your photos from your phone, you don’t need your Mac at all. Google Photos will sync directly.
I wouldn’t use BackBlaze (the $7 a month service). It doesn’t support NAS at all and it has to phone home every 30 days or it will erase anything that is stored on external drive.
I would use an app that backs up to their B2 service.
I personally just use my personal AWS account to back up my Plex media and just use the AWS s3 sync command using the AWS CLI and store everything in S3 Deep Archive. It’s less than $2 a month for 2TB.
I’m suggesting you do both One Drive and Google!
Also, to be clear, I’m saying to install One Drive and Google Photos on your iPhone.
My approach to this is based around having the least amount of things to manage.
In my scenario, I’m looking for the most simple out of the box systems to backing up that don’t require any self hosted solutions or NAS management and so on.
Just throw money at it!
Hence my suggestion to get a Mac, say a Mac Mini, with a decent sized drive and just sync everything to it.
Fully sync so it downloads the lot and need off loads.
The backup the internal drive with Backblaze.
I appreciate this advice isn’t for everyone, and it may not be the best solution.
But it’s a way to a least have some ultimate disaster recovery in place!
It may not be the cheapest, and cheapest doesn’t mean best. It may not be the best by some other measure of features, but it works and requires zero knowledge and additional hardware.
At least with some kind of backup, everything is not lost!
You don’t have to manage it or think about it. The services will sync for years without intervention until you upgrade your devices. For most people, that’s the important factor in having multi backups of their photos and documents!
2 replies →
Backblaze doesn’t erase after 30 days… I’ve had a computer be offline from it for several months and it still retained all data. And you can use the backblaze docker container to run on a NAS, much much much cheaper than B2.
Wasabi is much cheaper than AWS as well.
Finally the best solution for backing up your iCloud Photos is definitely Immich. Set it up on your own NAS or a VPS, back up to that, and then back up that server to an S3 storage using rsync or restic. I’ll note that I still backup to Backblaze because its so dang cheap.
I spent months trying to find the best setup a few months ago and this is by far the cheapest.
But still, this shouldn’t be required for normal people. They should get what they pay for.
4 replies →
> It’s less than $2 a month for 2TB.
What would be the egress fee to get your data back in case of disaster?
1 reply →
I run a separate Mac Mini that has the full iCloud Photos library on a massive external drive, set to "Download originals". I then rsync that filesystem to a separate Linux box. This works but you must not ever disconnect the external drive.
I don't have a solution for iCloud Drive, as there wasn't a keep offline setting last time I checked. So use it only ephemerally.
At least as of Sequoia, the Settings > iCloud > Drive > Optimize Mac Storage option enables iCloud Drive files to be stored offline. Likewise, right clicking any iCloud Drive files in the Finder includes a Keep Downloaded option. Since I minimally use iCloud Drive, in the past (older OSes) I also had Hazel make copies of iCloud Drive files so they were certain to be in backups.
Arq [1] has an option to "materialize" dataless files, basically forcing them to be locally available. The only issue is if it's a large file and it gets pushed off device often, you can burn a lot of bandwidth re-downloading it over and over again.
1. https://www.arqbackup.com
For iCloud Drive have a look at rclone. You can run it straight from your Linux machine
Time Machine backups to a samba share on the Linux box would get you both the Photos library database and the iCloud Drive stuff. It also means you don't need to bother with the external drive.
There is a keep all files offline setting for iCloud Drive (turn off "Optimize Mac Storage" in Systems Settings).
I'm not familiar with the "Photos Library.app", but I have an m4 mini with my photos in a Photo's Library. I'd love to know your script to rsync the photos into a separate drive/directory
The Photos library "file" is just a big folder, I just sync the whole thing.
#!/bin/sh rsync --iconv=utf-8-mac,utf-8 -avh --delete-after --partial --progress /Volumes/myExternalDrive/Photos\ Library.photoslibrary myuser@mylinuxmachine.local:"/srv/myExternalDriveBackup/"
(note: tested with brew rsync, IIRC the default rsync is outdated on macOS)
Somewhere in the directory structure is a folder /originals/ which has all the actual files.
Note that this is only a last resort backup. Restoring the library as a whole requires a Mac with a compatible OS version. Restoring without a Mac would only get you the originals, so only the out-of-camera files (jpg, heic, raw), with no edits or metadata changes from Apple Photos applied (Apple Photos doesn't touch the EXIF data). You'd probably also lose the video part of all live photos, as the live video files stored as separate files and not part of the .heic files. They're there, but not very usable.
An alternative to this workflow is to export all photos (with edits applied) from the Photos app, but honestly I'm not sure if that even works and how long it would take for multi-TB libraries.
I run Arq Backup automatically in the background.
It copy Photos, iCloud files and my mails once every days to S3 with incremental backups.
It requires to have a full copy locally.
Works great!
It is not hard to configure once, with the proper folders and settings.
> It requires to have a full copy locally.
yeah that's the thing. When my iPhotos library exceeded 1TB I lost the ability to store the full local copies. Since then, iCloud itself has been the sole source.
Looks like there's some decent, reasonably priced apps to handle this like https://apps.apple.com/us/app/parachute-backup/id6748614170?... (no affiliation)
You can request an archive of all your data (including photos and drive) in 25gb chunks.
https://support.apple.com/en-us/102208
I recently rebuilt my home server as an unraid machine. Currently it’s mainly torrents and a Minecraft server but it’s got 10tb of locally redundant storage with a sightline to scale that to around 24tb, so it would be a logical place to store a full gphotos copy.
Thanks, I have the same problem and need to do something about it.
I wonder if it can calculate (estimate) how big of an external disk I'll need. My wife and I each have 40-50k photos and a few thousand videos in iCloud Photos.
5 replies →
10TB external harddrives are relatively affordable.
I run a Synology NAS with a docker container that periodically downloads new iCloud Photos to a local directory.
this? https://github.com/boredazfcuk/docker-icloudpd
seems pretty high touch. A lot of hoop-jumping if you don't have a mac in the middle
Thanks. I had no idea something like that existed.
How do we know using such a tool won’t trigger an account lockout? How ironic would that be.
1 reply →
Yeah that’s the one.
I do have a Mac so it didn’t seem difficult to me, but I accept it will be for those that don’t.
I'd like to give a special shoutout to the PhotoSync app. It has one killer feature that Immich does not (at least last time I looked): encryption at rest. I think someone breaking into my house and stealing my NAS is a real possibility (unlikely, but I'd give it higher odds that getting locked out of my account like what happened in the article), so this is super important to me.
You could put Immich data on a LUKS volume I suppose, but then you have to fiddle with your server every time it reboots.
I did PhotoSync for a while, but now I just set up my Mac to download my whole photos library, and do Time Machine backups of my Mac. This gets two copies of the data not tied to my Apple ID (the one on my Mac's local disk, and the one on my NAS on the time machine volume).
immich is an extremely polished, FOSS alternative to google/apple photos. It's an investment, but a 4 bay NAS running immich should do nicely. Additionally I backup snapshots to Backblaze B2 via restic which runs another $5/TB
For me personally Immich is a non-starter because its not end-to-end encrypted.
It runs on your own hardware. There is nobody else who has access to unencrypted data.
1 reply →
Why would you need it to be end to end encrypted anyway? You’re running it. Set it to only upload photos when you’re on your home network and you’re fine. Or fork it and make a PR and make it e2e encrypted.
3 replies →
I simply manually periodically download everything to disk/software raid. Really important/sentimental stuff like baby photos and videos I have on DVD with par2s.
> How are people handling this these days?
Syncthing is wonderful, and does a great job of syncing between an Android phone's photos/videos and a laptop. And if you have regular automated backups of the laptop, you'll have backups of the photos/videos too.
For an iPhone, perhaps you could use iTunes to sync to a computer and back up that computer.
sushtrain seems like the best option for syncthing at the moment. its a bit more polished than mobius. neither of them sync in the background but i think i remember seeing someone using shortcuts to open the sushitrain app every now and again to wake it up so it would sync
Sync to Dropbox -> Dropbox hourly & monthly backups to my NAS using Bvckup2.
(One of these days I’ll setup my NAS to backup offsite fo a #3 backup).
I know that others with Macbooks sync their whole library to their Macbook and then Time Machine to a NAS as their copy #2. Is this vulnerable to the problem in TFA?
I keep copies of any important stuff i need on my server, and in a few hard drives at my home. i don't use any "cloud".
Back in the iPhoto days I used to symlink the library to an external drive.
Not an iCloud user, but I use Immich on my NAS.