Hey all, i would like to get some feedback on my backup strategy.
I have a debian webserver with a ZFS pool running nextcloud aio, immich and jellyfin. Thinking about adding other services as well but nextcloud and immich are the most important ones. The docker volumes of these services point of course to the zfs pool. My backup strategy would now be to use the internal backup solutions for nextcloud and immich to backup their databases, then stop the docker containers and do a borg backup of the zfs pool. The backups would be stored an extern hard drive (I want to expand on this but for now this is all I can afford). is this a viable approach or do i miss something? Could there be problems in case of a backup with the databases etc? The docker compose files are also stored on another machines together with my server documentation.
You must log in or register to comment.
Seems pretty plausible, not 3-2-1 yet, but on the way, and should get the habits established well enough. Just having an offline backup is a huge step up from most. Consider a waterproof box (perhaps buried) in the back yard instead of just another room (in case of fire / flood).
If you have a friend with a similar setup, or who perhaps wants one, you can sync over internet and both get your offsite without the expense of online backups or the inconvenience of lugging HDDs around.
If you don’t have off site backups then anything that destroys your entire site (fire, flood, earthquake) destroys all of your data.
yes this is supposed to be an off site backup. I want to backup everything with this on an external drive (later more) and keep the drive first in another room. If i can later afford more drives i want to store on at a friends place
For backups, use 3-2-1. 3 different copies on two different kinds of media, 1 of which is off-site (in a safety deposit box, for example)
Also, since you’re using ZFS I’d also add a rotation of snapshots. They don’t use much space and it’s a lot less work to recover recently deleted files from a snapshot than going through the trouble of getting to you backups.
1 of which is off-site (in a safety deposit box, for example)
A potential compromise these days may be a block storage service. Safety deposit box is good, but because of it’s inconvenience people are very likely to do it seldom, which defeats the purpose.
But the snapshots are only on the ZFS filesystem? or do you mean i should replace the borgbackup with the zfs snapshots?
You can use the
zfs sendcommand to copy snapshots from one dataset to another.Your backup could be a ZFS dataset stored on an external drive(s) which would contain the snapshots of your online dataset. You could then encrypt and compress (by setting the appropriate ZFS dataset properties) the backup dataset for size efficiency, and security.
To restore the backup you would use
zfs sendto move your backedup snapshots into a new dataset on your new un-disastered hardware.Since this is all done via CLI, you could write a bash script to create periodic snapshots, one to backup snapshots to the external dataset and another to delete old snapshots in your dataset. Toss 'em in your cron service of choice (or use systemd timers) and you’ve got a whole ZFS native backup system.
There may be backup software that’ll do this for you. I’ve seen that Timeshift supports snapshot-based backups for btrfs so you can probably find a GUI app to handle the automation.
If you are storing everything on the ZFS filesystem, taking a snapshot in ZFS will include all that data. So if you keep hourly snapshots for the past 24 hours, and daily snapshots for the past week, and then monthly for 3 months for example, you can often dip into those to do recovery when the issue is “oops I deleted something I didn’t want to” rather than going to your huge backup and restoring the entire system
Yes, but zfs snapshots are not an off-site backup which is more what i am looking for. Besides this, zfs snapshots are of course something i want to implement
If you use zfs for docker, dbs and vms you don’t have to shut down anything. Just snapshot and send/recv to sync snapshots to another ZFS drive.
You can even mount and copy the latest snapshot to the cloud with rsync/rclone; probably also borg/restic/kopia. Each applications state will be internally consistent if the snapshot is performed for all data at the same time. If you’re paranoid you can stop everything for a few mins to perform the snapshot, but it’s not really necessary.
yes the ZFS snapshots are in the same disk, but the most common scenario when you need backups is to get a handful of files in which case the ZFS snapshots are super convenient and they use very little space. I use restic + (B2 | sftp) and zfs snapshots. I may literally go years without needing to restore from restic because most of the time I can get what I need from the zfs snapshots.
You did not mention if you are using a single disk or more. If you can afford it and the machine allows it, doing mirroring or RAID-Z1 (equivalent of RAID 5) is a good option
I have 4 HDDS drives running in RAID-Z1. But yeah, the zfs snapshots on the pool itself are a no brainer :) but good to hear that they work so well
If you wanted to get really in the weeds of ZFS, you can use ZFS send to send copies of your snapshots into a dataset that you store on your external.
You can enable encryption and compression on the external dataset as well.
This would use snapshots, give you the ability to make block-level incremental backups and allow encryption and compression using only ZFS tooling.
You’d have to script it though (it’s possible someone has already done this in some other backup application).
If your databases are on zfs, doing backups of both just duplicates data for no reason.
You also need to protect from accidental deletion. It should be difficult or impossible to delete a backup.
Use a cloud host for offsite storage unless you have a friend that will swap offsite backup storage with you.
I assume your ZFS system has plenty of redundancy. Thus, I would enable snapshots and do a regular rotation because ZFS snapshots are so easy to recover from for most disasters. What that leaves is the catastrophic losses from, say, your house burning down. So you still need something off-site, but the ZFS will cover most of your backup needs.
Yes, but what i am asking is more if my proposed strategy would work for an off site backup.
Suggest:
- Frequent ZFS snashots. There are scripts to make this easier like zfsSnap
- Two external backups which you rotate weekly [1]
- Instead of borg backup of ZFS pools if you have another machine you could sync the volumes to another machine or even use rsync to another machine of the data [2]
You did not mention where the target of the borg backup is, but you want an external service. I believe there is a service that works wells with borg backup, but have not used it.
Notes [1] Spinning disks are affordable. I suggest at least 2 because if you only have one and your machine was compromised, think disk encrypting malware, you disk may be encrypted too. Also, if the disk dies there goes your external drive backup
[2] If you have another machine with enough space to host a copy that is a good option. Also, there are services that offer backup/disk VMs. They have very slow CPUs and affordable disk. Those may be work checking
You did not mention where the target of the borg backup is, but you want an external service. I believe there is a service that works wells with borg backup, but have not used it.
I did write that the borg backups would be stored on an external drive, sorry that this point wasnt clearer.
I basicly for the start have one big usb 20tb hdd drive (it was discounted and costed only little more than the 12 tb same brand one) to store my backups and i plan to get more drives when i can afford them. The idea with an additional machine i had as well, but thats maybe something further down the road.
the question is that for the start, if a borgbackup of the zfs pool on an external drive with all those services would work. I am a bit worried about their databases yucking up or something.
