Hello,
As the title suggests, how do you manage your DBs for docker services.
Do you spin a new DB for every new docker cluster or do you have a centralized DB that is accessible to the docker clusters.
What are the pros and cons of both method?
For the moment, I spin a new DB for every services as I feel it is easier to backup the service in case of a problem.
I use a single centralized database running directly on the host (postgres) and make it accessible to the docker clusters.
Pros: It’s all in one place to backup all important data for every service, I find this a much easier and more reliable way to backup all services, and I’m confident if you tried it you would too. The application data becomes a first-class citizen instead of hiding inside some nameless docker volume. Significantly less database overhead. Consistent database version and tools.
Cons: Lots. You need to manage and backup the database. You need to manage the database users and passwords too. Making the DB accessible to the docker clusters is nontrivial and can be fragile. You can no longer use the default “official” docker compose files, since they will almost never support an actual database service without several modifications, they’re always built for spinning up their own docker container database. So you’re going to be doing a bunch of work setting up users and passwords and performing extensive surgery on the docker-compose before you even start up the application, which adds a lot of friction, with lots of opportunity for error. All things considered, it’s actually quite painful. Technically if one application abuses your database hard enough and exhausts its memory to crash it or something it would affect other applications too, but that’s true of any services running on shared hardware abusing anything on that hardware, so it’s not a realistically concerning con.
I consider it worthwhile, but I might be wrong. Also I hate docker in general. I understand why people use it. It’s the same reason I use it. But I still hate it. I think system installations are so much easier to manage in the long run, but initially more work, and you need to invest that work at a time when you’re not even sure if you really want to run this application or if it’s going to be compatible with the rest of your environment. So docker is the easy solution. But then you’re basically trapped in dockerland. It’s not that bad, I just hate it in principle. I wish there were a better way.
Containers lower the bar since the developer doesn’t need to make their program work on every system - just the container’s system.
Price we pay for more programs. And they bring boons like read-only, rootless, limited capabilities, and constrained perf limits (esp if you use Podman with Quadlets).
And don’t feel trapped - the Dockerfile is a recipe to build that program. Probably want to do it in an LXC container since it’ll want to use /data for its data or something. But the LXC container can also be run as a user but the program thinks it’s root. Plenty of security abounds!
I think it’s worth the price and you’re not trapped. They’re trapped with you and your robust Quadlet files
Same, exactly what I do. For the part about backups, there are tools that make it really easy. I’m using databasus for instance. Once I had set up a couple applications, adding a new db, user, and backup config took just a few minutes every time really. In the end figuring out how to backup every docker stack’s individual DB sounds more complicated to me.
Edit: plus my first service hosted in this lab was my own matrix instance so I needed a solid DB, then it was already there so might as well use it.
I started doing the One True Database method because I got worried that the high write count on all the little db’s was abusing a raspberry pi’s SD card. Moved them all to a bigger server with NVME and mirroring to a RAID.
Not all the compose files make obvious how to reconfigure the db host. Homeassistant uses s a sqlite db built into the container, rather than a separate unit, but you can force it to use a remote db through its config file. May or may not be worth hiding db user/pass in a .env And sometimes there’s trouble restarting after power failure, depending on what order the database, pi, and various containers come back up.
I also feel it’s worthwhile. I feel better being able to check on all the databases. Feel better not writing to the SD card so much. Feel better offloading those megabytes and cpu cycles from the little pi. It’s been fun snooping through database structures. There have been a couple times where I decided to query one of the ccontain databases directly, or cross from one project to another, and it’s easier (for me) to give a different user privileges to the database and query some deep bit of data than to figure out how to extract it from an API or frontend.
I’m not even running that many services, but why would I want the overhead of 6 separate mysql instances when I could just have one?
depends_on solves this problem
That’s a good perspective.
I am the same. But many services offer docker as the main installation method and many times, the bare metal method is poorly documented.
So docker it is. And it’s a good skill to have no matter what since it is so widespread.
I never thought about the issues of setting a docker service with an external database. I don’t mind dealing with the users and tables of a database, but having to dig deep in docker compose settings is always a bad time.