You must log in or register to comment.
Backups. You’re forgetting them.
Don’t worry, you’re one Docker pull away from having to look up how to manually migrate Postgres databases within running containers!
(Looks at my PaperlessNGX container still down. Still irritated.)
I should do some breaking network changes… While tunneled in.
One word: chaos engineering!
The rare moment when everything actually works. 😄
Quick! Break something!
Maybe try this…
How is the kubernetes (k3s/rke2) migration coming along?
Actually, one thing I want to do is switch from services being on a subdomain to services being on a path.
immich.myserver.com -> myserver.com/immich jellyfin.myserver.com -> myserver.com/jellyfinI’m getting tired of having to update DNS records every time I want to add a new service.
I guess the tricky part will be making sure the services support this kind of routing…
Why are you having to update your DNS records when you add a new service? Just set up a wildcard A record to send *.myserver.com to the reverse proxy and you never have to touch it again. If your DNS doesn’t let you set wildcard A records, then switch to a better DNS.
Not OP but a lot of people probably use pi-hole which doesn’t support wildcards for some inane reason
That’s my case. I send every new subdomain to my nginx IP on pi-hole and then use nginx as a reverse proxy
That wasy exact setup as well until I switched to a different router which supported both custom DNS entries and blocklists, thereby making the pi-hole redundant
I run opnsense, so I need to dump pi-hole. But I don’t have the energy right now to do that.
Pi-Hole was pretty straightforward at the time and I did not look back since then. Annoying, but easy.
I switched to Technitium and I’ve been pretty happy. Seems very robust, and as a bonus was easy to use it to stop DNS leaks (each upstream has a static route through a different Mullvad VPN, and since they’re queried in parallel, a VPN connection can go down without losing any DNS…maybe this is how pihole would have handled it too though).
And of course, wildcards supported no problem.
Wildcard CNAME pointing to your reverse proxy who then figures out where to route the request to? That’s what I’ve been doing - this way there’s no need to ever update DNS at all :)
I find the path a bit clunky because the apps themselves will oftentimes get confused (especially front-ends). So keeping everything “bare” wrt path, and just on “separate” subdomains is usually my preferred approach.
In Nginx you can do rewrites so services think they are at the root.
Alternatively if you’re tired of manual DNS configuration:
FreeIPA, like AD but fer ur *Nix boxes
Configures users, sudoer group, ssh keys, and DNS in one go.
Also lotta services can be integrated using LDAP auth too.
So far I’ve got proxmox, jellyfin, zoneminder, mediawiki, and forgejo authing against freeipa in top of my samba shares.
Ansible works too just because its uses ssh, but I’ve yet to figure out how to build ansible inventories dynamically off of freeIPA host groups. Seen a coupla old scripts but that’s about it.
Current freeipa plugin for it seems more about automagic deployment of new domains.
Having a very similar infrastructure, I would love to know if you ever find anything that works for this. I’ve been maintaining a SnipeIT instance manually, but that’s a real PITA. Tried the same with ITSM-NG, but haven’t even lookid in it for months.
I had the same idea, but the solution I thought about is finding a way to define my DNS records as code, so I can automate the deployment. But the pain is tolerable so far (I have maybe 30 subdomains?), I haven’t done anything yet
Time to start documenting it!
At 71, I have to document. I started a long time ago. I worked for a mec. contractor long ago, and the rule was: ‘If you didn’t write it down, it didn’t happen.’ That just carried over to everything I do.
Do you write down what you write down on the internet?
As in a blog or wiki? I do not because I am not authoritative. What I know came from reading, doing, screwing it up, ad nauseam. When something finally clicks for me, I write it down because 9 times out of 10, I will need that info later. But my writing would be so full of inaccuracies that it would be embarrassing and possibly lead someone astray.
NEVER1!!!11!!
Don’t look too closely you can jinx it.
Living the good life
If it’s stable, it’s not a lab.
That’s infrastructure.
I’ve moved my homelab twice because it became stable, I really liked the services it was running, and I didn’t want to disturb the last lab**cough**prod server.
My current homelab will be moar containers. I’m sure I’ll push it to prod instead of changing the IP address and swapping name tags this time.
Can’t believe nobody here mentioned nixOS so far? How about moving all of your configs in a flake and manage all of your systems with it?
I made a git repo and started putting all of my dot files in a Stow and then I forgot why I was doing it in the first place.
So that when setting up a new system, you can migrate all your user configuration easily, while also version-controlling it.
git commit --message 'So that when setting up a new system, you can migrate all your user configuration easily, while also version-controlling it.'
I already have Ansible to manage my system and I like to have the same base between my pc and my server build muscle memory.
If I was managing a pc fleet I would consider NixOS, but I don’t see the appeal right now.
Okay, but why not create more work for yourself by rebuilding everything from scratch?
Have you tried introducing unnecessary complexity?
If you know how your setup works, then that’s a great time for another project that breaks everything.
Saturday morning: “Incus and podman seem interesting. I bet I could swap everything over while the family is out this afternoon”
Sunday evening: “Dad, when will the lights work again?”
“Dad, when will the lights work again?
As soon as selinux decides I have permission.
The old lighting wasn’t that great anyway. If I were to just put lighting on a DMX512-controlled network, then all of it could be synchronized to whole-house audio…
Don’t forget to integrate it into Home Assistant so you can alert the ISS when the mail man is on the porch.
unnecessary complexity?
I can help with that. It’s a skill I have. LOL
Infrastructure diagram? No! In this homelab we refer to the infrastructure hyperdodecahedron.
It seems like a good time to learn graphviz’s dot format for the network layout diagrams, with automated layout.
https://mamchenkov.net/wordpress/2015/08/20/graphviz-dot-erds-network-diagrams/
TIL. Thank you!
Sure. What that guy is using is actually not the most-interesting diagram style, IMHO, for automatic layout of network maps, if you want large-scale stuff, which is where the automatic layout gets more interesting. I have some scripts floating around somewhere that will generate very large network maps — run a bunch of traceroutes, geolocate IPs, dump the results into an sqlite database, and then generate an automatically laid-out Internet network map. I don’t want to go to the trouble of anonymizing the addresses and locations right now, but if you have a graphviz graph and want to try playing with it, I used:
goes looking
Ugh, it’s Python 2, a decade-and-a-half old, and never got ported to Python 3. Lemme gin up an example for the non-hierarchical graphviz stuff:
graph.dot:
graph foo { a--b a--d b--c d--e c--e e--f b--d }Processed with:
$ sfdp -Goverlap=prism -Gsep=+5 -Gesep=+4 -Gremincross -Gpack -Gsplines=true -Tpdf -o graph.pdf graph.dotGenerates something like this:
That’ll take a ton of graphviz edges and nicely lay them out while trying to avoid crossing edges and stuff, in a non-hierarchical map. Get more complicated maps that it can’t use direct lines on, it’ll use splines to curve lines around nodes. You can create massive network maps like this. Note that I was last looking at graphviz’s automated layout stuff about 15 years ago, so it’s possible that they have better layout algorithms now, but this can deal with enormous numbers of nodes and will do reasonable things with them.
I just grabbed his example because it was the first graphviz network map example that came up on a Web search.
Haha too right mate
This is just as true in my non-computer hobbies that involve physical systems instead of code and configs!
If I had to just barely meet the requirements using as little budget as possible while making it easy for other people to work on, that would be called “work.” My brain needs to indulge in some over-engineering and “I need to see it for myself” kind of design decisions.
Now try migrating all your docker containers to podman.
And then migrate all your podman containers to proxmox
Don’t encourage me.
And then try turning on SELinux!
I set my homelab up on Bazzite immutable with podman and SELinux. It took a while to work everything out and have it boot up into a valid state hahaha
Any reason you chose Bazzite for your homelab distro? First I’ve heard of someone doing that!
At the start I just wanted a desktop machine that runs Steam through sunshine/moonlight so hardware support and gaming stuff such was very important.
My homelab used to run on my laptop when it could all fit within a couple 100s of GB and I was the only user but moving it was tricky. Since I’m a programmer I’m not afraid of this stuff so I just spent the hours to figure out one problem at a time.
I ended up figuring out adding HDD whitelist in SELinux, make it accessible in podman, manually edit fstab because tools didn’t work, systemd service for startup, logging in automatically where I already forgot everything and would have not had to do any of this on a bog standard Ubuntu server.
Wouldn’t an immutable OS be overall a pretty good idea for a stable server?
Good for stability, bad for flexibility for when the homelab grows more complex.
I honestly don’t know a ton about immutable distros other than that they let you front-load some difficulty in getting things set up in exchange for making it harder to break. I was just surprised that the distro of choice was Bazzite, since its target audience seems to be gamers.
It’s not that difficult to get SELinux working with podman quadlets, especially if you run things rootless. I have a kerberized service account for each application I host and my quadlets are configured to run under those. I very rarely encounter applications that simoky can’t be run rootless but I usually can find an adequate alternative. I think right now the only thing that runs as root is one of the talk or collabora containers in my nextcloud stack. No selinux issues either.
I use podman-compose with system accounts and I don’t have a ton of issues. The biggest one is that I can’t seem to get bluetooth and pip working on Home Assistant at the same time. Most of the servers I manage have SELinux and it works fine as long as I use
:z/:Zwith bind mounts.A few years ago, I set up a VPS for my friend’s business; at the time, I didn’t know how to work with SELinux so I just turned it off. I tried to flip it back on, and it somehow bricked the system. We had to restore from a backup. Since then, I’ve been afraid to enable it on my flagship homelab server.
are you sure it really bricked it? when turning it on, on next boot it needs to go over all the files and retag them or something like that, and it can take a significant amount of time
Honestly, I don’t know what happened, but it was unreachable via SSH and the web console. There shouldn’t have been a ton of files to tag since it was an Almalinux system that started with SELinux enabled, and all we added was a container app or two.
Just did that last weekend. Nothing to do anymore. 😢
Did you do Quadlets?
I had problems getting apps with multiple containers working in quadlets (definitely a knowledge issue on my part, but didn’t feel the time learning it was beneficial, but will probably revisit during kubernetes learning) so went back to podman with docker compose.
I think it’s kinda better using quadlets, because I wrote some custom scripts, and quadlets made the process better. But podman compose is probably file too.
Yes of course. Had to spend a couple of hours fixing permission related issues.
But did you run them as rootful or the intended rootless way.
Rootless. The docker containers were rootful, hence the permission struggles.
Have you already tried implementing an identity provider like Authentik, so you can add OIDC and ldap for all your services, while you are the only one that’s using them? 🤔
Behind a traefik reverse proxy with lets encrypt for ssl even though the services aren’t exposed to the internet?
Don’t forget about Anubis and crowdsec to make it even safer inside your LAN
To be fair a lot of apps don’t handle custom CAs like they should. Looking at you Home Assistant! 😠
Who cares if it’s exposed to the internet?
-
Encrypting your local traffic is still valuable to protect your systems from any bad actors on your local network (neighbor kid cracks your wifi password, some device on your network decides to start snooping on your local traffic, etc)
-
Many services require HTTPS with a valid cert to function correctly, eg: Bitwarden. Having a real cert for a real domain is much simpler and easier to maintain than setting up your own CA
-
Probably a good idea to switch over to WPA-Enterprise using Authentik’s RADIUS server support and let all of the users of your wireless access point log in with their own network credentials, while you’re at it.
Hey my wife uses some of them too!
Let’s tinker around and accidentally break something.
My
manperson!and debug it until you have to reinstall your entire stack from scarch
GET OUT OF MY HOUSE!
Are you implying it’s possible to debug without having to reinstall from scratch? Preposterous! 😂
Scarched arth
Guess this is a good time to test my infrastructure automation.
“Damn, I’ve got this Debian server shit down. I wonder how an opensuse server would work out”
*installs tumbleweed*
True story
