Plenty of us are using Docker, Podman, Incus, chroot jails, etc to isolate services.
It has become good practice and it makes setting up yet another service, usually, so convenient.
Some services like YunoHost, StartOS, Cloudron and others try to facilitate the process.
What I haven’t seen though is a way to facilitate interoperability BETWEEN services we self-host. Sure there are plugins for each service, e.g. https://www.npmjs.com/package/peertube-plugin-livechat to provide XMPP chat for PeerTube, or anecdotal discussions e.g. https://github.com/jitsi/jitsi-meet/issues/7601 to embed PeerTube on Jitsi Meet.
So… how do YOU do it? How do you make on self-hosted service with another? Do you check after each one you install in the plugin category? Do you write your own plugins or extensions? Do you have a design pattern (e.g. Swagger API discovery with token generation per service, “cheat” via sockets, use a dedicate new service or even host) which you repeat?
I do ask because I bet most of you have a moment like this :
- Hey how about we start this new project together?
- Yes, let’s change the World!
- OK let’s write manifesto.md
- Where are we going to host it?
- Hmmm we could use my Cryptpad instance…
- OK but I don’t get notification on my GMail, could we use GoogleDocs instead?
So… I feel like FLOSS self-hosting is honestly on-par functionality-wise with proprietary solutions. I might be bias but it’s rare when I think “Damn… that’s cool, shame I can’t have it at home”. I can nearly always (in fact I have a hard time thinking of an example) self-host functional equivalent solutions myself. The ONE thing that I feel is often missing is integration which relies on interoperability.
How do YOU it?
PS: this isn’t about ntfy, PeerTube, HA or any specific service to a specific problem, it’s about HOW to facilitate, when one wants to, already great services work together.
You must log in or register to comment.
I use authentik, which emables single sign on (the same account) between services.
Authentik is a bit complex and irritating at times, so I would recommend voidauth or kanidm as alternatives for most self hosters.
Thanks, that’s indeed exactly the kind of thing I’m looking for “The authentication glue you need.” but even more generalized than that, e.g. just “the glue you need.” not solely for authentication.
Edit: to clarify and coming back after leaving few other comments, the 1 thing authentik has is that it is a cross-service need, namely nearly all services do need authentication AND, probably consequence of that, there are conventions and standards already in place, e.g. SAML, OAuth2/OIDC, LDAP, Auth0. So that makes everything much easier.
I haven’t tried it yet, but I think you want something like n8n. Yeah, I know they claim to be very AI nowadays, but I had them on my radar before that. It’s essentially an automation platform you can use to glue services together that weren’t intended for each other. https://github.com/n8n-io/n8n
Thanks I’ll dig deeper. I guess I do want something like n8n but ideally :
- no AI
- self hosted FLOSS services first, if possibly only (can just be filters like F-Droid has)
- no need for no-code / low-code, code is great
which makes me wonder what they do provide, e.g. is it mostly indexing existing plugins and then some scaffolding for non coders?
APIs. Or the ends are achieved by sharing data between apps in common data storage. But I prefer to be a tourist in my infrastructure, I no longer hand-bomb changes to systems.
My design pattern is essentially to integrate more and more of the container creation into config. Right now I’m using ansible and it’s nice. More automation means troubleshooting has fewer variables.
I had issues yesterday with a package upgrade across several containers, and it ended up being two config changes. I cycle the apps and done. That’s it.
Indeed and for PeerTube for example it has an API, cf https://docs.joinpeertube.org/api-rest-reference.html which I did use. It also provides SWAGGER so that could facilitate integration with others services also providing APIs. I was starting to think that the meta service could have read only public only token generated for each new service and provide a SWAGGER endpoint to facilitate using the API of more than 1 service.
Hmmm we could use my Cryptpad instance… OK but I don’t get notification on my GMail
😡 ntfy.sh 😡
eh… yes thanks and I do actually have ntfy (I also warmly recommend it) but this was just an example.
It sounds like you’re describing Home Assistant? HA has a ton of integrations into a lot of self-hosted services not just IoT devices.
Well I do have Home Assistant, been running it for years, but HA is solely for … well home assisting (or IoT). HA as integrations but let’s say I want to use HA with … any of my other services, e.g. CopyParty to maybe store logs and makes them available or PeerTube to have videos from my camera, I can look at HA integrations, or CopyParty… issues maybe, or PeerTube npm registry.
My point being that HA is a good example with integrations but it’s just one example. If I do take this example seriously though, is there a mechanism beside manual search in the list of integration that would list integrations with my services directly?
@utopiah @yaroto98 re #HomeAssistant integrations, you can also look at what is available under #HACS library, search or ask for help in any of the community forums for #HA such as https://community.home-assistant.io/ or on reddit, discord , git hub. It integrates nicely with #homebridge and also with containerized systems e.g., docker, portainer.
Can’t say I’ve run into a need for such consideration yet. Excluding stacks explicitly meant to work together to some degree most of my services are an island to themselves and I like it that way. Then as far as notifications are concerned pretty much every supports at least email or ntfy.sh.
most of my services are an island to themselves
same
and I like it that way.
… well that’s the part I’m challenging. I was thinking like this but I’m wondering if that could be improved.
PS: I use ntfy and like it, that was just an example.
… well that’s the part I’m challenging. I was thinking like this but I’m wondering if that could be improved.
Do you have a specific use case for two containers that you want to talk to each other? There are some Docker containers that will cozy up, but as you point out, there are some that don’t. Maybe it would be worth the time to consult with the dev teams of said containers, and see if they have any suggestions or ways to go about it. It would seem to me that would be your best point to deviate from. Couldn’t hurt. I would create a defined road map of what you are trying to accomplish before hand, and run it by the dev teams. Just give them a little time to respond. They are real people with real lives too.
As I’m thinking about it, I wonder if your solution might be automation? Something like n8n might allow cross container exchanges. Of course, those data points have to be present for you to tap them tho.
Do you have a specific use case for two containers that you want to talk to each other?
Sure, for example once a Jitsi Meet meeting ends (more than 1 person in a room in, everybody gone), save the chat log to CopyParty e.g. WebDAV push to /meetingname_date.txt would be enough to be useful. It’s something we tend to do manually on a regular basis.
road map of what you are trying to accomplish before hand, and run it by the dev teams.
Yes no rush and I can code so I would be able to test before suggesting anything.
As I’m thinking about it, I wonder if your solution might be automation?
I don’t touch AI but I do think conventions, e.g. not “just” an API but SWAGGER, specific filesystem on mountpoints, etc could facilitate this.
I don’t touch AI
Automation doesn’t always mean AI. The app I mentioned, n8n, has two versions: with and without. The plain n8n app is very capable of doing a ton of stuff.
The plain n8n app is very capable of doing a ton of stuff.
Sorry if I’m a bit slow but what does it actually do? I skimmed through “automations” earlier this morning and I mostly found paid-for GenAI related stuff.
As an example, what I do with n8n to hook services together:
- We use Zello in firefighting to document the incident with short verbal notes or take pictures. Voice gets transcribed, sent to MQTT for dashboards, taken by Telegraf to influx. Pictures and mp3s get downloaded from zello, put into minio, linked back with the metadata and also sent to influx. Finally all is displayed in chronological order on a Dashboard. Zello, Minio, MQTT, influx all glued together with n8n.
- Another one is the appsmith app for equipment maintenance. We need reports and appsmith is bad at that. n8n queries the Source pgsql, formats it to JSON, sends it to an online service to generate a PDF (with a template), downloads the PDF and puts it into Nextcloud.
There is more, but just as two examples. Quite easy in n8n, because many integrations are preconfigured (e.g. Nextcloud), but also plain REST API if necessary (zello).
All without AI…
Edit: Forgot one, Rocket Chat to keep everybody updated on a bunch of stuff, e.g. new maintenance report available, message on the day of a training, based on caldav. Also all with n8n.
You’re not slow…sheesh.
The version I’m using does not include AI, basically because I don’t have the equipment to run AI 100% local, and n8n AI makes you tap public AI outlets. Both versions are available. The community version of n8n is open-source.
A workflow:
spoiler
n8n is a workflow automation platform that gives technical teams the flexibility of code with the speed of no-code. With 400+ integrations,
native AI capabilities, and a fair-code license, n8n lets you build powerful automations while maintaining full control over your data and deployments.There are over 400 different integrations: https://n8n.io/integrations Of course, the AI stuff you can skip. And when you get to the point of proficiency, you can build your own templates and workflows.
For instance, here is a workflow designed to automatically convert media: https://noted.lol/self-running-video-encoder-bash-n8n/
Another example is pulling logs: https://noted.lol/system-log-dashboard-n8n/
There are literally endless things you can integrate and build with n8n.
@utopiah #YunoHost, #StartOS, #Cloudron, #coopcloud, #uncloud and the people that write the recipies for the applications running on them are the answer.
Modelled solutions that check for related applications as they install are the way that encourage more people to self host by making installation, configuration, backup, update and removal easier.
Thanks, are you saying there is a mechanism in place, e.g. does YunoHost suggests plugins or integrations for services it manages?
My process typically goes: ooh, shiny new service!
If it has a docker compose, I’ll read up a little more on it, then set it up.
At that point, I’ll cry a bit in having to change my nginx config, because it’s new and unsupported, figure out what’s actually needed, then realize I didn’t set it up with authentik or even check if it was supported.
Usually I’ll dig around and someone got something close, so I’ll set it up that way, and if it works, great! If not, dig through the documents.Any further interoperability is luck based.
(I’m currently trying to get element-call working before I open it to my family, the whole matrix stack is leaving me scarred )
Yes I can relate to the process.
Any further interoperability is luck based.
Unfortunately I can relate to that, hence the question here :D
