I doubt I’ll be alone when I say that my server setup is a mess. It is a monolith of different services slammed together on the same machine. Some run on the bare metal, some run in a VM, some run in containers.
Recently I put some work into migrating my applications away from bare metal and into containers. While that
effort is not complete just yet, it has already broken on me after
podman-compose stopped creating pods per
compose file. Apparently that was a bug, and I relied on this behaviour with a wrapper script.
The sequence this script followed is rather simple:
- Remove any existing podman-generated systemd unit files from
- Generate new system files with
podman generate systemd
- Move all
*.servicefiles generated to the
However, this relied on the previous behaviour in podman-compose which created a pod for all services to live in per
compose file, so it can guess the pod name and generate a set of systemd units for it with one invocation of the
podman generate systemd command. Nifty, worked for a while, but now it’s time to move on.
The puzzle pieces
Let’s lay out what we want to achieve.
- For security, containers should be rootless
- For maintainability, containers should be defined in a
- For usability, containers should auto-start on boot
- Purely for the sake of laziness we’ll throw in the extra requirement that containers should be self-updating
- This might haunt me later. We’ll see about that, but because of this it should be trivial to disable
Target 1: Rootless
I refuse to run my containers in a rootful environment. I strongly believe that podman is designed to remedy this design decision in Docker and I should take advantage of that.
My containers run as a separate
podman user with virtually no privileges on the system. Most containers run the
software they host under a different user as well, which means that it’s practically impossible for container X to
touch container Y’s files.
This imposes challenges when you want to share folders between containers, but that’s something a little GID magic can solve; give the containers the same GID and set a restrictive permission mask on that, so the actual owner gets write permissions yet other containers get read permission only.
Target 2: docker-compose
The standard docker-compose now works with podman v3!
A standard docker-compose.yml file should be sufficient. As an example, below is the configuration file for my Nextcloud instance:
The only thing really of note is that every service has
restart: always set. This will be explained later.
Please note that rootless containers might not have the same privileges as a rootful container in terms of networking etcetera. Always double check the compose files you copy!
It is preferred that you use the podman secrets infrastructure for passwords instead of hardcoding them into this text file. This is beyond the scope of this article, do some research!
Target 2.1: SELinux
Fedora has SELinux enabled by default, and we don’t want to make Dan Walsh weep. Thankfully, setting up SELinux to play nice with your containers is trivial.
What I like to do is create a
/containers directory (yes, in the root folder) whose owner I set to the podman user. After this it is just a matter of making sure the directory has the right SELinux type label, which is
container_file_t. You can achieve this with the following command:
This will ensure that the
/containers directory and all of its sub-directories and files will get the appropriate SELinux type applied. This however does not immediately apply to existing files or directories, so we can apply it by running:
If all is well you should be seeing a lot of label changes being printed to the console, depending on how many files exist in this directory.
Target 3: Autostart on boot
This took me too long to figure out, but it turns out that Podman already ships with a solution built in. Remember the
restart: always that I mentioned earlier?
It turns out Podman ships with a systemd unit file specifically to start all containers marked to always restart. It is called
podman-restart.service and it can be enabled as a user service!
Start a container marked with
restart: always, and then reboot your system. Nothing happens, congratulations!
By default, user units are only started whenever the user starts a session. Since nobody has logged in to your podman user, no session was ever started. Even worse, should someone login as the podman user, the session will start, which will start your container - and then it will get killed when that person logs out. That’s super inconvenient.
systemd-logind has a concept of lingering users. The
loginctl man page describes it as:
… If enabled for a specific user, a user manager is spawned for the user at boot and kept around after logouts. This allows users who are not logged in to run long-running services. …
We can enable this for our podman user with the following command:
Set that, reboot again and you should see it all start to come together.
Target 4: Automatic updates
The tricky part. We can have dnf automatic updates through
dnf-automatic, and Podman even has its own auto-update system! However, we cannot use this in this setup, because we do not rely on per-container systemd unit files, because we use podman-compose and the global
Instead, what we can do is replicate the setup using custom unit files per docker-compose file. Create the following file at
This is a template file which can be used multiple times with a different input parameter (the string after the @ sign).
Make sure to check the WorkingDirectory setting! This assumes you have your docker-compose.yml files stored in ~/infra/[subfolder]/docker-compose.yml
And add the following timer at
For every subfolder, you can now enable the timer, which will call the unit:
nextcloud is a subfolder of
~/infra. Your nextcloud container will now update daily at midnight!
Bonus: Mail on unit start/failure
To adjust this approach for user unit files, create
Then uncomment the commented line in the systemd service above. Don’t forget to
chmod +x the file and add SELinux rules: