Chapter 4. Managing Services with systemd
Every time you start your Linux computer, its initialization system launches a batch of processes, from a few dozen to hundreds, depending on how the system is set up. You can see this on your startup screen (Figure 4-1; press the Escape key to hide your graphical startup screen and see the startup messages).
In olden times we had the Unix System V initialization system (SysV init), BSD init, and Linux Standard Base (LSB) init for launching processes at startup. SysV init was the most common. Those days are fading away, and now systemd is the shiny new init system for Linux. It has been adopted by all the major Linux distributions, though of course there are a number of distributions that still use the legacy init systems.
In this chapter you will learn if your Linux distribution uses systemd. You will learn what processes, threads, services, and daemons are, and how to use systemd to manage services: start, stop, enable, disable, and check status. You will become acquainted with the systemctl command, which is the systemd system and service manager.
systemd is designed to provide functionality suited to modern complex server and desktop systems, and it does considerably more than the legacy init systems. It provides complete service management from startup to shutdown, starting processes at boot, on-demand after boot, and shutting down services when they are not needed. It manages functions such as system logging, automounting filesystems, automatic service dependency resolution, name services, device management, network connection management, login management, and a host of other tasks.
This sounds like a lot until you realize that processes do everything on a computer, and all of this functionality used to be provided by a large assortment of other programs. systemd brings it all together in an integrated software suite that should operate the same way on all Linux systems, though as always with Linux there are some minor exceptions, such as file locations and service names. Be aware that your particular Linux may have some differences from the examples in this chapter.
systemd attempts to decrease boot times and parcel out system resources more efficiently by starting processes concurrently and in parallel, and starting only necessary services, leaving other services to start after boot as needed. A service that is dependent on other services no longer has to wait to start for those services to become available because all it needs is a waiting Unix socket to become available. Recipe 4.9 shows how to find processes that are slowing down your system startup.
systemd binaries are written in C, which provides some performance enhancement. The legacy inits are masses of shell scripts, and any compiled language operates faster than shell scripts.
systemd is backwards compatible with SysV init. Most Linux distributions retain the legacy SysV configuration files and scripts, including /etc/inittab and the /etc/rc.d/ and /etc/init.d/ directories. When a service does not have a systemd configuration file, systemd looks for a SysV configuration file. systemd is also backward compatible with Linux Standard Base (LSB) init.
systemd service files are smaller and easier to understand than SysV init files. Compare a SysV init file for sshd with its systemd service file. This is a snippet of the sshd init file, /etc/init.d/ssh, from MX Linux:
#! /bin/sh ### BEGIN INIT INFO # Provides: sshd # Required-Start: $remote_fs $syslog # Required-Stop: $remote_fs $syslog # Default-Start: 2 3 4 5 # Default-Stop: # Short-Description: OpenBSD Secure Shell server ### END INIT INFO set -e # /etc/init.d/ssh: start and stop the OpenBSD "secure shell(tm)" daemon test -x /usr/sbin/sshd || exit 0 umask 022 if test -f /etc/default/ssh; then [...]
This goes on for a total of 162 lines. This is a complete systemd service file from Ubuntu 20.04, /lib/systemd/system/ssh.service:
[Unit] Description=OpenBSD Secure Shell server Documentation=man:sshd(8) man:sshd_config(5) After=network.target auditd.service ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service] EnvironmentFile=-/etc/default/ssh ExecStartPre=/usr/sbin/sshd -t ExecStart=/usr/sbin/sshd -D $SSHD_OPTS ExecReload=/usr/sbin/sshd -t ExecReload=/bin/kill -HUP $MAINPID KillMode=process Restart=on-failure RestartPreventExitStatus=255 Type=notify RuntimeDirectory=sshd RuntimeDirectoryMode=0755 [Install] WantedBy=multi-user.target Alias=sshd.service
Even without reading the documentation, or knowing anything about systemd, you can understand some of what this file is supposed to do.
See Rethinking PID 1 for a detailed introduction to systemd by one of its inventors and maintainers, Lennart Poettering. Rethinking PID 1 details the rationale behind building a new init system, its architecture, advantages, and how it uses existing Linux kernel features in place of duplicating existing functionality.
4.1 Learning if Your Linux Uses systemd
Solution
Look for the /run/systemd/system/ directory. If this exists, then your init system is systemd.
Discussion
The /run/systemd/ directory may be present on your system if your distribution supports multiple init systems. But systemd is not the active init unless you see /run/systemd/system/.
There are several other ways to learn which init system your system is using. Try querying /sbin/init. Originally this was the SysV executable, and now most Linux distributions preserve the name and symlink it to the systemd executable. This example confirms that the init is systemd:
$ stat /sbin/init File: /sbin/init -> /lib/systemd/systemd [...]
On a system using SysV init, it has no symlink:
$ stat /sbin/init File: /sbin/init [...]
The /proc pseudofilesystem is an interface to your Linux kernel, and contains the current state of a running system. It is called a pseudofilesystem because it exists only in memory and not on disk. In this example, /proc/1/exe is symlinked to the systemd executable:
$ sudo stat /proc/1/exe File: /proc/1/exe -> /lib/systemd/systemd [...]
On a SysV system, it links to init:
$ sudo stat /proc/1/exe File: /proc/1/exe -> /sbin/init [...]
The /proc/1/comm file reports your active init system:
$ cat /proc/1/comm systemd
On a SysV system, it reports init:
$ cat /proc/1/comm init
The command attached to process ID (PID) 1 is your init. PID 1 is the first process launched at startup, which then starts all other processes. You can see this with the ps command:
$ ps -p 1 PID TTY TIME CMD 1 ? 00:00:00 systemd
When the init is SysV, it looks like this:
$ ps -p 1 PID TTY TIME CMD 1 ? 00:00:00 init
See Recipe 4.2 for more information on PID 1.
Linux support for systemd varies. Most of the major Linux distributions have adoped systemd, including Fedora, Red Hat, CentOS, openSUSE, SUSE Linux Enterprise, Debian, Ubuntu, Linux Mint, Arch, Manjaro, Elementary, and Mageia Linux.
Some popular distributions that do not support systemd, or include it but not as the default init, are Slackware, PCLinuxOS, Gentoo Linux, MX Linux, and antiX.
See Also
-
Distrowatch for information on hundreds of Linux distributions
-
man 5 proc
-
man 1 pstree
-
man 1 ps
4.2 Understanding PID 1, the Mother of All Processes
Solution
PID 1 is the mother of all processes on Linux systems. This is the first process to start, and then it launches all other processes.
Processes are one or more running instances of a program. Every task in a Linux system is performed by a process. Processes can create independent copies of themselves, that is, they can fork. The forked copies are called children, and the original is the parent. Each child has its own unique PID, and its own allocation of system resources, such as CPU and memory. Threads are lightweight processes that run in parallel and share system resources with their parents.
Some processes run in the background and do not interact with users. Linux calls these processes services or daemons, and their names tend to end with the letter D, such as httpd, sshd, and systemd.
Every Linux system starts PID 1 first, which then launches all other processes. Use the ps command to list all running processes in PID order:
$ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 10:06 ? 00:00:01 /sbin/init splash root 2 0 0 10:06 ? 00:00:00 [kthreadd] root 3 2 0 10:06 ? 00:00:00 [rcu_gp] root 4 2 0 10:06 ? 00:00:00 [rcu_par_gp] [...]
The pstree command organizes this mass of information into a tree diagram. This example shows all processes, their child processes, PIDs, and threads, which are enclosed in curly braces:
$ pstree -p systemd(1)─┬─ModemManager(925)─┬─{ModemManager}(944) │ └─{ModemManager}(949) ├─NetworkManager(950)─┬─dhclient(1981) │ ├─{NetworkManager}(989) │ └─{NetworkManager}(991) ├─accounts-daemon(927)─┬─{accounts-daemon}(938) │ └─{accounts-daemon}(948) ├─acpid(934) ├─agetty(1103) ├─avahi-daemon(953)───avahi-daemon(970) [...]
The full pstree output is quite large. You can view a single process, identified by its PID, and its parents, children, and threads, like the following example for the Kate text editor:
$ pstree -sp 5193 systemd(1)───kate(5193)─┬─bash(5218) ├─{kate}(5195) ├─{kate}(5196) ├─{kate}(5197) ├─{kate}(5198) ├─{kate}(5199) [...]
This shows that systemd(1)
is Kate’s parent, bash(5218)
is Kate’s child,
and all the processes in curly braces are Kate’s threads.
Discussion
Processes always exist in one of several states, and these states change according to system activity. The following pstree example displays the PID, user, state, and command fields:
$ ps -eo pid,user,stat,comm PID USER STAT COMMAND 1 root Ss systemd 2 root S kthreadd 32 root I< kworker/3:0H-kb 68 root SN khugepaged 11222 duchess Rl konsole
-
R is either currently running or waiting in the run queue.
-
l means the process is multithreaded.
-
S is interruptable sleep; the process is waiting for an event to complete.
-
s is a session leader. Sessions are related processes managed as a unit.
-
I is an idle kernel thread.
-
< means high priority.
-
N is low priority.
There are several rarely used states you can read about in man 1 ps.
See Also
- Recipe 4.7
- man 5 proc
- man 1 pstree
- man 1 ps
4.3 Listing Services and Their States with systemctl
Solution
systemctl, the systemd manager command, tells all. Run it with no options to see a detailed list of all loaded units. A systemd unit is any related batch of processes defined in a unit configuration file and managed by systemd:
$ systemctl
This prints a giant pile of information: 177 active loaded units on my test system with the full unit names, status, and long descriptions. Redirect the the output to a text file for easier study:
$ systemctl > /tmp/systemctl-units.txt
Treat yourself to more information overload by listing all units, active and inactive:
$ systemctl --all
This results in 349 loaded units listed on my test system, including not-found and inactive units. How many total unit files? The following example shows 5 out of 322:
$ systemctl list-unit-files UNIT FILE STATE proc-sys-fs-binfmt_misc.automount static -.mount generated mount generated dev-hugepages.mount static home.mount generated [...] 322 unit files listed.
We are interested in service files because Linux users and administrators interact mainly with service files and rarely need to bother with any other type of unit file. How many are installed? Let’s see:
$ systemctl list-unit-files --type=service UNIT FILE STATE accounts-daemon.service enabled acpid.service disabled alsa-state.service static alsa-utils.service masked anacron.service enabled [...] 212 unit files listed.
The preceding example displays the four most common states that a service can be in: enabled, disabled, static, or masked.
$ systemctl list-unit-files --type=service --state=enabled UNIT FILE STATE accounts-daemon.service enabled anacron.service enabled apparmor.service enabled autovt@.service enabled avahi-daemon.service enabled [...] 62 unit files listed.
$ systemctl list-unit-files --type=service --state=disabled UNIT FILE STATE acpid.service disabled brltty.service disabled console-getty.service disabled mariadb@.service disabled [...] 12 unit files listed.
$ systemctl list-unit-files --type=service --state=static UNIT FILE STATE alsa-restore.service static alsa-state.service static apt-daily-upgrade.service static apt-daily.service static [...] 106 unit files listed.
$ systemctl list-unit-files --type=service --state=masked UNIT FILE STATE alsa-utils.service masked bootlogd.service masked bootlogs.service masked checkfs.service masked [...] 36 unit files listed.
Discussion
Service unit files are in /usr/lib/systemd/system/ or /lib/systemd/system/, according to where your Linux distribution puts them. These are plain-text files you can read.
- enabled
-
This shows that the service has become available and is managed by systemd. When a service is enabled, systemd creates a symlink in /etc/systemd/system/ from the unit file in /lib/systemd/system/. It can be started, stopped, reloaded, and disabled by the user with the systemctl command.
Note
Enabling a service does not immediately start it, and disabling a service does not immediately stop it (see Recipe 4.6).
- disabled
-
Diabled means that there is no symlink in /etc/systemd/system/, and it will not start automatically at boot. You can stop and start it manually.
- masked
-
This means the service is linked to /dev/null/. It is completely disabled and cannot be started by any means.
- static
-
This means that the unit file is a dependency of other unit files, and cannot be started or stopped by the user.
Some less-common service states you will see:
See Also
-
man 1 systemctl
4.4 Querying the Status of Selected Services
Solution
systemctl status provides a nice little bundle of useful status information. The following example queries the CUPS service. CUPS, the Common Unix Printing System, should be on all Linux systems:
$ systemctl status cups.service ● cups.service - CUPS Scheduler Loaded: loaded (/lib/systemd/system/cups.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2021-11-22 11:01:48 PST; 4h 17min ago TriggeredBy: ● cups.path ● cups.socket Docs: man:cupsd(8) Main PID: 1403 (cupsd) Tasks: 2 (limit: 18760) Memory: 3.8M CGroup: /system.slice/cups.service ├─1403 /usr/sbin/cupsd -l └─1421 /usr/lib/cups/notifier/dbus dbus:// Nov 22 11:01:48 host1 systemd[1]: Started CUPS Scheduler.
Query multiple services with a space-delimited list:
$ systemctl status mariadb.service bluetooth.service lm-sensors.service
Discussion
There is a lot of useful information in this little bit of output (Figure 4-2).
The dot next to the service name is a quick status indicator. It appears in colors on most terminals. White is an inactive or deactivating state. Red is a failed or error state. Green indicates an active, reloading, or activating state. The rest of the information in the output is described in the following:
- Loaded
-
Verifies that the unit file has been loaded into memory, displays its full path, the service is enabled (see the Discussion about states in Recipe 4.3), and
vendor preset: disabled/enabled
indicates if the installation default is to start at boot or not. When it is disabled, the vendor default is to not start at boot. This only shows the vendor preference and does not indicate if it is currently enabled or disabled. - Active
-
Tells you if the service is active or inactive, and how long it has been in that state.
- Process
-
Reports the PIDs and their commands and daemons.
- Main PID
-
This is the process number for the cgroup slice.
- Tasks
-
Reports how many tasks the service has started. Tasks are PIDs.
- CGroup
-
Shows which unit slice the service belongs to and its PID. The three default unit slices are user.slice, system.slice, and machine.slice.
Linux control groups (cgroups) are sets of related processes and all of their future children. In systemd, a slice is a subdivision of a cgroup, and each slice manages a particular group of processes. Run systemctl status to see a diagram of the cgroup hierarchy.
By default, service and scope units are grouped in /lib/systemd/system/system.slice.
User sessions are grouped in /lib/systemd/system/user.slice.
Virtual machines and containers registered with systemd are grouped in /lib/systemd/system/machine.slice.
The remaining lines are the most recent log entries from journalctl, the systemd log manager.
See Also
-
man 1 systemctl
-
man 5 systemd.slice
-
man 1 journalctl
4.5 Starting and Stopping Services
Solution
This is a job for systemctl. The following examples use the SSH service to demonstrate service management.
Start a service:
$ sudo systemctl start sshd.service
Stop a service:
$ sudo systemctl stop sshd.service
Stop and then restart a service:
$ sudo systemctl restart sshd.service
Reload the service’s configuration. For example, you made a change to sshd_config and want to load the new configuration without restarting the service:
$ sudo systemctl reload sshd.service
Discussion
All of these commands also work with multiple services, space-delimited, for example:
$ sudo systemctl start sshd.service mariadb.service firewalld.service
If you’re curious about the commands that systemd runs behind the scenes to start, reload, or stop the individual daemons, look in their unit files. Some services have start, reload, stop, and other instructions in their unit files, like this example for httpd:
ExecStart=/usr/sbin/httpd/ $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/bin/kill -WINCH ${MAINPID}
You don’t have to do anything special with this information; it is there when you want to know how systemctl is managing a particular service.
See Also
-
man 1 systemctl
4.6 Enabling and Disabling Services
Solution
Enabling a service configures it to automatically start at boot.
Disabling a service stops it from starting at boot, but it can be started and stopped manually.
Masking a service disables it so that it cannot be started at all.
The following example enables the sshd service:
$ sudo systemctl enable sshd.service Created symlink /etc/systemd/system/multi-user.target.wants/sshd.service → /usr/lib/systemd/system/sshd.service
The output shows that enabling a service means creating a symlink from the service file in /lib/systemd/system/ to /etc/systemd/system/. This does not start the service. You can start the service with systemctl start, or enable and start the service in one command with the --now option:
$ sudo systemctl enable --now sshd.service
This command disables the sshd service. It does not stop the service, so you must stop it manually after disabling it:
$ sudo systemctl disable sshd.service Removed /etc/systemd/system/multi-user.target.wants/sshd.service $ sudo systemctl stop sshd.service
Or, disable and stop it with one command:
$ sudo systemctl disable --now sshd.service
This command reenables the mariadb service, which disables and then enables it. If you have created the symlinks manually for a service, this is useful for quickly resetting them to the defaults:
$ sudo systemctl reenable mariadb.service Removed /etc/systemd/system/multi-user.target.wants/mariadb.service. Removed /etc/systemd/system/mysqld.service. Removed /etc/systemd/system/mysql.service. Created symlink /etc/systemd/system/mysql.service → /lib/systemd/system/mariadb.service. Created symlink /etc/systemd/system/mysqld.service → /lib/systemd/system/mariadb.service. Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /lib/systemd/system/mariadb.service.
The following command disables the bluetooth service completely by masking it, so that it cannot be started at all:
$ sudo systemctl mask bluetooth.service Created symlink /etc/systemd/system/bluetooth.service → /dev/null.
Unmasking the bluetooth service does not enable it, so it must be started manually:
$ sudo systemctl unmask bluetooth.service Removed /etc/systemd/system/bluetooth.service. $ sudo systemctl start bluetooth.service
Discussion
When you enable, disable, mask, or unmask a service, it remains in its current state unless you use the --now option. The --now option works with enable, disable, and mask to immediately stop or start the service, but it does not work with unmask.
See the Discussion in Recipe 4.3 to learn more about how systemd uses symlinks to manage services.
See Also
-
man 1 systemctl
-
The Discussion in Recipe 4.3 to learn how systemd uses symlinks to manage services
4.7 Stopping Troublesome Processes
Solution
Stopping a process is called killing the process. On Linux systems with systemd, you should use systemctl kill. On systems without systemd, use the legacy kill command.
systemctl kill is preferable because it stops all processes that belong to a service and leaves no orphan processes, nor any processes that might restart the service and continue to make trouble. First, try it with no options other than the service name, then check the status:
$ sudo systemctl kill mariadb $ systemctl status mariadb ● mariadb.service - MariaDB 10.1.44 database server Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled) Active: inactive (dead) since Sun 2020-06-28 19:57:49 PDT; 6s ago [...]
The service has cleanly stopped. If this does not work, then try the nuclear option:
$ sudo systemctl kill -9 mariadb
The legacy kill command does not recognize service or command names, but rather requires the PID of the offending process:
$ sudo kill 1234
If this does not stop it, use the nuclear option:
$ sudo kill -9 1234
Discussion
Use the top command to identify runaway processes. Run it with no options, and the processes using up the most CPU resources are listed at the top. Press the q key to exit top.
$ top top - 20:30:13 up 4:24, 6 users, load average: 0.00, 0.03, 0.06 Tasks: 246 total, 1 running, 170 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.4 us, 0.2 sy, 0.0 ni, 99.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16071016 total, 7295284 free, 1911276 used, 6864456 buff/cache KiB Swap: 8928604 total, 8928604 free, 0 used. 13505600 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3504 madmax 20 0 99.844g 177588 88712 S 2.6 1.1 0:08.68 evolution 2081 madmax 20 0 3818636 517756 177744 S 0.7 3.2 5:07.56 firefox 1064 root 20 0 567244 148432 125572 S 0.3 0.9 12:54.75 Xorg 2362 stash 20 0 2997732 230508 145444 S 0.3 1.4 0:40.72 Web Content [...]
kill sends signals to processes, and the default signal is SIGTERM (signal terminate). SIGTERM is gentle, allowing processes to shut down cleanly. SIGTERM is also ignorable, and processes don’t have to pay attention to it. Signals can be identified by name or number; for most folks the numbers are easier to remember, so spelling out the default looks like this:
$ sudo kill -1 1234
kill -9 is SIGKILL. SIGKILL stops processes immediately and uncleanly, and also attempts to stop all child processes.
Killing services with systemctl kill is easier than with kill, and more reliable. You only need the service name, and you don’t have to hunt down PIDs. It ensures that all processes belonging to the service are stopped, which kill cannot ensure.
There are a ton of signals that have accumulated over the years, and you can read all about them in man 7 signal. In my experience, the most relevant signals are SIGTERM and SIGKILL, but don’t let that stop you from learning more about the others.
If you are uncomfortable with terminology like kill, parents, children, and orphans, so am I. Maybe someday it will change.
See Also
-
man 5 systemd.kill
-
man 1 systemctl
-
man 1 kill
-
man 7 signal
4.8 Managing Runlevels with systemd
Solution
systemd targets are similar to SysV runlevels. These are boot profiles that start your system with different options, such as multiuser mode with a graphical desktop, multiuser mode with no graphical desktop, and emergency and rescue modes to use when your current target will not boot. (See the Discussion for more information on runlevels.)
The following command checks if the system is running and reports its state:
$ systemctl is-system-running running
What is the default target?
$ systemctl get-default graphical.target
Get the current runlevel:
$ runlevel N 5
Reboot to rescue mode:
$ sudo systemctl rescue
Reboot to emergency mode:
$ sudo systemctl emergency
Reboot to the default mode:
$ sudo systemctl reboot
Reboot to a different target without changing the default:
$ sudo systemctl isolate multi-user.target
Set a different default runlevel:
$ sudo systemctl set-default multi-user.target
List the runlevel target files and their symlinks on your system:
$ ls -l /lib/systemd/system/runlevel*
List the dependencies in a runlevel target:
$ systemctl list-dependencies graphical.target
Discussion
SysV runlevels are different states that your system can boot to, for example, with a graphical desktop, without a graphical desktop, and with emergency runlevels to use when your default runlevel has problems and will not boot.
systemd targets approximately correspond to the legacy SysV runlevels:
-
runlevel0.target, poweroff.target, halt
-
runlevel1.target, rescue.target, single-user text mode, all local filesystems mounted, root user only, no networking
-
runlevel3.target, multi-user.target, multiuser text mode (no graphical environment)
-
runlevel5.target, graphical.target, multiuser graphical mode
-
runlevel6.target, reboot.target, reboot
systemctl emergency is a special target that is more restricted than rescue mode: no services, no mount points other than the root filesystem, no networking, root user only. It is the most minimal running system for debugging problems. You may see options to boot into a rescue or emergency mode in your GRUB2 bootloader screen.
systemctl is-system-running reports various system states:
-
initializing means the system has not completed startup.
-
starting means the system is in the final stages of startup.
-
running is fully operational, and all processes are started.
-
degraded means the system is operational, but one or more systemd units have failed. Run systemctl | grep failed to see which units failed.
-
maintenance means that either the rescue or emergency target is active.
-
stopping means that systemd is shutting down.
-
offline means that systemd is not running.
-
unknown means that there is a problem preventing systemd from determining the operational state.
See Also
-
man 1 systemctl
-
man 8 systemd-halt.service
4.9 Diagnosing Slow Startups
Solution
You want systemd-analyze blame. Run it with no options to see a list of system processes and how long they took to start:
$ systemd-analyze blame 34.590s apt-daily.service 6.782s NetworkManager-wait-online.service 6.181s dev-sda2.device 4.444s systemd-journal-flush.service 3.609s udisks2.service 2.450s snapd.service [...]
Analyze only user processes:
$ systemd-analyze blame --user 3.991s pulseaudio.service 553ms at-spi-dbus-bus.service 380ms evolution-calendar-factory.service 331ms evolution-addressbook-factory.service 280ms xfce4-notifyd.service [...]
Discussion
It is useful to review everything that starts at boot and perhaps find services you don’t want starting at boot. My favorite to disable is Bluetooth because I don’t use it on my servers or PCs, but many Linux distros enable it by default.
See Also
-
man 1 systemd-analyze
Get Linux Cookbook, 2nd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.