Structural and semantic deficiencies in the systemd architecture, another jeremiad | V.R.

Some dude using the self-asserted identity token V.R.; Structural and semantic deficiencies in the systemd architecture for real-world service management, a technical treatise; In Some Blog; 2015-10-11.

tl;dr → A jeremiad; 8100 words; systemd is bad, he doesn’t like it.

Mentions

His point, and he does have one, is that
  • There are simply too many notes
  • And there is insufficient reference to the priors in the art
  • Oh! and it’s full of bugs! Lots! Of! Bugs!
Not shown:
  • the way forward
  • a complete viable alternative
    • complete in a technical sense of solving the problem
    • complete in a cultural sense of having an adiabatic transition to the new phase
  • that sticking with tangled masses of stylized /bin/sh (ahem, the SysV initscripts), better, possible or even an option.  That system worked “well enough ” that you knew getting away from it would be messy.
And yet
  • Mel Conway’s Law is iron
  • Lennart Poettering & Kay Sievers operate as a single organization.
  • Therefore systemd evolves onward as a single-process central-element architectural solution to the problems it addresses; all-in or out.
    Similar to the monolithic_macrokernel-vs-microkernel culture wars of the ’90s.  The Linux kernel is … go on, say it.

Outline

  • Preface and disclaimer (!)
  • Everything is a Unit (but it doesn’t mean a lot)
  • Job queuing
  • The transaction manager
  • To live is to depend
  • Every problem can be solved by a layer of indirection
  • Bus APIs, connections and object interface duplication
  • cgroup writing
  • Parsing in critical paths
  • Non-generic fd-holding and socket preopening
  • Inexpressive unit file options
  • Imbalance between promoting laziness or eagerness
  • Targets over milestones for synchronization
  • The (system-specific) problem of readiness notification
  • Intertwining of global system and service state
  • journald, central I/O bottleneck
  • In conclusion

Referenced

In order of appearance in the piece

Background

Unreferenced

And for a guy interest in respect for the elders who have trod the trails before stand silent

Via: backfill.

Enabling and configuring a static iptables firewall in Fedora 21 (Workstation or Server)

$ sudo yum install -y iptables-services
Loaded plugins: langpacks
Resolving Dependencies
--> Running transaction check
---> Package iptables-services.i686 0:1.4.21-13.fc21 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================
 Package                    Arch          Version                  Repository                  Size
====================================================================================================
Installing:
 iptables-services          i686          1.4.21-13.fc21           collected-by-file           53 k

Transaction Summary
====================================================================================================
Install  1 Package

Total download size: 53 k
Installed size: 19 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction (shutdown inhibited)
  Installing : iptables-services-1.4.21-13.fc21.i686                                            1/1 
warning: /etc/sysconfig/ip6tables created as /etc/sysconfig/ip6tables.rpmnew
warning: /etc/sysconfig/iptables created as /etc/sysconfig/iptables.rpmnew
  Verifying  : iptables-services-1.4.21-13.fc21.i686                                            1/1 

Installed:
  iptables-services.i686 0:1.4.21-13.fc21                                                           

Complete!
$ sudo systemctl enable iptables
Created symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service.

$ sudo systemctl start iptables
Job for iptables.service failed. See "systemctl status iptables.service" and "journalctl -xe" for details.

Folklore

Configuring Apache httpd on Fedora 22

Problem

Configure Apache httpd to  (temporarily) run out of a ServerRoot other than the default which is /etc/httpd.

Solution

Create the file /etc/systemd/system/httpd.service containing:

.include /lib/systemd/system/httpd.service
Environment="OPTIONS=-d /local/project/onstage/httpd"

Quotes go around the outside

Referenced

References

Actualities

/etc/sysconfig/httpd

#
# This file is no longer used to configure additional environment variables
# for the httpd process.
#
# It has been replaced by systemd. If you want to customize, the best
# way is to create a file "/etc/systemd/system/httpd.service",
# containing
#   .include /lib/systemd/system/httpd.service
#   ...make your changes here...
# For more info about custom unit files, see How do I set automatic login on a virtual console terminal?

# To set OPTIONS environment variable which was set in this file
# in older versions, you need to create a file named
# "/etc/systemd/system/httpd.service" containing:
#       .include /lib/systemd/system/httpd.service
#       [Service]
#       #
#       # To pass additional options (for instance, -D definitions) to the
#       # httpd binary at startup, set OPTIONS here.
#       #
#       Environment=OPTIONS=-DMY_DEFINE

# Note: With previous versions of httpd, the MPM could be changed by
# editing an "HTTPD" variable here.  With the current version, that
# variable is now ignored.  The MPM is a loadable module, and the
# choice of MPM can be changed by editing the configuration file
# /etc/httpd/conf.modules.d/00-mpm.conf.

Systemd Configuration of Apache httpd

$ find /lib/systemd/system/httpd.s*
/lib/systemd/system/httpd.service
/lib/systemd/system/httpd.service.d
/lib/systemd/system/httpd.socket
/lib/systemd/system/httpd.socket.d

/lib/systemd/system/httpd.service

# It's not recommended to modify this file in-place, because it will be
# overwritten during package upgrades.  If you want to customize, the best
# way is to create a file "/etc/systemd/system/httpd.service",
# containing
#   .include /lib/systemd/system/httpd.service
#   ...make your changes here...
# For more info about custom unit files, see
# http://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F

# For example, to pass additional options (for instance, -D definitions) to the
# httpd binary at startup, you need to create a file named
# "/etc/systemd/system/httpd.service" containing:
#	.include /lib/systemd/system/httpd.service
#	[Service]
#	Environment=OPTIONS=-DMY_DEFINE

[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
Environment=LANG=C

ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
# Send SIGWINCH for graceful stop
KillSignal=SIGWINCH
KillMode=mixed
PrivateTmp=true

[Install]
WantedBy=multi-user.target

SOLVED: MySQL command ‘describe TABLE’ fails with ERROR 1 (HY000): Can’t create/write to file ‘/tmp/#sql_718d_0.MYI’ (Errcode: 2)

Diagnostic

mysql> describe TABLE;
ERROR 1 (HY000): Can't create/write to file '/tmp/#sql_718d_0.MYI' (Errcode: 2)

Remediation

Restart mysqld, which will create its own temporary areas upon startup

systemctl restart mysqld.service

Create a tmpfiles specification that preserves the temporary spaces

$ cd /etc/tmpfiles.d
$ sudo vi mysql.conf
$ cat mysql.conf 
x /tmp/systemd-namespace-* x /tmp/systemd-namespace-*/private

Modify in place the cron entry for tmpwatch in /etc/cron.daily/tmpwatch to preserve the temporary spaces

$ cat /etc/cron.daily/tmpwatch 
#! /bin/sh
flags=-umc
/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
	-x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \
	-X '/tmp/systemd-namespace*' \
	-X '/tmp/hsperfdata_*' 10d /tmp
/usr/sbin/tmpwatch "$flags" 30d /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
    if [ -d "$d" ]; then
	/usr/sbin/tmpwatch "$flags" -f 30d "$d"
    fi
done

Explanation

MySQL uses a temporary directory within /tmp that is only visible to mysqld.  If this directory is not used enough then it is removed by either or both tmpwatch or systemd-tmpfiles.

Background

$ grep names /proc/22489/mountinfo 
148 104 253:2 /tmp/systemd-namespace-UAhzsT/private /tmp rw,relatime - ext4 /dev/mapper/vg_hangie-lv_root rw,seclabel,data=ordered

Referenced

SOLVED httpd fails to start | Failed to load environment files: No such file or directory

Diagnosis

tl;dr => the file /etc/sysconfig/httpd is missing.

Remediation

Add one

Indications

$ sudo systemctl start httpd.service
Job failed. See system journal and 'systemctl status' for details.
$ systemctl status httpd
httpd.service - The Apache HTTP Server (prefork MPM)
	  Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
	  Active: failed (Result: resources)
	  CGroup: name=systemd:/system/httpd.service
Feb 18 14:13:44 loopy systemd[1]: Failed to load environment files: No such file or directory
Feb 18 14:13:44 loopy systemd[1]: httpd.service failed to run 'start' task: No such file or directory
Feb 18 14:13:44 loopy systemd[1]: Unit httpd.service entered failed state.

Background

$ cat /etc/fedora-release 
Fedora release 17 (Beefy Miracle)
$ rpm -q httpd systemd
httpd-2.2.22-4.fc17.i686
systemd-44-21.fc17.i686

/lib/systemd/system/httpd.service

[Unit]
Description=The Apache HTTP Server (prefork MPM)
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/var/run/httpd/httpd.pid
EnvironmentFile=/etc/sysconfig/httpd
ExecStart=/usr/sbin/httpd $OPTIONS -k start
ExecReload=/usr/sbin/httpd $OPTIONS -t
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/sbin/httpd $OPTIONS -k stop
PrivateTmp=true

[Install]
WantedBy=multi-user.target

/etc/sysconfig/httpd

# Configuration file for the httpd service.

#
# Note: With previous versions of httpd, the MPM could be changed by
# editing an "HTTPD" variable here. With the current version, that
# variable is now ignored. A particular systemd service must be
# chosen corresponding to the desired MPM:
#
# httpd.service => prefork MPM
# httpd-worker.service => worker MPM
# httpd-event.service => event MPM
#
# Use systemctl to stop/start between MPMs, and to disable/enable
# whichever service is required to start at boot time.
#

#
# To pass additional options (for instance, -D definitions) to the
# httpd binary at startup, set OPTIONS here.
#
#OPTIONS=

#
# This setting ensures the httpd process is started in the "C"
# locale by default:
#
LANG=C

systemd cgroup changes | Lennart Poettering

Lennart Poettering (Red Hat)

Via backfill

Mentions

Mostly from Part I

  • Tejun Heo, (Linux kernel) cgroup maintainer

Deprecating:

Introducing:

  • Slices
  • single kernel cgroup hierarchy

Capabilites

  • partition system resources in a tree
  • move to arbitrarity places within a tree
    • units
    • users
    • machines

Reduction

  • Only a single cgroup tree
  • Controllers enabled/disabled separately per cgroup

Removed

  • ControlGroup=
    Replaced with

    • Slice=
    • EnableControllers=
    • and others
  • ControlGroupPersistent=
    Replaced with

    • only systemd sets up the cgroup tree
  • ControlGroupAttribute=
    Replace with (called the High Level Attributes)

    • CPUShares=
    • MemoryLimit=
    • others TBD
  • systemctl set-cgroup
    Replaced with

    • systemctl set-slice (or similar)
  • systemctl set-cgroup-attr
    Replaced with

    • systemctl set-attr (which only sets the high-level attributes
  • undocumented APIs (of systemd), no replacement

From Part II

Current dev (in git)

  • the Slice concept
  • logind will now also keep track of running containers/VMs.
  • ps can show the cgroups (and containers?) of a process.

Theory

  • Only a one single kernel cgroup.
  • The controllers individually enabled for each cgroup.
  • The cgroup hierarchy is private property of systemd
    • systemd sets it up.
    • systemd maintains it.
  • Any software wishing to manipulate cgroups will do so via systemd APIs.
  • Slices map to cgroups internally
  • Slices only allow high-level constraints.
  • There will be at least three slices:
    1. The system.slice where all system services are located by default,
    2. The user.slice where all logged in users are located by default,
    3. The machine.slice where all running VMs/containers are located by default.
    4. Others as created by administrators
  • systemd-logind is responsible for
    • users & sessions
    • machines & containers
  • Something about registration of machines / containers / VMs with systemd
    • so that ps will work
    • so that it behaves like Solaris’ zones concept.

Removed

  • unit configuration options
    • ControlGroup=
    • ControlGroupModify=
    • ControlGroupPersistent=
    • ControlGroupAttribute=
  • DefaultControllers=cpu

Remains

  • CPUShares=
  • MemoryLimit=
  • other high-level constraints

Timeline

  • #1 (single cgroup) => long term, not now
  • #2 (single cgroup owned by systemd) => long term, not now
  • #3 (systemd removes ControlGroup settings) => in development today
  • #4 (systemd slice concept) => implemented in systemd upstream
  • #5 (systemd-logind owns users & vms) => implemented in systemd upstream

Compatibilities

  • not supported; becomes deprecated

  • ok for now, but on notice to evolve (per #1 & #2 not scheduled)

The Thread

[systemd-devel] [HEADSUP] cgroup changes Lennart Poettering

[systemd-devel] [HEADSUP] cgroup changes Lennart Poettering