Mendeley on Fedora

Fedora

  • use apt (Debian’s Advanced Packaging Tool with RPM support)

Availability

Ubuntu 12.04 or Debian Squeeze and newer

Folklore

Fedora 25, installation notes & experiences

Issues

  • IPv6 addresses come up with RFC7217 privacy mode enabled
    As such, the local radvd does not tag the machine with a “known” address.
    Remediation: turn off IPV6_ADDR_GEN_MODE=stable-privacy or set IPV6_ADDR_GEN_MODE=eui64 in the relevant /etc/sysconfig/network-scripts/enp1s0.

Reminder

Fedora Live Workstation…

  • … does not enable sshd. The firewall is configured to allow it, but the service is not enabled or started after the build.
  • … builds to graphical.target.  To back down to the non-graphical mode, systemctl set-default multi-user.target.  See the guidance in the (legacy) /etc/inittab commentary.
  • … uses firewalld to manage the iptables.  If you need to install a custom iptables setup, e.g. with xtables-addons xt_geoip rules then you need iptable-services.

Actualities

sudo dnf install -y xtables-addons

See the separate recipe for bringing down firewalld and bringing up the separable iptables services

systemctl get-default
sudo systemctl set-default multi-user.target
sudo systemctl enable sshd
sudo systemctl start sshd
nmcli reload
nmcli modify enp1s0 ipv5.addr-gen-mode eui64
nmcli con down enp1s0
nmcli con up enp1s0
$ cat /etc/sysconfig/network-scripts/ifcfg-enp1s0
HWADDR=00:EC:AC:CD:E6:12
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp1s0
UUID=6c463f92-11d2-30ba-8273-d86bb3c58859
ONBOOT=yes
AUTOCONNECT_PRIORITY=-999
PEERDNS=yes
PEERROUTES=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes

References

Standards

RFC 7217
A Method for Generating Semantically Opaque Interface Identifiers with IPv6 Stateless Address Autoconfiguration (SLAAC)
F. Gont (SI6 Networks & UTN-FRH); IETF; 2014-04.
Abstract: This document specifies a method for generating IPv6 Interface Identifiers to be used with IPv6 Stateless Address Autoconfiguration (SLAAC), such that an IPv6 address configured using this method is stable within each subnet, but the corresponding Interface Identifier changes when the host moves from one network to another. This method is meant to be an alternative to generating Interface Identifiers based on hardware addresses (e.g., IEEE LAN Media Access Control (MAC) addresses), such that the benefits of stable addresses can be achieved without sacrificing the security and privacy of users. The method specified in this document applies to all prefixes a host may be employing, including link-local, global, and unique-local prefixes (and their corresponding addresses).
RFC 4941
Privacy Extensions for Stateless Address Autoconfiguration in IPv6
Narten (IBM), Draves (Microsoft) Krishnan (Ericsson); IETF; 2007-09.
Abstract: Nodes use IPv6 stateless address autoconfiguration to generate addresses using a combination of locally available information and information advertised by routers. Addresses are formed by combining network prefixes with an interface identifier. On an interface that contains an embedded IEEE Identifier, the interface identifier is typically derived from it. On other interface types, the interface identifier is generated through other means, for example, via random number generation. This document describes an extension to IPv6 stateless address autoconfiguration for interfaces whose interface identifier is derived from an IEEE identifier. Use of the extension causes nodes to generate global scope addresses from interface identifiers that change over time, even in cases where the interface contains an embedded IEEE identifier. Changing the interface identifier (and the global scope addresses generated from it) over time makes it more difficult for eavesdroppers and other information collectors to identify when different addresses used in different transactions actually correspond to the same node.

How much swap space for Fedora?

The answer has evolved over time

Quoting, paraphrasing…

Circa Fedora 25

Recommended swap space
System RAM No hibernation Allowing for hibernation
less than 2 GB 2 times the amount of RAM 3 times the amount of RAM
2 GB – 8 GB Equal to the amount of RAM 2 times the amount of RAM
8 GB – 64 GB 0.5 times the amount of RAM 1.5 times the amount of RAM
more than 64 GB workload dependent hibernation not recommended

At the border between each range listed above (for example, a system with 2 GB, 8 GB, or 64 GB of system RAM), discretion can be exercised with regard to chosen swap space and hibernation support. If the system resources allow for it, increasing the swap space may lead to better performance.

Via Installation, GUI Manual Partitioning Recommendation, In Installation Guide, Fedora 25

Circa Fedora 16

M = Amount of RAM in GB, and
S = Amount of swap in GB, then

If M < 2
    S = M *2
Else
    S = M + 2
System RAM Recommended Amount of Swap Space
4GB of RAM or less a minimum of 2GB of swap space
4GB to 16GB of RAM a minimum of 4GB of swap space
16GB to 64GB of RAM minimum of 8GB of swap space
64GB to 256GB of RAM a minimum of 16GB of swap space
256GB to 512GB of RAM a minimum of 32GB of swap space

One can obtain better performance by distributing swap space over multiple storage devices, particularly on systems with fast drives, controllers, and interfaces.

Via Disk Partition Recommendation for x86, In Installation Guide, Fedora 16.

Circa Fedora 14

There is a rule for swap space that is some think as follows:

  • For machines up to 4 gigs of ram, it is 1.5 times the amount of ram.
  • For machines above, it is the larger of 6 gigs or the amount of ram in your system. stopping at 8 gigs.

Since you may want to also use hybernate or suspend, add 2 gigs to the above.

[There is] doubt that one would ever use even 8 gigs for swap.
16 gigs is extremely generous (waste of diskspace).
One can also use two swap files of 4 gigs each.

Via Swap Space, In Storage Administration Guide, Fedora 14.

Experience with Let’s Encrypt certbot for Fedora 23 (fails)

At certbot.eff.org with Apache on Fedora 23+

sudo dnf install -y python-certbot-apache
Error: nothing provides python2-augeas needed by python2-certbot-apache-0.8.1-1.fc23.noarch
(try to add '--allowerasing' to command line to replace conflicting packages)

Flailing

dnf install -y augeas
dnf install -y python-augeas

Therefore: certbot isn’t ready for Fedora 23 yet.

Fedora 22?

Fail.

wget https://dl.eff.org/certbot-auto

Nope … too big and complicated … it will never work … and they didn’t test it on Fedora anyway.

Manual

Prerequisites of python-certbot-apache

dialog
python-parsedatetime
python-zope-component
python-zope-event
python-zope-interface
python2-acme
python2-certbot
python2-certbot-apache
python2-configargparse
python2-configobj
python2-dialog
python2-funcsigs
python2-mock
python2-pbr
python2-psutil
python2-pyrfc3339
pytz

Still fails

$ sudo dnf install python2-certbot-apache
Last metadata expiration check performed 2:49:52 ago on Wed Sep 28 04:06:26 2016.
Error: nothing provides python2-augeas needed by python2-certbot-apache-0.8.1-1.fc23.noarch
(try to add '--allowerasing' to command line to replace conflicting packages)

Workaround

wget https://dl.fedoraproject.org/pub/fedora/linux/updates/23/x86_64/p/python2-certbot-apache-0.8.1-1.fc23.noarch.rpm
sudo rpm --install --nodeps python2-certbot-apache-0.8.1-1.fc23.noarch.rpm

What got installed?

$ rpm -q -l -p ./python2-certbot-apache-0.8.1-1.fc23.noarch.rpm  | grep -v test
/usr/lib/python2.7/site-packages/certbot_apache
/usr/lib/python2.7/site-packages/certbot_apache-0.8.1-py2.7.egg-info
/usr/lib/python2.7/site-packages/certbot_apache-0.8.1-py2.7.egg-info/PKG-INFO
/usr/lib/python2.7/site-packages/certbot_apache-0.8.1-py2.7.egg-info/SOURCES.txt
/usr/lib/python2.7/site-packages/certbot_apache-0.8.1-py2.7.egg-info/dependency_links.txt
/usr/lib/python2.7/site-packages/certbot_apache-0.8.1-py2.7.egg-info/entry_points.txt
/usr/lib/python2.7/site-packages/certbot_apache-0.8.1-py2.7.egg-info/requires.txt
/usr/lib/python2.7/site-packages/certbot_apache-0.8.1-py2.7.egg-info/top_level.txt
/usr/lib/python2.7/site-packages/certbot_apache/__init__.py
/usr/lib/python2.7/site-packages/certbot_apache/__init__.pyc
/usr/lib/python2.7/site-packages/certbot_apache/__init__.pyo
/usr/lib/python2.7/site-packages/certbot_apache/augeas_configurator.py
/usr/lib/python2.7/site-packages/certbot_apache/augeas_configurator.pyc
/usr/lib/python2.7/site-packages/certbot_apache/augeas_configurator.pyo
/usr/lib/python2.7/site-packages/certbot_apache/augeas_lens
/usr/lib/python2.7/site-packages/certbot_apache/augeas_lens/httpd.aug
/usr/lib/python2.7/site-packages/certbot_apache/centos-options-ssl-apache.conf
/usr/lib/python2.7/site-packages/certbot_apache/configurator.py
/usr/lib/python2.7/site-packages/certbot_apache/configurator.pyc
/usr/lib/python2.7/site-packages/certbot_apache/configurator.pyo
/usr/lib/python2.7/site-packages/certbot_apache/constants.py
/usr/lib/python2.7/site-packages/certbot_apache/constants.pyc
/usr/lib/python2.7/site-packages/certbot_apache/constants.pyo
/usr/lib/python2.7/site-packages/certbot_apache/display_ops.py
/usr/lib/python2.7/site-packages/certbot_apache/display_ops.pyc
/usr/lib/python2.7/site-packages/certbot_apache/display_ops.pyo
/usr/lib/python2.7/site-packages/certbot_apache/obj.py
/usr/lib/python2.7/site-packages/certbot_apache/obj.pyc
/usr/lib/python2.7/site-packages/certbot_apache/obj.pyo
/usr/lib/python2.7/site-packages/certbot_apache/options-ssl-apache.conf
/usr/lib/python2.7/site-packages/certbot_apache/parser.py
/usr/lib/python2.7/site-packages/certbot_apache/parser.pyc
/usr/lib/python2.7/site-packages/certbot_apache/parser.pyo
/usr/lib/python2.7/site-packages/certbot_apache/tls_sni_01.py
/usr/lib/python2.7/site-packages/certbot_apache/tls_sni_01.pyc
/usr/lib/python2.7/site-packages/certbot_apache/tls_sni_01.pyo
/usr/share/doc/python2-certbot-apache
/usr/share/doc/python2-certbot-apache/README.rst
/usr/share/licenses/python2-certbot-apache
/usr/share/licenses/python2-certbot-apache/LICENSE.txt

You also have to install

certbot

. It will list, but fails to create, the directories /etc/letsencrypt and /var/lib/letsencrypt

$ sudo dnf install certbot
Last metadata expiration check performed 0:18:54 ago on Wed Sep 28 07:09:29 2016.
Dependencies resolved.
====================================================================================================
 Package               Arch                 Version                     Repository             Size
====================================================================================================
Installing:
 certbot               noarch               0.8.1-2.fc23                updates                20 k

Transaction Summary
====================================================================================================
Install  1 Package

Total download size: 20 k
Installed size: 20 k
Is this ok [y/N]: y
Downloading Packages:
certbot-0.8.1-2.fc23.noarch.rpm                                      42 kB/s |  20 kB     00:00    
----------------------------------------------------------------------------------------------------
Total                                                                16 kB/s |  20 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Installing  : certbot-0.8.1-2.fc23.noarch                                                     1/1 
  Verifying   : certbot-0.8.1-2.fc23.noarch                                                     1/1 

Installed:
  certbot.noarch 0.8.1-2.fc23                                                                       

Complete!
$ rpm -q -l certbot
/etc/letsencrypt
/usr/bin/certbot
/usr/bin/letsencrypt
/usr/share/doc/certbot
/usr/share/doc/certbot/CHANGES.rst
/usr/share/doc/certbot/CONTRIBUTING.md
/usr/share/doc/certbot/README.rst
/usr/share/licenses/certbot
/usr/share/licenses/certbot/LICENSE.txt
/var/lib/letsencrypt
$ rpm -q -l certbot | xargs ls -ld
ls: cannot access /etc/letsencrypt: No such file or directory
ls: cannot access /var/lib/letsencrypt: No such file or directory
-rwxr-xr-x. 1 root root   302 Jul  6 06:42 /usr/bin/certbot
lrwxrwxrwx. 1 root root    16 Jul  6 06:42 /usr/bin/letsencrypt -> /usr/bin/certbot
drwxr-xr-x. 2 root root  4096 Sep 28 07:28 /usr/share/doc/certbot
-rw-r--r--. 1 root root   362 Jun 14 16:46 /usr/share/doc/certbot/CHANGES.rst
-rw-r--r--. 1 root root   604 Jun 14 16:46 /usr/share/doc/certbot/CONTRIBUTING.md
-rw-r--r--. 1 root root  7702 Jun 14 16:46 /usr/share/doc/certbot/README.rst
drwxr-xr-x. 2 root root  4096 Sep 28 07:28 /usr/share/licenses/certbot
-rw-r--r--. 1 root root 11456 Jun 14 16:46 /usr/share/licenses/certbot/LICENSE.txt
$ certbot plugins
An unexpected error occurred:
OSError: [Errno 13] Permission denied: '/etc/letsencrypt'
Please see the logfile 'certbot.log' for more details.

You have to do it yourself:

sudo mkdir /etc/letsencrypt /var/lib/letsencrypt

Pretty much, RSA is your only reasonable, reliable & compatible option in OpenSSH

Whereas

  • DSA is deprecated in OpenSSH 7.0
  • ECDSA is not supported by GNOME Keyring.
  • Ed25519 is not supported by GNOME Keyring.

Folklore

Via SSH Keys, in Arch Linux Wiki

<quote>

<snip/>

  • As of July 10, 2015, GNOME Keyring does not handle ECDSA[4] and Ed25519[5] keys. Users will have to turn to other SSH agents or stick to RSA keys.
  • These keys are used only to authenticate you; choosing stronger keys will not increase CPU load when transferring data over SSH.

</quote>

Via How to save an SSH key passphrase in gnome-keyring? in Stack Exchange for Unix & Linux

cd $HOME/.ssh
/usr/lib/seahorse/seahorse-ssh-askpass my_key

References

In Arch Linux Wiki

In GNOME Wiki

SOLVED On the origin of the “No supported key exchange algorithms” error message of sshd

sshd shuts down with “No supported key exchange algorithms” error; Dmitry Gladkov; in Server Fault; 2010-07-07.

Actualities

us
Feb 23 12:19:36 host.example.com sshd[967]: Server listening on 0.0.0.0 port 22.
Feb 23 12:19:36 host.example.com sshd[967]: Server listening on :: port 22.
Feb 23 12:19:56 host.example.com sshd[1361]: fatal: No supported key exchange algorithms [preauth]

Diagnosis

The ssh host key files are readable by more than merely the owner

Incorrect

$ ls -l /etc/ssh/ssh*key*
-rw-r-----. 1 root root  227 Feb 23 11:56 /etc/ssh/ssh_host_ecdsa_key
-rw-r--r--. 1 root root  162 Feb 23 11:56 /etc/ssh/ssh_host_ecdsa_key.pub
-rw-r-----. 1 root root  387 Feb 23 11:56 /etc/ssh/ssh_host_ed25519_key
-rw-r--r--. 1 root root   82 Feb 23 11:56 /etc/ssh/ssh_host_ed25519_key.pub
-rw-r-----. 1 root root 1675 Feb 23 11:56 /etc/ssh/ssh_host_rsa_key
-rw-r--r--. 1 root root  382 Feb 23 11:56 /etc/ssh/ssh_host_rsa_key.pub

Correct

$ ls -l /etc/ssh/ssh*key*
-rw-------. 1 root root  227 Feb 23 11:56 /etc/ssh/ssh_host_ecdsa_key
-rw-r--r--. 1 root root  162 Feb 23 11:56 /etc/ssh/ssh_host_ecdsa_key.pub
-rw-------. 1 root root  387 Feb 23 11:56 /etc/ssh/ssh_host_ed25519_key
-rw-r--r--. 1 root root   82 Feb 23 11:56 /etc/ssh/ssh_host_ed25519_key.pub
-rw-------. 1 root root 1675 Feb 23 11:56 /etc/ssh/ssh_host_rsa_key
-rw-r--r--. 1 root root  382 Feb 23 11:56 /etc/ssh/ssh_host_rsa_key.pub

Structural and semantic deficiencies in the systemd architecture, another jeremiad | V.R.

Some dude using the self-asserted identity token V.R.; Structural and semantic deficiencies in the systemd architecture for real-world service management, a technical treatise; In Some Blog; 2015-10-11.

tl;dr → A jeremiad; 8100 words; systemd is bad, he doesn’t like it.

Mentions

His point, and he does have one, is that
  • There are simply too many notes
  • And there is insufficient reference to the priors in the art
  • Oh! and it’s full of bugs! Lots! Of! Bugs!
Not shown:
  • the way forward
  • a complete viable alternative
    • complete in a technical sense of solving the problem
    • complete in a cultural sense of having an adiabatic transition to the new phase
  • that sticking with tangled masses of stylized /bin/sh (ahem, the SysV initscripts), better, possible or even an option.  That system worked “well enough ” that you knew getting away from it would be messy.
And yet
  • Mel Conway’s Law is iron
  • Lennart Poettering & Kay Sievers operate as a single organization.
  • Therefore systemd evolves onward as a single-process central-element architectural solution to the problems it addresses; all-in or out.
    Similar to the monolithic_macrokernel-vs-microkernel culture wars of the ’90s.  The Linux kernel is … go on, say it.

Outline

  • Preface and disclaimer (!)
  • Everything is a Unit (but it doesn’t mean a lot)
  • Job queuing
  • The transaction manager
  • To live is to depend
  • Every problem can be solved by a layer of indirection
  • Bus APIs, connections and object interface duplication
  • cgroup writing
  • Parsing in critical paths
  • Non-generic fd-holding and socket preopening
  • Inexpressive unit file options
  • Imbalance between promoting laziness or eagerness
  • Targets over milestones for synchronization
  • The (system-specific) problem of readiness notification
  • Intertwining of global system and service state
  • journald, central I/O bottleneck
  • In conclusion

Referenced

In order of appearance in the piece

Background

Unreferenced

And for a guy interest in respect for the elders who have trod the trails before stand silent

Via: backfill.

Brave (browser)

Brave (browser)

Mentions

  • Available
    • no
    • circa v0.7
    • assemble the sources yourself
    • closed beta program.
  • Cultures
    • Linux
    • Mac (OS/X)
    • Windows (sic)
    • Android
    • iPhone (iOS)
  • Basis
    • Chromium → Linux, Mac, Windows
    • iOS → Firefox for iOS
    • Android → Bubble (linkbubble)
  • linkbubble
  • Funding
    • $2.5 million
    • Unnamed individuals
      “angel” investment.
  • Features
    • Known
      • HTTPS Everywhere add-on
    • Expected, not declared as existing
      • a UI
      • cross-platform sync
      • incognito mode
      • password manager

Source

Promotions

  • Mozilla co-founder unveils Brave, a Web browser that blocks ads by default; ; In Ars Technica; 2016-01-21.
    Teaser: … but Brave then replaces blocked ads with its own ads, taking a 15% cut of revenues.
    Mentions

    • <quote>In practice, Brave just sounds like a cash-grab. Brave isn’t just a glorified adblocker: after removing ads from a Web page, Brave then inserts its own programmatic ads</quote>
  • Brendan Eich Launches Brave New Browser Ian Elliot; In I Programmer; 20165-01-20.
    Teaser: Brendan Eich, the man who invented JavaScript and the co-founder of Mozilla, has just launched a new browser called Brave. Is this a Firefox fork?

Via: backfill.

PEERROUTES is created by NetworkManager, controls ignore-auto-routes, controls the peer routes, use PEERROUTES=yes

PEERROUTES and IPv6_PEERROUTES is created by NetworkManager, it controls ignore-auto-routes, and the establishment of the peer routes, you want

PEERROUTES=yes
IPV6_PEERROUTES=yes

Absent those settings, there will be no peer routes (you typically want to be route to the peers on the link)
e.g.

  • Address 2001:db8::1/64
  • Route to 2001:db8::/64 via DEVICE

Specimen

/etc/sysconfig/network-scripts/ifcfg-enp2s0

# Initially generated by dracut initrd
DEVICE="enp2s0"
ONBOOT=yes
NETBOOT=yes
UUID="550adb0f-d9ba-4da3-9214-1ffdc18dae7e"
IPV6INIT=yes
BOOTPROTO=dhcp
TYPE=Ethernet
NAME="enp2s0"
IPV4_FAILURE_FATAL=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
DEFROUTE=yes
PEERDNS=no
PEERROUTES=yes
IPv6_DEFROUTE=yes
IPV6_PEERDNS=no
IPV6_PEERROUTES=yes

Concept

Key Name Value Type DefaultValue Value Description
ignore-auto-routes boolean FALSE When the method is set to ‘auto’ and this property to TRUE, automatically configured routes are ignored and only routes specified in the ‘routes’ property, if any, are used.

Folklore

  • 1107328“PEERROUTES” can be found in “/etc/sysconfig/network-scripts/ifcfg-em1″ on new system. Isn’t that obsolete? ; In Bugzilla of Red Hat; 2014-06-09; CLOSED.

References

  • ignore-auto-routes for IPv4 (Table 11) & IPv6 (Table 12); In NetworkManager D-Bus Reference Manual, for NetworkManager v0.9

[SOLVED] rngd: read error, No entropy sources working, exiting rngd

Explanation

<quote>

rngd has three potential sources of randomness:

  • the RdRand instruction present in some x86 CPUs.
  • a system hardware random number generator at /dev/hwrng (not /dev/hwrandom).
  • a trusted platform module at /dev/tpm0

If your CPU doesn’t support RdRand and you don’t have either of those devices, rngd won’t get triggered to start (and if it did, it would fail on startup).

</quote>

Via: commentariat; Shea Levy; In archives of some mailing list; 2012-11-29

Context

Folklore

Actualities

There are enough files...

$ ls -ld /dev/*random* /dev/*rng* /dev/tpm0
ls: cannot access /dev/tpm0: No such file or directory
crw-------. 1 root root 10, 183 Dec 27 17:47 /dev/hwrng
crw-rw-rw-. 1 root root  1,   8 Dec 27 17:47 /dev/random
crw-rw-rw-. 1 root root  1,   9 Dec 27 17:47 /dev/urandom

The daemon attempts to read, but fails and then exits.

Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: read error
Dec 27 18:20:22 server rngd: No entropy sources working, exiting rngd

On modern Fedora, use iptables-services instead of firewalld for edge hosts

For when the firewall rules are terribly complex, or you need to use a nonstandard module such as geoip from xtables-addons


$ yum search iptables-service
Loaded plugins: langpacks
===================================================== N/S matched: iptables-service ======================================================
iptables-services.i686 : iptables and ip6tables services for iptables

Name and summary matches only, use "search all" for everything.
$ sudo yum install -y iptables-services
Loaded plugins: langpacks
collected-by-file                                                                                                  | 3.0 kB  00:00:00
collected-by-http                                                                                                  | 3.0 kB  00:00:00
rpmfusion-free-updates                                                                                             | 2.7 kB  00:00:00
rpmfusion-nonfree-updates                                                                                          | 2.7 kB  00:00:00
updates/21/i386/metalink                                                                                           |  12 kB  00:00:00
Package iptables-services-1.4.21-13.fc21.i686 already installed and latest version
Nothing to do

Notes on the Configuration of Kerberos: Services nfs-secure and nfs-secure-server must be restarted together

NFS (Client) and also NFS Server

Indications

  • syslog shows
    • gssproxy complaining
    • rpc-gssd segfaulting (only on i686?)
    • NFSv4 kernel error message nfs4_discover_server_trunking unhandled
  • NFS client services do not work at all (they hang)
  • yet all other configurations are correct (believed correct & consistent)
On an i686
Dec 20 13:37:46 flowerpot kernel: [14516673.028093] rpc.gssd[12963]: segfault at 2 ip b74de64a sp bf754450 error 4 in libc-2.20.so[b7466000+1c5000]
Dec 20 13:37:46 flowerpot gssproxy: gssproxy[12962]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may provide more information, No credentials cache found
Dec 20 13:37:46 flowerpot kernel: rpc.gssd[12963]: segfault at 2 ip b74de64a sp bf754450 error 4 in libc-2.20.so[b7466000+1c5000]
Dec 20 13:37:46 flowerpot rpc.gssd[12956]: WARNING: forked child was killed with signal 11

Dec 20 13:55:22 flowerpot kernel: [14517728.877846] NFS: nfs4_discover_server_trunking unhandled error -32. Exiting with error EIO
Dec 20 13:55:22 flowerpot kernel: NFS: nfs4_discover_server_trunking unhandled error -32. Exiting with error EIO
On an x86_64
Dec 21 18:21:40 truckfarm rpc.gssd[336]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clntfa)
Dec 21 18:21:40 truckfarm rpc.gssd[336]: handle_gssd_upcall: 'mech=krb5 uid=0 service=* enctypes=18,17,16,23,3,1,2 '
Dec 21 18:21:40 truckfarm rpc.gssd[384]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clntfa)
Dec 21 18:21:40 truckfarm rpc.gssd[384]: process_krb5_upcall: service is '*'
Dec 21 18:21:40 truckfarm rpc.gssd[384]: Full hostname for 'trout.department.example.com' is 'trout.department.example.com'
Dec 21 18:21:40 truckfarm rpc.gssd[384]: Full hostname for 'truckfarm.department.example.com' is 'truckfarm.department.example.com'
Dec 21 18:21:40 truckfarm rpc.gssd[384]: No key table entry found for TRUCKFARM$@EXAMPLE.COM while getting keytab entry for 'TRUCKFARM$@EXAMPLE.COM'
Dec 21 18:21:40 truckfarm rpc.gssd[384]: No key table entry found for root/truckfarm.department.example.com@EXAMPLE.COM while getting keytab entry for 'root/truckfarm.department.example.com@EXAMPLE.COM'
Dec 21 18:21:40 truckfarm rpc.gssd[384]: Success getting keytab entry for 'nfs/truckfarm.department.example.com@EXAMPLE.COM'
Dec 21 18:21:40 truckfarm rpc.gssd[384]: Successfully obtained machine credentials for principal 'nfs/truckfarm.department.example.com@EXAMPLE.COM' stored in ccache 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM'
Dec 21 18:21:40 truckfarm rpc.gssd[384]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until 1450837300
Dec 21 18:21:40 truckfarm rpc.gssd[384]: using FILE:/tmp/krb5ccmachine_EXAMPLE.COM as credentials cache for machine creds
Dec 21 18:21:40 truckfarm rpc.gssd[384]: using environment variable to select krb5 ccache FILE:/tmp/krb5ccmachine_EXAMPLE.COM
Dec 21 18:21:40 truckfarm kernel: [14614771.266147] rpc.gssd[384]: segfault at 2 ip b747564a sp bf8a4790 error 4 in libc-2.20.so[b73fd000+1c5000]
Dec 21 18:21:40 truckfarm gssproxy: gssproxy[329]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may provide more information, No credentials cache found
Dec 21 18:21:40 truckfarm kernel: rpc.gssd[384]: segfault at 2 ip b747564a sp bf8a4790 error 4 in libc-2.20.so[b73fd000+1c5000]
Dec 21 18:21:40 truckfarm rpc.gssd[336]: WARNING: forked child was killed with signal 11

Remediation

  1. restart nfs-secure-server (remember, you didn’t do that because it wasn’t supposed to be running)
  2.   disable nfs-server nfs-secure-server

Actualities

[as wbaker@truckfarm F21.Twenty_One]

$ systemctl status nfs-secure-server
● rpc-svcgssd.service - RPC security service for NFS server
Loaded: loaded (/usr/lib/systemd/system/rpc-svcgssd.service; static)
Active: inactive (dead)
start condition failed at Sun 2015-12-20 13:53:02 PST; 57s ago

$ systemctl is-enabled nfs-secure-server
static

$ systemctl is-active nfs-secure-server
inactive

$ systemctl is-active nfs-server
active

$ systemctl is-enabled nfs-server
enabled

$ sudo systemctl restart nfs-secure nfs-secure-server nfs-idmapd gssproxy

$ systemctl status nfs-secure-server
● rpc-svcgssd.service - RPC security service for NFS server
Loaded: loaded (/usr/lib/systemd/system/rpc-svcgssd.service; static)
Active: inactive (dead)
start condition failed at Sun 2015-12-20 13:55:22 PST; 1min 20s ago
none of the trigger conditions were met

$ systemctl status nfs-secure
● rpc-gssd.service - RPC security service for NFS client and server
Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static)
Active: active (running) since Sun 2015-12-20 13:55:22 PST; 1min 34s ago
Process: 13810 ExecStart=/usr/sbin/rpc.gssd $GSSDARGS (code=exited, status=0/SUCCESS)
Main PID: 13811 (rpc.gssd)
CGroup: /system.slice/rpc-gssd.service
└─13811 /usr/sbin/rpc.gssd -v -v -v

$ cat /etc/exports
<empty>

$ sudo systemctl disable nfs-secure-server nfs-server
Removed symlink /etc/systemd/system/multi-user.target.wants/nfs-server.service.

$ sudo systemctl stop nfs-secure-server nfs-server

Conditions

The ssh error message

debug1: Unspecified GSS failure.  Minor code may provide more information
Generic error (see e-text)

Means that the hostname (/bin/hostname) is not the same as the PTR hostname of the address by which you contacted the host. The three host indicators must be the same.

Let $HOSTNAME be the name by which you contacted the server
e.g. ssh $HOSTNAME

    1. The value given by /bin/hostname
    2. /usr/bin/host $HOSTNAME
    3. /usr/bin/host -t ptr $(address-of $HOSTNAME)

Example

      • The host capstone
      • must be DNS fqdn as capstone.department.example.com
      • must be hostname as capstone.department.example.com
      • must NOT be hostname 'capstone.example.com'
      • even if other relevant IPv6 addresses are bound to that interface

To wit:

[as wbaker:wbaker@capstone F21.Twenty_One]
$ hostname
capstone.department.example.com

$ host capstone
capstone.department.example.com has address 192.168.0.149
capstone.department.example.com has IPv6 address 2001:db8::223:26ff:fe6a:1451

$ host fdd3:34cd:f133:0:223:26ff:fe6a:1451
1.5.4.1.a.6.e.f.f.f.6.2.3.2.2.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa domain name pointer capstone.department.example.com.

Notes on the Operation of Kerberos: Mounting NFS (vers=4) with Kerberos (sec=krb5p) segfaults rpc.gssd

Following

tl;dr

On Fedora 21

  • yum update -y nfs-utils libtirpc gssproxy kernel kernel-PAE
  • reboot

Indications

The NFS client doesn’t “work” …  with options vers=4,sec=krb5. It’s clearly a client-side thing because the server “works” with other clients. Specifically, rpc.gssd segfaults.

On Fedora 21, after a fresh reboot…

Dec 27 17:12:33 client.example.com kernel: [  340.689185] fuse init (API version 7.23)
Dec 27 17:12:33 client.example.com kernel: fuse init (API version 7.23)
Dec 27 17:13:37 client.example.com kernel: [  404.817046] FS-Cache: Loaded
Dec 27 17:13:37 client.example.com kernel: FS-Cache: Loaded
Dec 27 17:13:37 client.example.com kernel: [  404.870948] FS-Cache: Netfs 'nfs' registered for caching
Dec 27 17:13:37 client.example.com kernel: FS-Cache: Netfs 'nfs' registered for caching
Dec 27 17:13:37 client.example.com kernel: [  404.929654] Key type dns_resolver registered
Dec 27 17:13:37 client.example.com kernel: Key type dns_resolver registered
Dec 27 17:13:37 client.example.com kernel: [  405.021340] NFS: Registering the id_resolver key type
Dec 27 17:13:37 client.example.com kernel: [  405.021370] Key type id_resolver registered
Dec 27 17:13:37 client.example.com kernel: [  405.021374] Key type id_legacy registered
Dec 27 17:13:37 client.example.com kernel: NFS: Registering the id_resolver key type
Dec 27 17:13:37 client.example.com kernel: Key type id_resolver registered
Dec 27 17:13:37 client.example.com kernel: Key type id_legacy registered
Dec 27 17:13:37 client.example.com gssproxy: gssproxy[2863]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may provide more information, No credentials cache found
Dec 27 17:13:37 client.example.com kernel: [ 405.167553] rpc.gssd[3833]: segfault at 2 ip b741464a sp bf7a5e00 error 4 in libc-2.20.so[b739c000+1c5000]
Dec 27 17:13:37 client.example.com kernel: rpc.gssd[3833]: segfault at 2 ip b741464a sp bf7a5e00 error 4 in libc-2.20.so[b739c000+1c5000]
Dec 27 17:13:37 client.example.com rpc.gssd[2873]: WARNING: forked child was killed with signal 11
Dec 27 17:13:40 client.example.com abrt-server: Deleting problem directory ccpp-2015-12-27-17:13:37-3833 (dup of ccpp-2015-12-23-18:52:49-2628)

With package constellation

$ rpm -q nfs-utils libtirpc gssproxy kernel kernel-PAE
nfs-utils-1.3.1-6.3.fc21.i686
libtirpc-0.2.5-2.0.fc21.i686
gssproxy-0.4.1-1.fc21.i686
kernel-3.19.4-200.fc21.i686
kernel-PAE-3.19.4-200.fc21.i686

Diagnosis

  • Versionitis between two libraries.
  • There is no workaround
    no configuration that avoids the problem.
  • You must upgrade… something.

Confounds

  • Fedora 21
    which has addressed many other issues in Kerberos operations.

    • some package constellations work flawlessly.
    • some package constellations are unworkable.
  • Fedora 18
    may have different issues (not characterized here)

Folklore

  • #755703 – libtirpc1 0.2.4-1 causes rpc.gssd to crash on nfs4 sec=krb5 mount; In Bugs of Debian; 2014-07-22.
    Mentions:

    • The exhibited error message text is similar:
      kernel: [ 285.086078] rpc.gssd[1611]: segfault at 6c ip 00007f24c8f9e72f sp 00007fff60b1df10 error 4 in libgssapi_krb5.so.2.2[7f24c8f8b000+45000
    • The library is libgssapi_krb5.so.2.2 (though our issue is consistently manifested as error 4 in libc-2.20.so)
    • Remediation
      downgrade to libtirpc-0.2.3-2
    • Repair
      • libtirpc1 0.2.4-2
      • rpcbind 0.2.1-5
      • nfs-common 1:1.2.8-7
      • nfs-kernel-server 1:1.2.8-7 (recommended, unvalidated)
  • #707960 - rpc.gssd segfaults when mounting a nfsv4 volume; In Bugs of Debian; 2013-05-12→2013-06-01.
    Mentions:

    • The exhibited error message text is similar
    • [2262594.734234] rpc.gssd[2729]: segfault at 1 ip 00000000f74714ba sp 00000000ff830170 error 4 in libgssglue.so.1.0.0[f746e000+8000]
    • The library is libgssglue.so.1.0.0 (though our issue is consistently manifested as error 4 in libc-2.20.so)
    • References
    • Remediation
      revert to nfs-utils-1.2.6-3 down from nfs-utils-1:1.2.8-2
    • Repair
      • nfs-utils-1.2.8-4
      • <quote>The configure option name is --with-gssglue, not --with-libgssglue.</quote>
  • 841788gssd crashes at rcnfs start with NFSv4 and Kerberos; In Bugzilla of Novell; 2013-09-23→2014-09-19.
    Mentions:

    • The exhibited error message text is similar:
      kernel: [348509.305940] rpc.gssd[23614]: segfault at 1 ip 00007f1a4dd69be5 sp 00007fff0f6160f0 error 4 in libgssglue.so.1.0.0[7f1a4dd66000+9000]
      rhea kernel: [ 4977.928970] NFS: nfs4_discover_server_trunking unhandled error -512. Exiting with error EIO
    • Explanation of Comment 18, Neil Brown, 2013-11-12:
      There are a collection of ‘gss’ symbols that a device in each of two libraries.
      libgssapi_krb5 and libgssglue.
      For example gss_acquire_cred().
      The cred data structure has different contents in these two libraries!!!! So it is very important that one or the other is used consistently.
      A field that is a pointer in one structure lines up with a counter with value ’1′ in the other structure. When confusion happens we try to dereference ’1′ and that crashes. <snip>this</snip> seems likely.
  • ANNOUNCE: nfs-utils-1.2.2 released.; Lukás Hejtmánek; In linux=nfsv4; 2010-03-09.
    Mentions

    • The conflict for the (function) name gss_acquire_cred
      • nm libgssapi_krb5.so | grep gss_acquire_cred
        000000000000b3a0 T gss_acquire_cred
      • nm libgssglue.so | grep gss_acquire
        00000000000004d0 T gss_acquire_cred
    • Something about autotools choosing the wrong one
    • Suggest rebuilding gssd of nfs-utils
      • use -lgssglue
      • remove -lgssapi_krb5

Actualities

1. Failing … Fedora 21

$ rpm -q nfs-utils libtirpc gssproxy kernel kernel-PAE
nfs-utils-1.3.1-6.3.fc21.i686
libtirpc-0.2.5-2.0.fc21.i686
gssproxy-0.4.1-1.fc21.i686
kernel-3.19.4-200.fc21.i686
kernel-PAE-3.19.4-200.fc21.i686

Running kernel 3.19
Reboot!

Fixed!
$ yum update -y nfs-utils libtirpc gssproxy kernel kernel-PAE
$ tail /var/log/messages
Dec 27 17:31:22 client.example.com yum[4071]: Installed: kernel-PAE-core-4.1.13-100.fc21.i686
Dec 27 17:31:30 client.example.com yum[4071]: Installed: kernel-core-4.1.13-100.fc21.i686
Dec 27 17:31:45 client.example.com yum[4071]: Installed: kernel-modules-4.1.13-100.fc21.i686
Dec 27 17:32:00 client.example.com yum[4071]: Installed: kernel-PAE-modules-4.1.13-100.fc21.i686
Dec 27 17:32:03 client.example.com yum[4071]: Updated: gssproxy-0.4.1-2.fc21.i686
Dec 27 17:32:04 client.example.com yum[4071]: Updated: libtirpc-0.2.5-2.1.fc21.i686
Dec 27 17:32:04 client.example.com yum[4071]: Updated: kernel-PAE-4.1.13-100.fc21.i686
Dec 27 17:32:04 client.example.com yum[4071]: Installed: kernel-4.1.13-100.fc21.i686
Dec 27 17:32:06 client.example.com yum[4071]: Updated: 1:nfs-utils-1.3.1-6.4.fc21.i686
$ rpm -q nfs-utils libtirpc gssproxy kernel kernel-PAE
nfs-utils-1.3.1-6.4.fc21.i686
libtirpc-0.2.5-2.1.fc21.i686
gssproxy-0.4.1-2.fc21.i686
kernel-3.19.4-200.fc21.i686
kernel-4.1.13-100.fc21.i686
kernel-PAE-4.1.13-100.fc21.i686
$ uname -a
Linux client1.example.com 4.1.13-100.fc21.i686+PAE #1 SMP Tue Nov 10 13:30:58 UTC 2015 i686 i686 i386 GNU/Linux

2. Working … Fedora 21

$ rpm -q nfs-utils libtirpc gssproxy kernel-PAE
nfs-utils-1.3.1-6.4.fc21.i686
libtirpc-0.2.5-2.0.fc21.i686
gssproxy-0.4.1-2.fc21.i686
kernel-PAE-3.19.4-200.fc21.i686

3. Working … Fedora 21

$ rpm -q nfs-utils libtirpc gssproxy kernel
nfs-utils-1.3.1-6.4.fc21.i686
libtirpc-0.2.5-2.1.fc21.i686
gssproxy-0.4.1-2.fc21.i686
kernel-4.0.7-200.fc21.i686
kernel-4.1.6-100.fc21.i686
kernel-4.1.13-100.fc21.i686

4. Failing … Fedora 21

$ rpm -q nfs-utils libtirpc gssproxy kernel kernel-PAE | sort
gssproxy-0.4.1-1.fc21.i686
kernel-PAE-3.19.4-200.fc21.i686
libtirpc-0.2.5-2.0.fc21.i686
nfs-utils-1.3.1-6.3.fc21.i686
package kernel is not installed
$ sudo yum update -y nfs-utils libtirpc gssproxy kernel-PAE

Running Kernel 3.19

$ uname -a
Linux client4.example.com 3.19.4-200.fc21.i686+PAE #1 SMP Mon Apr 13 22:00:24 UTC 2015 i686 i686 i386 GNU/Linux
reboot
$ sudo yum update -y nfs-utils libtirpc gssproxy kernel-PAE
Loaded plugins: langpacks
Resolving Dependencies
--> Running transaction check
---> Package gssproxy.i686 0:0.4.1-1.fc21 will be updated
---> Package gssproxy.i686 0:0.4.1-2.fc21 will be an update
---> Package kernel-PAE.i686 0:3.19.4-200.fc21 will be updated
---> Package kernel-PAE.i686 0:4.1.13-100.fc21 will be an update
--> Processing Dependency: kernel-PAE-modules-uname-r = 4.1.13-100.fc21.i686+PAE for package: kernel-PAE-4.1.13-100.fc21.i686
--> Processing Dependency: kernel-PAE-core-uname-r = 4.1.13-100.fc21.i686+PAE for package: kernel-PAE-4.1.13-100.fc21.i686
---> Package libtirpc.i686 0:0.2.5-2.0.fc21 will be updated
---> Package libtirpc.i686 0:0.2.5-2.1.fc21 will be an update
---> Package nfs-utils.i686 1:1.3.1-6.3.fc21 will be updated
---> Package nfs-utils.i686 1:1.3.1-6.4.fc21 will be an update
--> Running transaction check
---> Package kernel-PAE-core.i686 0:4.1.13-100.fc21 will be installed
---> Package kernel-PAE-modules.i686 0:4.1.13-100.fc21 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================
 Package                              Arch                   Version                            Repository                           Size
==========================================================================================================================================
Updating:
 gssproxy                             i686                   0.4.1-2.fc21                       collected-by-file                    89 k
 kernel-PAE                           i686                   4.1.13-100.fc21                    collected-by-file                    58 k
 libtirpc                             i686                   0.2.5-2.1.fc21                     collected-by-file                    91 k
 nfs-utils                            i686                   1:1.3.1-6.4.fc21                   collected-by-file                   375 k
Installing for dependencies:
 kernel-PAE-core                      i686                   4.1.13-100.fc21                    collected-by-file                    19 M
 kernel-PAE-modules                   i686                   4.1.13-100.fc21                    collected-by-file                    17 M

Transaction Summary
==========================================================================================================================================
Install             ( 2 Dependent packages)
Upgrade  4 Packages

Total download size: 36 M
Downloading packages:
------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                     4.2 MB/s |  36 MB  00:00:08     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction (shutdown inhibited)
  Installing : kernel-PAE-core-4.1.13-100.fc21.i686                                                                                  1/10 
  Installing : kernel-PAE-modules-4.1.13-100.fc21.i686                                                                               2/10 
  Updating   : gssproxy-0.4.1-2.fc21.i686                                                                                            3/10 
  Updating   : libtirpc-0.2.5-2.1.fc21.i686                                                                                          4/10 
  Updating   : kernel-PAE-4.1.13-100.fc21.i686                                                                                       5/10 
usermod: no changes
usermod: no changes
  Updating   : 1:nfs-utils-1.3.1-6.4.fc21.i686                                                                                       6/10 
  Cleanup    : 1:nfs-utils-1.3.1-6.3.fc21.i686                                                                                       7/10 
  Cleanup    : gssproxy-0.4.1-1.fc21.i686                                                                                            8/10 
  Cleanup    : libtirpc-0.2.5-2.0.fc21.i686                                                                                          9/10 
  Cleanup    : kernel-PAE-3.19.4-200.fc21.i686                                                                                      10/10 
  Verifying  : kernel-PAE-4.1.13-100.fc21.i686                                                                                       1/10 
  Verifying  : 1:nfs-utils-1.3.1-6.4.fc21.i686                                                                                       2/10 
  Verifying  : libtirpc-0.2.5-2.1.fc21.i686                                                                                          3/10 
  Verifying  : kernel-PAE-modules-4.1.13-100.fc21.i686                                                                               4/10 
  Verifying  : gssproxy-0.4.1-2.fc21.i686                                                                                            5/10 
  Verifying  : kernel-PAE-core-4.1.13-100.fc21.i686                                                                                  6/10 
  Verifying  : kernel-PAE-3.19.4-200.fc21.i686                                                                                       7/10 
  Verifying  : 1:nfs-utils-1.3.1-6.3.fc21.i686                                                                                       8/10 
  Verifying  : libtirpc-0.2.5-2.0.fc21.i686                                                                                          9/10 
  Verifying  : gssproxy-0.4.1-1.fc21.i686                                                                                           10/10 

Dependency Installed:
  kernel-PAE-core.i686 0:4.1.13-100.fc21                             kernel-PAE-modules.i686 0:4.1.13-100.fc21                            

Updated:
  gssproxy.i686 0:0.4.1-2.fc21   kernel-PAE.i686 0:4.1.13-100.fc21   libtirpc.i686 0:0.2.5-2.1.fc21   nfs-utils.i686 1:1.3.1-6.4.fc21  

Complete!

Question: But are xtables-addons and its kernel modules updated appropriately?
Answer: seems no>, the PAE variant is not installed.

$ rpm -q -a | grep kmod
kmod-19-1.fc21.i686
kmod-libs-19-1.fc21.i686
kmod-xtables-addons-2.9-1.fc21.2.i686
kmod-xtables-addons-3.19.4-200.fc21.i686+PAE-2.6-1.fc21.16.i686
kmod-xtables-addons-4.1.13-100.fc21.i686-2.9-1.fc21.2.i686
$ sudo yum install -y kmod-xtables-addons-4.1.13-100.fc21.i686+PAE-2.9-1.fc21.2.i686
Loaded plugins: langpacks
Resolving Dependencies
--> Running transaction check
---> Package kmod-xtables-addons-4.1.13-100.fc21.i686+PAE.i686 0:2.9-1.fc21.2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================
 Package                                         Arch    Version         Repository            Size
====================================================================================================
Installing:
 kmod-xtables-addons-4.1.13-100.fc21.i686+PAE    i686    2.9-1.fc21.2    collected-by-file    1.3 M

Transaction Summary
====================================================================================================
Install  1 Package

Total download size: 1.3 M
Installed size: 5.6 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction (shutdown inhibited)
  Installing : kmod-xtables-addons-4.1.13-100.fc21.i686+PAE-2.9-1.fc21.2.i686                   1/1 
  Verifying  : kmod-xtables-addons-4.1.13-100.fc21.i686+PAE-2.9-1.fc21.2.i686                   1/1 

Installed:
  kmod-xtables-addons-4.1.13-100.fc21.i686+PAE.i686 0:2.9-1.fc21.2                                  

Complete!

Reboot!

Fixed!

SOLVED: Diagnosis & remediation for gnome-terminal failure to start with Error constructing proxy for org.gnome.Terminal:/org/gnome/Terminal/Factory0: Error spawning command line ‘dbus-launch…’

Indications

** (gnome-terminal:28588): WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-jXttvb928F: Connection refused
Error constructing proxy for org.gnome.Terminal:/org/gnome/Terminal/Factory0: Error spawning command line 'dbus-launch --autolaunch=2f24fcda41304c13ac6826ea32e20941 --binary-syntax --close-stderr': Child process exited with code 1

Diagnosis

There are too many connections to he X11 server.

Remediation

Have fewer connections to your X11 server.

Background

The limit is only have ~255 (or maybe it’s 127, unclear). The error messaging when connections to the graphics server fail are … murky and misdirecting at best.

Folklore

Kerberos … something about delegated credentials or DNS-rDNS consistency

Delegated Credential

A delegated credential looks “simple” on the delegated host.

The (Forwarded) Delegated Credential

[as wbaker@remote.example.com]
$ klist
Ticket cache: KEYRING:persistent:500:500
Default principal: wbaker@EXAMPLE.COM

Valid starting       Expires              Service principal
12/20/2015 09:37:41  12/20/2015 16:36:10  krbtgt/EXAMPLE.COM@EXAMPLE.COM

The Origin Credential

Back on the origin host whence the credential came, there are many more principals

[as wbaker@origin.example.com]
$ klist
Ticket cache: DIR::/run/user/500/krb5cc/tkt
Default principal: wbaker@EXAMPLE.COM

Valid starting       Expires              Service principal
12/19/2015 16:36:10  12/20/2015 16:36:10  krbtgt/EXAMPLE.COM@EXAMPLE.COM
12/19/2015 16:36:14  12/20/2015 16:36:10  nfs/pickle.department.example.com@EXAMPLE.COM
12/19/2015 16:36:19  12/20/2015 16:36:10  nfs/steeple.department.example.com@EXAMPLE.COM
12/19/2015 16:37:30  12/20/2015 16:36:10  nfs/tagger.department.example.com@EXAMPLE.COM
12/19/2015 16:37:33  12/20/2015 16:36:10  nfs/badmouth.department.example.com@EXAMPLE.COM
12/19/2015 18:35:45  12/20/2015 16:36:10  host/munchy.department.example.com@EXAMPLE.COM
12/20/2015 05:42:22  12/20/2015 16:36:10  nfs/bopple.department.example.com@EXAMPLE.COM
12/20/2015 07:14:31  12/20/2015 16:36:10  host/flowerpot.department.example.com@EXAMPLE.COM
12/20/2015 07:57:49  12/20/2015 16:36:10  host/acorn.department.example.com@EXAMPLE.COM
12/20/2015 09:47:28  12/20/2015 16:36:10  host/welcome.department.example.com@EXAMPLE.COM
12/20/2015 10:19:03  12/20/2015 16:36:10  host/ravenswood.department.example.com@EXAMPLE.COM
12/20/2015 10:55:29  12/20/2015 16:36:10  host/capstone.department.example.com@EXAMPLE.COM

Updating Delegated Credentials

If GSSAPI Authentication is configured in OpenSSH client and server, then this “just works.” GSSAPI Authentication is on for server side in default configuration on Fedora. See /etc/ssh/sshd_config.

Forwarding Delegated Credentials
$ ssh -v -v capstone date 2>&1 | grep Delegating
debug1: Delegating credentials
debug1: Delegating credentials
~/.ssh/config
Host *
  GSSAPIAuthentication yes
  GSSAPIDelegateCredentials yes
  # trust DNS to canonicalize short names into fqdns (else delegation doesn't happen)
  GSSAPITrustDns yes

Server Configuration

The server is configured appriopriately by default; from openssh-5.8 of Fedora 16 onwards into openssh-7.1of the Fedora 23 era.
/etc/ssh/sshd_config
# GSSAPI options
#GSSAPIAuthentication no
GSSAPIAuthentication yes
#GSSAPICleanupCredentials yes
GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

Ignore these, leave them commented out, they pertain to the SSHv1 protocol:

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
#KerberosUseKuserok yes

Diagnosis & Observation

Correct Delegation

The -K flag is the same as -o GSSAPIAuthentication=yes.

$ ssh -K -v -v -v satellite
<snip/>
debug2: key: /home/wbaker/.ssh/id_ecdsa ((nil)),
debug2: key: /home/wbaker/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Delegating credentials
debug1: Delegating credentialsdebug1: Authentication succeeded (gssapi-with-mic).
Authenticated to satellite ([2001:db8::9876]:22).
debug1: channel 0: new [client-session]

Failure to Delegate

Without -Kor
$ ssh -v -v -v -o GSSAPIAuthentication=no satellite
<snip/>
debug2: key: /home/wbaker/.ssh/id_rsa ((nil)),
debug2: key: /home/wbaker/.ssh/id_dsa ((nil)),
debug2: key: /home/wbaker/.ssh/id_ecdsa ((nil)),
debug2: key: /home/wbaker/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
MISSING Delegating credentials message(s)
debug1: Authentication succeeded (gssapi-with-mic).
Authenticated to satellite ([2001:db8::9876]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 0: request auth-agent-req@openssh.com confirm 0
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0

The Error of Reverse DNS (rDNS) Inconsistency

$ ssh -v -v -v -o GSSAPIAuthentication=yes flowerpot
<snip/>
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Delegating credentials
debug1: Delegating credentials
debug1: Unspecified GSS failure. Minor code may provide more information
Generic error (see e-text)
<blank/>
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug1: Next authentication method: publickey
Diagnosis:

The DNS and reverse DNS is not synchronized e.g.
/bin/hostname gives flowerpot.example.com

flowerpot.department.example.com. AAAA 2001:db8::9876:1234
4.3.2.1.6.7.8.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2 PTR flowerpot2.department.example.com.
Note

After correcting the hostname, you do not need to restart sshd. The running sshd will continue to recover the system’s hostname dynamically.

Missing Host Principals

The Error of Reverse DNS (rDNS) Inconsistency

$ ssh -v -v -v -o GSSAPIAuthentication=yes flowerpot
<snip/>
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information 
Server host/flowerpot.department.example.com@EXAMPLE.COM not found in Kerberos database 

debug1: Unspecified GSS failure. Minor code may provide more information 
Server host/flowerpot.department.example.com@EXAMPLE.COM not found in Kerberos database 

debug1: Unspecified GSS failure. Minor code may provide more information 
<blank/>
<blank/>
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug1: Next authentication method: publickey
[as wbaker@flowerpot]
$ sudo klist -k /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COMThis example is missing host principals for flowerpot.

Remediation: add the host principals.
[as wbaker@flowerpot]
$ sudo kadmin -k wbaker/admin
kadmin: addprinc -randkey host/flowerpot.department.example.com
kadmin: ktadd host/flowerpot.department.example.com
Optionally, if necessary, but probably not necessary
kadmin: addprinc -randkey host/flowerpot.example.com
kadmin: ktadd host/flowerpot.example.com
kadmin: quit
[as wbaker@flowerpot]
$ sudo klist -k /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM

Recall that certain encryption algorithms have to be removed from the keytab on Fedora 18 & prior..

[as wbaker@flowerpot]
$ sudo systemctl restart nfs-secure nfs-idmap sshd

The expectation here is that this is not an NFS server and as such nfs-secure-server is not active, and does not need a restart. Failing to restart both nfs-secure and nfs-secure-server manifests in its own separate set of error (impossibly cryptic) error indications (as segfaults).

[as wbaker@origin]

$ ssh -v -v flowerpot date 2>&1 | grep Delegate
debug1: Delegating credentials
debug1: Delegating credentials

Restarting the GNOME keyring daemon after (it) crashes

tl;dr

setsid /usr/bin/gnome-keyring-daemon > ~/tmp/o.gnome-keyring-daemon.out 2>&1 &
$ cd /run/user/500
$ sudo mount --bind keyring-$NEW keyring-$OLD

Concept

  • Restart the daemon by hand
  • Modify /run/user/$UID to contain the appropriate directory
    • Both the old and the new keyring directories (names) need to “work”
    • Do not use a symlink to get the defunct name to point to the operational one.
      The keyring daemon will observe this after a time and will shut down on its own.
    • Use a bind mount to mount the directory of the operating daemon onto the name of the inoperative one

Background

  • For various reasons gnome-keyring daemon can die
  • You cannot (conveniently) establish a new GNOME session (logout, login again)
  • Many sessions depend upon access to the existing security apparatus
    • ssh-agent
    • gpg-agent
    • etc.

Error Messages

The keyring daemon dies spontaneously:

Dec 28 12:56:08 vast kernel: [2230783.652433] traps: gnome-keyring-d[2679] trap int3 ip:3fd704ed9d sp:7fff617dc420 error:0

If the keyring directory itself is not actually a directory or does not have the correct permissions, then the keyring daemon exits without further recourse:

** Message: Replacing daemon, using directory: /run/user/500/keyring-IIUwlw
** Message: The gnome-keyring control directory has invalid permissions. It must be only be accessible by its owner (ie: 0700): /run/user/500/keyring-IIUwlw

Configuration

  • Fedora 19
  • GNOME 3.8
  • gnome-keyring-3.8.2-1.fc19.x86_64

References

Folklore

  • How to start the keyring daemon after a gnome-shell crash?; Some droid using the self-asserted identity token l0b0; In Stack Exchange; 2012-02-25.
    tl;dr → refers to GNOME circa 2012 (what was that?), but the concepts are valid.
  • GNOME Keyring; In Some Document Landfill; undated.
    tl;dr → seems to have good & current information; sortof, is involved in the GNOME2 & Mate dissent etc.

Actualities

There are too many (user) processes running. In the gnome-keyring-daemon spontaneously exits.

-bash: fork: retry: No child processes
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: No child processes
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: No child processes
-bash: fork: retry: Resource temporarily unavailable
<snip/>
-bash: fork: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

You have to resolve this resource issue before moving in. Perhaps there are to many ssh clients reaching out into the network. Perhaps

$ setsid /usr/bin/gnome-keyring-daemon) > ~/tmp/o.gnome-keyring-daemon.out 2>&1 &
[1] 15202

Restarting the gnome-keyring-daemon results in names on the standard output, which would normally become environment variables for descendants.

GNOME_KEYRING_CONTROL=/run/user/500/keyring-J2oEff
SSH_AUTH_SOCK=/run/user/500/keyring-J2oEff/ssh
GPG_AGENT_INFO=/run/user/500/keyring-J2oEff/gpg:0:1
GNOME_KEYRING_PID=15208
$ cd /run/user/500
$ rm keyring-1Vplme
$ mkdir keyring-1Vplme
$ sudo mount --bind keyring-J2oEff keyring-1Vplme

Notes on the Operation of Kerberos: Ticket Delegation via OpenSSH GSSAPI (diagnostics & remediations)

Delegated Credential

A delegated credential looks “simple” on the delegated host.

The (Forwarded) Delegated Credential

[as wbaker@remote.example.com]
$ klist
Ticket cache: KEYRING:persistent:500:500
Default principal: wbaker@EXAMPLE.COM

Valid starting       Expires              Service principal
12/20/2015 09:37:41  12/20/2015 16:36:10  krbtgt/EXAMPLE.COM@EXAMPLE.COM

The Origin Credential

Back on the origin host whence the credential came, there are many more principals

[as wbaker@origin.example.com]
$ klist
Ticket cache: DIR::/run/user/500/krb5cc/tkt
Default principal: wbaker@EXAMPLE.COM

Valid starting       Expires              Service principal
12/19/2015 16:36:10  12/20/2015 16:36:10  krbtgt/EXAMPLE.COM@EXAMPLE.COM
12/19/2015 16:36:14  12/20/2015 16:36:10  nfs/pickle.department.example.com@EXAMPLE.COM
12/19/2015 16:36:19  12/20/2015 16:36:10  nfs/steeple.department.example.com@EXAMPLE.COM
12/19/2015 16:37:30  12/20/2015 16:36:10  nfs/tagger.department.example.com@EXAMPLE.COM
12/19/2015 16:37:33  12/20/2015 16:36:10  nfs/badmouth.department.example.com@EXAMPLE.COM
12/19/2015 18:35:45  12/20/2015 16:36:10  host/munchy.department.example.com@EXAMPLE.COM
12/20/2015 05:42:22  12/20/2015 16:36:10  nfs/bopple.department.example.com@EXAMPLE.COM
12/20/2015 07:14:31  12/20/2015 16:36:10  host/flowerpot.department.example.com@EXAMPLE.COM
12/20/2015 07:57:49  12/20/2015 16:36:10  host/acorn.department.example.com@EXAMPLE.COM
12/20/2015 09:47:28  12/20/2015 16:36:10  host/welcome.department.example.com@EXAMPLE.COM
12/20/2015 10:19:03  12/20/2015 16:36:10  host/ravenswood.department.example.com@EXAMPLE.COM
12/20/2015 10:55:29  12/20/2015 16:36:10  host/capstone.department.example.com@EXAMPLE.COM

Updating Delegated Credentials

If GSSAPI Authentication is configured in OpenSSH client and server, then this “just works.” GSSAPI Authentication is on for server side in default configuration on Fedora. See /etc/ssh/sshd_config.

Forwarding Delegated Credentials
$ ssh -v -v capstone date 2>&1 | grep Delegating
debug1: Delegating credentials
debug1: Delegating credentials
~/.ssh/config
Host *
  GSSAPIAuthentication yes
  GSSAPIDelegateCredentials yes
  # trust DNS to canonicalize short names into fqdns (else delegation doesn't happen)
  GSSAPITrustDns yes

Server Configuration

The server is configured appriopriately by default; from openssh-5.8 of Fedora 16 onwards into openssh-7.1 of the Fedora 23 era.

/etc/ssh/sshd_config
# GSSAPI options
#GSSAPIAuthentication no
GSSAPIAuthentication yes
#GSSAPICleanupCredentials yes
GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

Ignore these, leave them commented out, they pertain to the SSHv1 protocol:

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
#KerberosUseKuserok yes

Diagnosis & Observation

Correct Delegation

The -K flag is the same as -o GSSAPIAuthentication=yes.

$ ssh -K -v -v -v satellite
<snip/>
debug2: key: /home/wbaker/.ssh/id_ecdsa ((nil)),
debug2: key: /home/wbaker/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Delegating credentials
debug1: Delegating credentials
debug1: Authentication succeeded (gssapi-with-mic).
Authenticated to satellite ([2001:db8::9876]:22).
debug1: channel 0: new [client-session]

Failure to Delegate

Without -K or as -o GSSAPIAuthentication=no.

$ ssh -v -v -v -o GSSAPIAuthentication=no satellite
<snip/>
debug2: key: /home/wbaker/.ssh/id_rsa ((nil)),
debug2: key: /home/wbaker/.ssh/id_dsa ((nil)),
debug2: key: /home/wbaker/.ssh/id_ecdsa ((nil)),
debug2: key: /home/wbaker/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
<snip>MISSING Delegating credentials message(s)</snip>
debug1: Authentication succeeded (gssapi-with-mic).
Authenticated to satellite ([2001:db8::9876]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 0: request auth-agent-req@openssh.com confirm 0
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0

The Error of Reverse DNS (rDNS) Inconsistency

$ ssh -v -v -v -o GSSAPIAuthentication=yes flowerpot
<snip/>
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Delegating credentials
debug1: Delegating credentials
debug1: Unspecified GSS failure. Minor code may provide more information
Generic error (see e-text)
<blank/>
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug1: Next authentication method: publickey
Diagnosis:

The DNS and reverse DNS is not synchronized e.g.
/bin/hostname gives flowerpot.example.com

flowerpot.department.example.com. AAAA 2001:db8::9876:1234
4.3.2.1.6.7.8.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2 PTR flowerpot2.department.example.com.
Note

After correcting the hostname, you do not need to restart sshd. The running sshd will continue to recover the system’s hostname dynamically.

Missing Host Principals

The Error of Reverse DNS (rDNS) Inconsistency

$ ssh -v -v -v -o GSSAPIAuthentication=yes flowerpot
<snip/>
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information 
Server host/flowerpot.department.example.com@EXAMPLE.COM not found in Kerberos database 

debug1: Unspecified GSS failure. Minor code may provide more information 
Server host/flowerpot.department.example.com@EXAMPLE.COM not found in Kerberos database 

debug1: Unspecified GSS failure. Minor code may provide more information 
<blank/>
<blank/>
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug1: Next authentication method: publickey
[as wbaker@flowerpot]
$ sudo klist -k /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM

This example is missing host principals for flowerpot. Remediation: add the host principals.

[as wbaker@flowerpot]
$ sudo kadmin -k wbaker/admin
kadmin: addprinc -randkey host/flowerpot.department.example.com
kadmin: ktadd host/flowerpot.department.example.com
Optionally, if necessary, but probably not necessary
kadmin: addprinc -randkey host/flowerpot.example.com
kadmin: ktadd host/flowerpot.example.com
kadmin: quit
[as wbaker@flowerpot]
$ sudo klist -k /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
3 nfs/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.department.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM
2 host/flowerpot.example.com@EXAMPLE.COM

Recall that certain encryption algorithms have to be removed from the keytab on Fedora 18 & prior..

[as wbaker@flowerpot]
$ sudo systemctl restart nfs-secure nfs-idmap sshd

The expectation here is that this is not an NFS server and as such nfs-secure-server is not active, and does not need a restart. Failing to restart both nfs-secure and nfs-secure-server manifests in its own separate set of error (impossibly cryptic) error indications (as segfaults).

[as wbaker@origin]

$ ssh -v -v flowerpot date 2>&1 | grep Delegate
debug1: Delegating credentials
debug1: Delegating credentials

Notes on the Operation of Kerberos: Increasing Ticket Lifetime (beyond the default)

Following

Ticket Lifetime

The ticket lifetime is the minimum of the following values:

  • max_life in kdc.conf on the KDC.
  • ticket_lifetime in krb5.conf on the client.
  • maxlife for the user principal user/REALM@REALM.
  • maxlife for the service principal krbtgt/REALM@REALM.
  • requested lifetime in the ticket request.

Actualities

There is no indication whether a principal is renewable or not. You just have to “know.”

$ kadmin -p wbaker/admin
Couldn't open log file /var/log/kadmind.log: Permission denied
Authenticating as principal wbaker/admin with password.
Password for wbaker/admin@EXAMPLE.COM: 

kadmin:  getprinc wbaker
Principal: wbaker@EXAMPLE.COM
Expiration date: [never]
Last password change: Sun Nov 29 12:40:11 PST 2015
Password expiration date: [none]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 0 days 00:00:00
Last modified: Sun Nov 29 12:40:11 PST 2015 (wbaker/admin@EXAMPLE.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 8
Key: vno 1, aes256-cts-hmac-sha1-96, no salt
Key: vno 1, aes128-cts-hmac-sha1-96, no salt
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, arcfour-hmac, no salt
Key: vno 1, camellia256-cts-cmac, no salt
Key: vno 1, camellia128-cts-cmac, no salt
Key: vno 1, des-hmac-sha1, no salt
Key: vno 1, des-cbc-md5, no salt
MKey: vno 1
Attributes:
Policy: [none]

kadmin:  modprinc +allow_renewable wbaker

kadmin:  getprinc krbtgt/EXAMPLE.COM
Principal: krbtgt/EXAMPLE.COM@EXAMPLE.COM
Expiration date: [never]
Last password change: [never]
Password expiration date: [none]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 0 days 00:00:00
Last modified: Sat Nov 28 18:05:08 PST 2015 (db_creation@EXAMPLE.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 9
Key: vno 1, aes256-cts-hmac-sha1-96, no salt
Key: vno 1, aes128-cts-hmac-sha1-96, no salt
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, arcfour-hmac, no salt
Key: vno 1, camellia256-cts-cmac, no salt
Key: vno 1, camellia128-cts-cmac, no salt
Key: vno 1, des-hmac-sha1, no salt
Key: vno 1, des-cbc-md5, no salt
Key: vno 1, des-cbc-crc, no salt
MKey: vno 1
Attributes:
Policy: [none]

modprinc -maxlife 125hour -maxrenewlife 750hour  krbtgt/EXAMPLE.COM
modprinc +allow_renewable  krbtgt/EXAMPLE.COM

kadmin:  getprinc krbtgt/EMERSON.BAKER.ORG
Principal: krbtgt/EMERSON.BAKER.ORG@EMERSON.BAKER.ORG
Expiration date: [never]
Last password change: [never]
Password expiration date: [none]
Maximum ticket life: 5 days 05:00:00
Maximum renewable life: 31 days 06:00:00
Last modified: Sun Dec 20 19:17:29 PST 2015 (wbaker/admin@EMERSON.BAKER.ORG)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 9
Key: vno 1, aes256-cts-hmac-sha1-96, no salt
Key: vno 1, aes128-cts-hmac-sha1-96, no salt
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, arcfour-hmac, no salt
Key: vno 1, camellia256-cts-cmac, no salt
Key: vno 1, camellia128-cts-cmac, no salt
Key: vno 1, des-hmac-sha1, no salt
Key: vno 1, des-cbc-md5, no salt
Key: vno 1, des-cbc-crc, no salt
MKey: vno 1
Attributes:
Policy: [none]

Folklore

Continued, refined & summarized: Bringing up Kerberized NFSv4 on Fedora 16 through Fedora 23

Finally from

Recipe

On client.example.com

  • Establish /etc/krb5.confwith the appropriate default realm and realm-to-DNS associations
    • sudo mv /etc/krb5.conf /etc/krb5.conf.orig
    • sudo install -m 444 krb5.conf /etc/.
  • sudo kadmin -p wbaker/admin
    This will ask for the administrative principal’s password
    The sudo is required because you’ll be writing into /etc/krb5.keytab

    • Host Principals
      typically you’ll need multiple principals for all the aliases to the host

      • Create the new host principals for the client hostname, all possible names
        addprinc -randkey host/client.example.com@EXAMPLE.COM
        addprinc -randkey host/interface.client.example.com@EXAMPLE.COM
      • Add the new host principals to the system keytab on the host
        ktadd host/client.example.com
        ktadd host/interface.client.example.com
    • NFS Principal
      typically only one principal is needed

      • Create the new NFS principal for the client hostname
        addprinc -randkey nfs/interface.client.example.com@EXAMPLE.COM
      • Add the new NFS principal to the system keytab on the host
        ktadd nfs/interface.client.example.com
  • If you are on “older” Fedora, then see the subrecipe for deleting the keytab entries pertaining to  unuseable encryption algorithms See SOLVED
    • Fixup /etc/krb5.keytab, removing the unuseable algorithms
      • sudo ktutil
        • Read rkt /etc/krb5.keytab
        • Use list to show the available algorithms
        • Use list -e to exhibit the unsupported algorithms
          the command will abort/crash/stop-abruptly upon encountering an unsupported algorithm (number).  Delete that entry. Rinse.  Repeat.
        • Use delent the encryption unuseable algorithms
        • Write wkt /etc/krb5.NEWtab to a new file
          Be sure to write the updated keytab to a NEW file and move that into place; do not attempt to update the existing keytab (there is no update/overwrite operation in wkt).
      • sudo mv /etc/krb5.NEWtab /etc/krb5.keytab
    • Ensure that /etc/identd.confhas relevant entries:
      • Domain
        e.g. Domain = DEPARTMENT.EXAMPLE.COM
      • Local-Realms (may need to be a comma-list)
        e.g. Local-Realms = DEPARTMENT.EXAMPLE.COM,EXAMPLE.COM
  • Enable and start the Secure NFS client service:
    systemctl enable nfs-secure.service
    systemctl start nfs-secure.service

On server.example.com

  • Kerberos Configuration
    • Create /etc/krb5.conf, as above
  • Kerberos Principals
    • Create the host principal keys, as above.
    • If necessary, remove unsupported algorithms, as above.
  • Enable and start the Secure NFS client service
    systemctl enable nfs-secure.service nfs-secure-server.service
    systemctl start nfs-secure-server.service
  • Exporting volumes in /etc/exports
    Export the relevant volumes with appropriate security scheme

    • sec=sys (avoid)
    • sec=krb5
    • sec=krb5i
    • sec=krb5p (use)

Specimen krb5.conf

[logging]
 default = FILE:/var/log/krb5libs.log
 # kdc = FILE:/var/log/krb5kdc.log
 # admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = true
 dns_lookup_kdc = true
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_realm = EXAMPLE.COM
 default_ccache_name = KEYRING:persistent:%{uid}

[realms]
EXAMPLE.COM = {
  kdc = kerberos.example.com
  admin_server = kerberos.example.com
}

[domain_realm]
# .example.com = EXAMPLE.COM
# example.com = EXAMPLE.COM

References

[SOLVED] Continuing the Bringup of Kerberized NFSv4 on Fedora 16 through Fedora 23

Continued from bringing up Kerberized NFSv4 on Fedora 16 through Fedora 23
Onward as continued, refined & summarized.

tl;dr

  • To make Fedora 17 clients “work,” one must remove nfs host keys encrypted with
    • camellia128-cts-cmac
    • camellia256-cts-cmac
  • To make Fedora 18 servers “work,” one must remove nfs host keys encrypted with
    • camellia128-cts-cmac
    • camellia256-cts-cmac

Also

  • Ensure that /etc/imapd.conf has appropriate definitions for
    • Domain = the domain of the NFS clinet’s address
    • Local-Realms = the Domain and any sibling or ancestor settings

Configuration

Release Packages
Fedora 16 krb5-libs-1.9.4-3.fc16.i686
krb5-workstation-1.9.4-3.fc16.i686
nfs-utils-1.2.5-8.fc16.i686
Fedora 17 krb5-libs-1.10.2-6.fc17.i686
krb5-workstation-1.10.2-6.fc17.i686
nfs-utils-1.2.6-5.fc17.i686
Fedora 18 krb5-libs-1.10.3-17.fc18.i686
krb5-workstation-1.10.3-17.fc18.i686
nfs-utils-1.2.7-6.fc18.i686
Fedora 19 krb5-libs-1.11.3-24.fc19.x86_64
krb5-workstation-1.11.3-24.fc19.x86_64
nfs-utils-1.2.8-6.3.fc19.x86_64
Fedora 20 krb5-libs-1.11.5-19.fc20.x86_64
krb5-workstation-1.11.5-19.fc20.x86_64
nfs-utils-1.3.0-2.4.fc20.x86_64
Fedora 21 krb5-libs-1.12.2-15.fc21.x86_64
krb5-workstation-1.12.2-15.fc21.x86_64
nfs-utils-1.3.1-6.3.fc21.x86_64
Fedora 22 krb5-libs-1.13.1-3.fc22.x86_64
nfs-utils
(some version)
Fedora 23 krb5-libs-1.13.2-13.fc23.x86_64
krb5-workstation-1.13.2-13.fc23.x86_64
nfs-utils-1.3.3-1.rc1.fc23.x86_64

References

Configuration

allow_weak_crypto
defaults to false starting with krb5-1.8. When false, removes single-DES enctypes (and other weak enctypes) from permitted_enctypes, default_tkt_enctypes, and default_tgs_enctypes. Do not set this to true unless the use of weak enctypes is an acceptable risk for your environment and the weak enctypes are required for backward compatibility.
permitted_enctypes
controls the set of enctypes that a service will accept as session keys.
default_tkt_enctypes
controls the default set of enctypes that the Kerberos client library requests when making an AS-REQ. Do not set this unless required for specific backward compatibility purposes; stale values of this setting can prevent clients from taking advantage of new stronger enctypes when the libraries are upgraded.
default_tgs_enctypes
controls the default set of enctypes that the Kerberos client library requests when making a TGS-REQ. Do not set this unless required for specific backward compatibility purposes; stale values of this setting can prevent clients from taking advantage of new stronger enctypes when the libraries are upgraded.

The following per-realm setting in kdc.conf affects the generation of long-term keys.

supported_enctypes
controls the default set of enctype-salttype pairs that kadmind will use for generating long-term keys, either randomly or from passwords.
enctype weak? krb5
des-cbc-crc weak all
des-cbc-md4 weak all
des-cbc-md5 weak all
des3-cbc-sha1 notyet >=1.1
arcfour-hmac notyet >=1.3
arcfour-hmac-exp weak >=1.3
aes128-cts-hmac-sha1-96 notyet >=1.3
aes256-cts-hmac-sha1-96 notyet >=1.3
camellia128-cts-cmac notyet >=1.9
camellia256-cts-cmac notyet >=1.9