I replaced grub with systemd-boot
To be able to investigate and work on the the measured boot features I have switched from grub to systemd-boot (sd-boot).
This initial step is optional, but it is useful because this way /etc/kernel/cmdline will become the new place where the kernel command line can be configured:
. /etc/default/grub echo "root=/dev/mapper/root $GRUB_CMDLINE_LINUX $GRUB_CMDLINE_LINUX_DEFAULT" > /etc/kernel/cmdline
Do not forget to set the correct root file system there, because initramfs-tools does not support discovering it at boot time using the Discoverable Partitions Specification.
The installation has been automated since systemd version 252.6-1, so installing the package has the effect of installing sd-boot in the ESP, enabling it in the UEFI boot sequence and then creating boot loader entries for the kernels already installed on the system:
apt install systemd-boot
If needed, it could be manually installed again just by running bootctl install.
I like to show the boot menu by default, at least until I will be more familiar with sd-boot:
bootctl set-timeout 4
Since other UEFI binaries can be easily chainloaded, I am also going to keep around grub for a while, just to be sure:
cat <<END > /boot/efi/loader/entries/grub.conf title Grub linux /EFI/debian/grubx64.efi END
At this point sd-boot works, but I still had to enable secure boot. So far sd-boot has not been signed with a Debian key known to the shim bootloader, so I needed to create a Machine Owner Key (MOK), enroll it in UEFI and then use it to sign everything.
I dislike the complexity of mokutil and the other related programs, so after removing it and the boot shim I have decided to use sbctl instead. With it I easily created new keys, enrolled them in the EFI key store and then signed everything:
sbctl create-keys sbctl enroll-keys for file in /boot/efi/*/*/linux /boot/efi/EFI/*/*.efi; do sbctl sign -s $file done
Since there is no sbctl package yet I need to make sure that also the kernels installed in the future will be automatically signed, so I have created a trivial script in /etc/kernel/install.d/ which automatically runs sbctl sign -s or sbctl remove-file.
The Debian wiki SecureBoot page documents how do do this with mokutil and sbsigntool, but I think that sbctl is much friendlier.
Since I am not using the boot shim, I also had to set DisableShimForSecureBoot=true in /etc/fwupd/uefi_capsule.conf to make firmware updates work automatically.
As a bonus, I have also added to the boot menu the excellent Debian-based GRML live distribution. Since sd-boot is not capable of loopback-mounting CD-ROM images like grub, I first had to extract the kernel and initramfs and copy them to the ESP:
mount -o loop /boot/grml/grml64-full_2022.11.iso /mnt/ mkdir /boot/efi/grml/ cp /mnt/boot/grml64full/* /boot/efi/grml/ umount /mnt/ cat <<END > /boot/efi/loader/entries/grml.conf title GRML linux /grml/vmlinuz initrd /grml/initrd.img options boot=live bootid=grml64full202211 findiso=/grml/grml64-full_2022.11.iso live-media-path=/live/grml64-full net.ifnames=0 END
As expected, after a reboot bootctl reports the new security features:
System:
Firmware: UEFI 2.70 (Lenovo 0.4496)
Firmware Arch: x64
Secure Boot: enabled (user)
TPM2 Support: yes
Boot into FW: supported
Current Boot Loader:
Product: systemd-boot 252.5-2
Features: ✓ Boot counting
✓ Menu timeout control
✓ One-shot menu timeout control
✓ Default entry control
✓ One-shot entry control
✓ Support for XBOOTLDR partition
✓ Support for passing random seed to OS
✓ Load drop-in drivers
✓ Support Type #1 sort-key field
✓ Support @saved pseudo-entry
✓ Support Type #1 devicetree field
✓ Boot loader sets ESP information
ESP: /dev/disk/by-partuuid/1b767f8e-70fa-5a48-b444-cfe5c272d66e
File: └─/EFI/systemd/systemd-bootx64.efi
...
Relevant documentation:
Using a custom domain as the Mastodon identity
I just did again the usual web search, and I have verified that Mastodon still does not support managing multiple domains on the same instance, and that there is still no way to migrate an account to a different instance without losing all posts (and more?).
As much as I like the idea of a federated social network, open standards and so on, I do not think that it would be wise for me to spend time developing a social network identity on somebody else's instance which could disappear at any time.
I have managed my own email server since the '90s, but I do not feel that the system administration effort required to maintain a private Mastodon instance would be justified at this point: there is not even a Debian package! Mastodon either needs to become much simpler to maintain or become much more socially important, and so far it is neither. Also, it would be wasteful to use so many computing resources for a single-user instance.
While it is not ideal, for the time being I compromised by redirecting WebFinger requests for md@linux.it using this Apache configuration:
<Location /.well-known/host-meta>
Header set Access-Control-Allow-Origin: "*"
Header set Content-Type: "application/xrd+xml; charset=utf-8"
Header set Cache-Control: "max-age=86400"
</Location>
<Location /.well-known/webfinger>
Header set Access-Control-Allow-Origin: "*"
Header set Content-Type: "application/jrd+json; charset=utf-8"
Header set Cache-Control: "max-age=86400"
</Location>
# WebFinger (https://www.rfc-editor.org/rfc/rfc7033)
RewriteMap lc int:tolower
RewriteMap unescape int:unescape
RewriteCond %{REQUEST_URI} ^/\.well-known/webfinger$
RewriteCond ${lc:${unescape:%{QUERY_STRING}}} (?:^|&)resource=acct:([^&]+)@linux\.it(?:$|&)
RewriteRule .+ /home/soci/%1/public_html/webfinger.json [L,QSD]
# answer 404 to requests missing "acct:" or for domains != linux.it
RewriteCond %{REQUEST_URI} ^/\.well-known/webfinger$
RewriteCond ${unescape:%{QUERY_STRING}} (?:^|&)resource=
RewriteRule .+ - [L,R=404]
# answer 400 to requests without the resource parameter
RewriteCond %{REQUEST_URI} ^/\.well-known/webfinger$
RewriteRule .+ - [L,R=400]
This is the equivalent lighttpd configuration:
$HTTP["url"] =~ "^/\.well-known/" {
setenv.add-response-header += (
"Access-Control-Allow-Origin" => "*",
"Cache-Control" => "max-age=86400",
)
}
$HTTP["url"] == "/.well-known/host-meta" {
mimetype.assign = ("" => "application/xrd+xml; charset=utf-8")
}
$HTTP["url"] == "/.well-known/webfinger" {
mimetype.assign = ("" => "application/jrd+json; charset=utf-8")
$HTTP["querystring"] =~ "(?:^|&)resource=acct:([^&]+)@linux\.it(?:$|&)" {
alias.url = ( "" => "/home/soci/$1/public_html/webfinger.json" )
} else {
alias.url = ( "" => "/does-not-exist" )
}
}
Debian bookworm on a Lenovo T14s Gen3 AMD
I recently upgraded my laptop to a Lenovo T14s Gen3 AMD and I am happy to report that it works just fine with Debian/unstable using a 5.19 kernel.
The only issue is that some firmware files are still missing and I had to install them manually.
Updates are needed for the firmware-amd-graphics package (#1019847) for the Radeon 680M GPU (AMD Rembrandt) and for the firmware-atheros package (#1021157) for the Qualcomm NFA725A Wi-Fi card (which is actually reported as a NFA765).
s2idle (AKA "modern suspend") works too, and a ~10 seconds delay on resume has been removed by setting iommu=pt on the kernel command line.
For improved energy efficiency it is recommended to switch from the acpi_cpufreq CPU frequency scaling driver to amd_pstate. Please note that so far it is not loaded automatically.
As expected, fwupdmgr can update the system BIOS and the firmware of the NVMe device. Everybody should do it immediately, because there are major suspend bugs with BIOS releases earlier than 1.25.
Run an Ansible playbook in a remote chroot
Running a playbook in a remote chroot or container is not supported by Ansible, but I have invented a good workaround to do it anyway.
The first step is to install Mitogen for Ansible (ansible-mitogen in Debian) and then configure ansible.cfg to use it:
[defaults] strategy = mitogen_linear
But everybody should use Mitogen anyway, because it makes Ansible much faster.
The trick to have Ansible operate in a chroot is to make it call a wrapper script instead of Python. The wrapper can be created manually or by another playbook, e.g.:
vars:
- fsroot: /mnt
tasks:
- name: Create the chroot wrapper
copy:
dest: "/usr/local/sbin/chroot_{{inventory_hostname_short}}"
mode: 0755
content: |
#!/bin/sh -e
exec chroot {{fsroot}} /usr/bin/python3 "$@"
- name: Continue with stage 2 inside the chroot
debug:
msg:
- "Please run:"
- "ansible-playbook therealplaybook.yaml -l {{inventory_hostname}} -e ansible_python_interpreter=/usr/local/sbin/chroot_{{inventory_hostname_short}}"
This works thanks to Mitogen, which funnels all remote tasks inside that single call to Python. It would not work with standard Ansible, because it copies files to the remote system with SFTP and would do it outside of the chroot.
The same principle can also be applied to containers by changing the wrapper script, e.g:
#!/bin/sh -e
exec systemd-run --quiet --pipe --machine={{container_name}} --service-type=exec /usr/bin/python3 "$@"
After the wrapper will have been installed then you can run the real playbook by setting the ansible_python_interpreter variable, either on the command line, in the inventory or anywhere else that variables can be defined:
ansible-playbook therealplaybook.yaml -l {{inventory_hostname}} -e ansible_python_interpreter=/usr/local/sbin/chroot_{{inventory_hostname_short}}
My resignation from freenode
As it is now known, the freenode IRC network has been taken over by a Trumpian wannabe korean royalty bitcoins millionaire. To make a long story short, the former freenode head of staff secretly "sold" the network to this person even if it was not hers to sell, and our lawyers have advised us that there is not much that we can do about it without some of us risking financial ruin. Fuck you Christel, lilo's life work did not deserve this.
What you knew as freenode after 12:00 UTC of May 19 will be managed by different people.
As I have no desire to volunteer under the new regime, this marks the end of my involvement with freenode. It had started in 1999 when I encouraged the good parts of #linux-it to leave ircnet, and soon after I became senior staff. Even if I have not been very active recently, at this point I was the longest-serving freenode staff member and now I expect that I will hold this record forever.
The people that I have met on IRC, on freenode and other networks, have been and still are a very important part of my life, second only to the ones that I have known thanks to Usenet. I am not fine, but I know that the communities which I have been a part of are not defined by a domain name and will regroup somewhere else.
The current freenode staff members have resigned with me, these are some of their farewell messages:
Together we have created Libera.Chat, a new IRC network based on the same principles of the old freenode.
RPKI validation with FORT Validator
This article documents how to install FORT Validator (an RPKI relying party software which also implements the RPKI to Router protocol in a single daemon) on Debian 10 to provide RPKI validation to routers. If you are using testing or unstable then you can just skip the part about apt pinnings.
The packages in bullseye (Debian testing) can be installed as is on Debian stable with no need to rebuild them, by configuring an appropriate pinning for apt:
cat <<END > /etc/apt/sources.list.d/bullseye.list deb http://deb.debian.org/debian/ bullseye main END cat <<END > /etc/apt/preferences.d/pin-rpki # by default do not install anything from bullseye Package: * Pin: release bullseye Pin-Priority: 100 Package: fort-validator rpki-trust-anchors Pin: release bullseye Pin-Priority: 990 END apt update
Before starting, make sure that curl (or wget) and the web PKI certificates are installed:
apt install curl ca-certificates
If you already know about the legal issues related to the ARIN TAL then you may instruct the package to automatically install it. If you skip this step then you will be asked at installation time about it, either way is fine.
echo 'rpki-trust-anchors rpki-trust-anchors/get_arin_tal boolean true' \ | debconf-set-selections
Install the package as usual:
apt install fort-validator
You may also install rpki-client and gortr on Debian 10, or maybe cfrpki and gortr. I have also tried packaging Routinator 3000 for Debian, but this effort is currently on hold because the Rust ecosystem is broken and hostile to the good packaging practices of Linux distributions.
RPKI validation with OpenBSD's rpki-client and Cloudflare's gortr
This article documents how to install rpki-client (an RPKI relying party software, the actual validator) and gortr (which implements the RPKI to Router protocol) on Debian 10 to provide RPKI validation to routers. If you are using testing or unstable then you can just skip the part about apt pinnings.
The packages in bullseye (Debian testing) can be installed as is on Debian stable with no need to rebuild them, by configuring an appropriate pinning for apt:
cat <<END > /etc/apt/sources.list.d/bullseye.list deb http://deb.debian.org/debian/ bullseye main END cat <<END > /etc/apt/preferences.d/pin-rpki # by default do not install anything from bullseye Package: * Pin: release bullseye Pin-Priority: 100 Package: gortr rpki-client rpki-trust-anchors Pin: release bullseye Pin-Priority: 990 END apt update
Before starting, make sure that curl (or wget) and the web PKI certificates are installed:
apt install curl ca-certificates
If you already know about the legal issues related to the ARIN TAL then you may instruct the package to automatically install it. If you skip this step then you will be asked at installation time about it, either way is fine.
echo 'rpki-trust-anchors rpki-trust-anchors/get_arin_tal boolean true' \ | debconf-set-selections
Install the packages as usual:
apt install rpki-client gortr
And then configure rpki-client to generate its output in the the JSON format needed by gortr:
echo 'OPTIONS=-j' > /etc/default/rpki-client
You may manually start the service unit to immediately generate the data instead of waiting for the next timer run:
systemctl start rpki-client &
gortr too needs to be configured to use the JSON data generated by rpki-client:
echo 'GORTR_ARGS=-bind :323 -verify=false -checktime=false -cache /var/lib/rpki-client/json' > /etc/default/gortr
And then it needs to be restarted to use the new configuration:
systemctl restart gortr
You may also install FORT Validator on Debian 10, or maybe cfrpki with gortr. I have also tried packaging Routinator 3000 for Debian, but this effort is currently on hold because the Rust ecosystem is broken and hostile to the packaging practices of Linux distributions.
Debian support for libxcrypt
glibc 2.29-7 and libxcrypt 1:4.4.10-10 today entered Debian testing: crypt(3) and the other related library functions in libcrypt from now on will be provided by libxcrypt instead of glibc.
After 18 months of packaging work, Debian finally supports modern password hashing methods like yescrypt: the details about them are documented in crypt(5).
For the time being there is still no support for libxcrypt in our release of PAM, but hopefully the Debian maintainer will update the package soon and this will allow using yescrypt by default.
If you want to test now the new algorithms then you can generate a password using my mkpasswd program and copy it to /etc/shadow:
# echo "marco:$(echo 12345 | mkpasswd --stdin)" | chpasswd --encrypted
Per-process netfilter rules
This article documents how the traffic of specific Linux processes can be subjected to a custom firewall or routing configuration, thanks to the magic of cgroups. We will use the Network classifier cgroup, which allows tagging the packets sent by specific processes.
To create the cgroup which will be used to identify the processes I added something like this to /etc/rc.local:
mkdir /sys/fs/cgroup/net_cls/unlocator /bin/echo 42 > /sys/fs/cgroup/net_cls/unlocator/net_cls.classid chown md: /sys/fs/cgroup/net_cls/unlocator/tasks
The tasks file, which controls the membership of processes in a cgroup, is made writeable by my user: this way I can add new processes without becoming root. 42 is the arbitrary class identifier that the kernel will associate with the packets generated by the member processes.
A command like systemd-cgls /sys/fs/cgroup/net_cls/ can be used to explore which processes are in which cgroup.
I use a simple shell wrapper to start a shell or a new program as members of this cgroup:
#!/bin/sh -e
CGROUP_NAME=unlocator
if [ ! -d /sys/fs/cgroup/net_cls/$CGROUP_NAME/ ]; then
echo "The $CGROUP_NAME net_cls cgroup does not exist!" >&2
exit 1
fi
/bin/echo $$ > /sys/fs/cgroup/net_cls/$CGROUP_NAME/tasks
if [ $# = 0 ]; then
exec ${SHELL:-/bin/sh}
fi
exec "$@"
My first goal is to use a special name server for the DNS queries of some processes, thanks to a second dnsmasq process which acts as a caching forwarder.
/etc/dnsmasq2.conf:
port=5354 listen-address=127.0.0.1 bind-interfaces no-dhcp-interface=* no-hosts no-resolv server=185.37.37.37 server=185.37.37.185
/etc/systemd/system/dnsmasq2.service:
[Unit] Description=dnsmasq - Second instance Requires=network.target [Service] ExecStartPre=/usr/sbin/dnsmasq --test ExecStart=/usr/sbin/dnsmasq --keep-in-foreground --conf-file=/etc/dnsmasq2.conf ExecReload=/bin/kill -HUP $MAINPID PIDFile=/run/dnsmasq/dnsmasq.pid [Install] WantedBy=multi-user.target
Do not forget to enable the new service:
systemctl enable dnsmasq2 systemctl start dnsmasq2
Since the cgroup match extension is not yet available in a released version of iptables, you will first need to build and install it manually:
git clone git://git.netfilter.org/iptables.git cd iptables ./autogen.sh ./configure make -k sudo cp extensions/libxt_cgroup.so /lib/xtables/ sudo chmod -x /lib/xtables/libxt_cgroup.so
The netfilter configuration required is very simple: all DNS traffic from the marked processes is redirected to the port of the local dnsmasq2:
iptables -t nat -A OUTPUT -m cgroup --cgroup 42 -p udp --dport 53 -j REDIRECT --to-ports 5354 iptables -t nat -A OUTPUT -m cgroup --cgroup 42 -p tcp --dport 53 -j REDIRECT --to-ports 5354
For related reasons, I also need to disable IPv6 for these processes:
ip6tables -A OUTPUT -m cgroup --cgroup 42 -j REJECT
I use a different cgroup to force some programs to use my office VPN by first setting a netfilter packet mark on their traffic:
iptables -t mangle -A OUTPUT -m cgroup --cgroup 43 -j MARK --set-mark 43
The packet mark is then used to policy-route this traffic using a dedicate VRF, i.e. routing table 43:
ip rule add fwmark 43 table 43
This VPN VRF just contains a default route for the VPN interface:
ip route add default dev tun0 table 43
Depending on your local configuration it may be a good idea to also add to the VPN VRF the routes of your local interfaces:
ip route show scope link proto kernel \ | xargs -I ROUTE ip route add ROUTE table 43
Since the source address selection happens before the traffic is diverted to the VPN, we also need to source-NAT to the VPN address the marked packets:
iptables -t nat -A POSTROUTING -m mark --mark 43 --out-interface tun0 -j MASQUERADE
Hacking Team and a case of BGP hijacking
After just a few hours the Hacking Team emails archive has already provided many tasty leaks. I want to focus on a routing security issue since this is my main research activity for this year.
Short summary: if these emails are true, and so far nobody has found any credible reason to believe that they are not, then some major italian ISPs hijacked the IP addresses of a foreign ISP on request of the section of the Carabinieri which investigates terrorism and organized crime.
The goal was to recover access to some copies of the Hacking Team malware which were controlled by "anonymizer" VPSes hosted on the hijacked network and that were abruptly disabled by their provider.
Thanks to the great RIPEstat service I have been able to verify that indeed the network 46.166.163.0/24 was announced by (elided) (a large italian hosting company) in 2013, from august 15 to 22.
Then I downloaded from the RIPE RIS archive a BGP table dump for august 20 2013 and processed it with my zebra-dump-parser software to extract the relevant routes. It shows that (elided) did not just advertise the hijacked network to the two italian ISPs mentioned in the emails, but apparently to all their other peers since the announce was also accepted by Hurricane Electric at MIX-IT. This means that the hijacking was not limited to a couple of local networks but involved many others all over the world.
As an italian network operator I am seriously concerned that some of my BGP peers appear to be involved in what would usually be considered a criminal activity.
As all operators know, there is still too much mutual trust involved in global BGP routing, and in some cases it is misplaced: we need better best practices, tools and protocols to make this kind of things impossible in the future.