Skip to main content

Hartwork Blog

Free Software, Music, Chinese Chess

Replacing Ansible with salt-ssh for speed and for good

First, where am I coming from with Ansible?

There is this machine (or "box") that I used to manage using Ansible until recently. I wanted configuration management on that box so that if ever disk or VM or the entire hosting provider would go away, I would have a magic button to start a rebuild from nothing, grab a coffee, and have things work the same again. I wanted Ansible for that task because it's fairly easy and approachable, requires nothing but working SSH access from the host system, and is written in Python. Unlike Puppet, Chef, CFEngine and SaltStack — or so I thought.

Over time using Ansible I noticed that when I made changes to a playbook I was repeatedly facing the same challenge: Either I run the whole playbook and wait for many de-facto no-op tasks or I invest in annotation with tags, save some runtime but need to deal with the shortcomings that tags have in Ansible.

Tags in Ansible: What shortcomings?

Tags in Ansible have two problems that bug me. First, you'll need to manually propagate the same tag to all dependency tasks, especially those referenced in when-conditionals or else you'll run into undefined-variable issues because the task due to register that variable has not been executed. So that's something I would have to take good care of manually.

Secondly, tags and loops do not work well together in Ansible. What I would like to do is use the iteration item as a tag like this:

- hosts: all
  tasks:
  - name: Install distro packages
    package:
      name: "{{ item }}"
      state: present
    with_list:
      - git
      - htop
    tags:
      # NOTE Does not work!
      #      Gets us: ERROR! 'item' is undefined
      - "{{ item }}"

Unfortunately, this gets me ERROR! 'item' is undefined because tags do not support loops like that in Ansible.

I can address this problem by

  • a) having two verbatim copies of that list,
  • b) extracting and re-using a variable, or
  • c) making use of YAML references (as pointed out by André Thénot).

A version using YAML references could look like this:

- hosts: all
  tasks:
  - name: Install distro packages
    package:
      name: "{{ item }}"
      state: present
    with_list: &packages
      - git
      - htop
    tags: *packages

More importantly though, I'll also need to be okay with the whole loop being run if I ask for any of those tags now, which means waiting without any added value. I'm asking Ansible for ~20 packages to be installed so there is quite a bit of waste in runtime here.

I didn't feel like I wanted to deal with these shortcomings of tags most of the time so I instead started multi-tasking during that time, ran the whole playbook, and got back to it when there were results.

It was hard to accept one other thing though: When I ran the playbook two times in a row, for the second run Ansible would take about 4 minutes to do nothing but confirming that all the work was already done. Why? Would I have to accept that it was that slow?

Ansible is slow, how fast can I get it to be?

So I started looking for ways to improve Ansible speed, and SSH pipelining, disabling fact gathering, and Mitogen helped but wouldn't get runtime below 3 minutes, so I was not very happy. On a sidenote Mitogen doesn't support Ansible >=2.10 as of this writing so that boost in speed would come at the cost of being stuck with Ansible 2.9 in the past for longer, which is not ideal either.

So I accepted 3 minutes as the minimum runtime of that playbook. And started wondering about looking elsewhere.

Can Salt be used like Ansible?

Maybe Salt had some way without all those minions, masters, daemons, agents that seemed like a given to me when I last had a few bits to do with SaltStack at a previous job a few years ago. To me delight, I did find salt-ssh this time. salt-ssh was introduced with the release of Salt 0.17.0 on 2013-09-26, it's not actually new.

So I was trying to answer the question:

Can I port my existing Ansible playbook to salt-ssh, will it be fun and work well, and will it be faster than 3 minutes for when it doesn't actually need to do anything?

A summary of my existing Ansible playbook

For some context, what is that playbook of mine doing anyway?

For an almost complete high-level summary (if you're interested):

  • Configure sshd, an SSH pubkey, restart the service as needed
  • Install Docker from a dedicated repository, having it running and enabled, install docker-compose
  • Configured firewalld to be friends with Docker
  • Create a specific Docker network for a Caddy-based SSL reverse proxy to talk to website containers
  • Configures and activates dnf-automatic so that it updates packages by itself, restarts outdated services and reboots the VM when tracer detects need to
  • Adjusts systemd-resolved config to no longer expose LLMNR port 5355 to the world without need to
  • Closes port 9000 to the world previously exposed by the cockpit service
  • Makes sure that ${HOME}/.local/bin is in $PATH for all users
  • Downgrade cgroup to v1 for Docker by adjusting the kernel command line and re-creating the GRUB config for the change to have actual effect
  • Install some tools for manual inspections, e.g. htop, tmux and ncdu
  • Create some bare Git repositories to host off-GitHub website content
  • Clone some Git repositories containing docker-compose website projects and keep them up to date with upstream
  • Spin up multiple docker-compose based service and have them do rebuilds and restarts whenever their underlying Git clone changed
  • Set machine hostname

It's not very different from this playbook actually, just a bit a bigger.

First steps and pains with salt-ssh

I started making my way through the official Agentless Salt: Get Started Tutorial and got stuck quickly. I wanted execution as an unprivileged user but despite obeying the tutorial in detail I got errors about not being able to write to /var/cache/ — for good reasons — like these:

# salt-ssh '*' test.ping
[ERROR   ] Unable to render roster file: Traceback (most recent call last):
[..]
PermissionError: [Errno 13] Permission denied: '/var/cache/salt/master/roots/mtime_map'

And while the docs used absolute paths like /home/vagrant/salt-ssh/ everywhere, I wanted relative paths that would work with a Git repository checked out anywhere. Not to mention that log_file needs to be ssh_log_file in the tutorial.

So with all of that figured out after a while, this minimal setup satisfied all of that: execution as an unprivileged user, relative paths with the help of root_dir: ., significantly less noisy output through state_output_diff: True, and a place to start adding playbook-like things to. For a bird's eye view:

# tree
.
├── master
├── pillar
│   ├── data.sls
│   └── top.sls
├── roster
├── salt
│   └── setup.sls
└── Saltfile

In more detail, looking into these files:

File Saltfile:

salt-ssh:
  roster_file: ./roster
  config_dir: .
  ssh_log_file: ./log.txt

File master:

root_dir: .

cachedir: ./cachedir

file_roots:
  base:
    - ./salt

pillar_roots:
  base:
    - ./pillar

state_output_diff: True

File roster:

host1:
  host: host1.tld
  user: root

host2:
  host: host2.tld
  user: root

File pillar/top.sls:

base:
  '*':
    - data

With that as a base I can now port the playbook over in a new file salt/setup.sls.

For example, let's adjust the Open SSH server config to know my public key (that I'll store at salt/ssh/files/authorized-keys-root.txt), to disable password-based log-in (to protect against brute-force log-in attempts) and be sure that the server makes use of the adjusted configuration:

ssh-daemon:
  # Set SSH public keys for root
  ssh_auth.present:
    - user: root
    - source: salt://ssh/files/authorized-keys-root.txt

  # Disable password-based log-ins to SSH
  file.keyvalue:
    - name: /etc/ssh/sshd_config
    - key: PasswordAuthentication
    - value: "no"
    - separator: " "
    - uncomment: "#"
    - require:
      - ssh_auth: ssh-daemon

  # Restart sshd service to apply changes in configuration
  service.running:
    - name: sshd
    - reload: True
    - watch:
      - file: ssh-daemon

That state file was made with Fedora 32 in mind, by the way.

With that local setup we can now run commands like:

# salt-ssh '*' test.ping

# salt-ssh '*' grains.items

# salt-ssh '*' state.apply setup test=True

# salt-ssh '*' state.apply setup

It took me maybe one and a half day to port the whole playbook to salt-ssh and be confident with the result. What did it get me?

  • Significant reduction of runtime: Down from 3-4 minutes with Ansible to about 1 minute with salt-ssh
  • A high-level language leveraging YAML with idempotency in mind, just like with Ansible
  • Being able to stay agentless: No minions, no masters, just SSH
  • More flexibility (but also some duty) with regard to state dependencies and order of execution
  • Being able to use Jinja templating right in the playbook ("state file") unlike with Ansible
  • Experience with a new tool to add to my DevOps toolbox

I think it's fair to say I consider myself an salt-ssh convert by now: new setups won't be Ansible with me anymore, but salt-ssh.

What I do hope is that SaltStack gets better at fixing bugs. All the hiccups and limitations I ran into with version 3001.1 were related to features that I consider mainstream enough that I shouldn't even have seen them, given the size of the community.

Things I ran into include:

I hope those are not a sign of structural issues with SaltStack. VMWare bought SaltStack in September 2020 so I'm hoping that it turns out for the best. I'm happy to help out with pull requests once I'm convinced that I won't be wasting my time.

For more about using salt-ssh to replace Ansible, maybe Duncan Mac-Vicar P.'s article "Using Salt like Ansible" is of interest to you.

That's enough Salt for me today. Did I miss anything? Please let me know.

Best, Sebastian

Expat 2.2.10 has been released

libexpat is a fast streaming XML parser. Alongside libxml2, Expat is one of the most widely used software libre XML parsers written in C, precisely C99. It is cross-platform and licensed under the MIT license.

Expat 2.2.10 has been released earlier today. This release fixes undefined behavior from pointer arithmetic with NULL pointers, fixes reads to uninitialized variables uncovered by Cppcheck 2.0, adds documentation on exit codes to the man page of command-line tool xmlwf, brings a pile of improvements to Expat's CMake build system, and more. For details, please check out the changelog.

If you maintain Expat packaging or a bundled copy of Expat or a pinned version of Expat somewhere, please update to 2.2.10. Thank you!

Sebastian Pipping

uriparser 0.9.4 released

A few minutes ago uriparser 0.9.4 has been released. Version 0.9.4 comes with a number of minor improvements to the build system and four new functions — uriMakeOwner[AW] and uriMakeOwnerMm[AW] — that make UriUri[AW] instances independent of the original URI string.

For more details please check the change log.

Last but not least: If you maintain uriparser packaging or a bundled version of uriparser somewhere, please update to 0.9.4. Thank you!

My Approach to Code Review

Not every developer approaches code review the same way. For instance, some people consider good naming essential while others argue that they don't have time to discuss naming. My take on that is clear: Naming is difficult but important, shapes thinking and helps understanding of code. It's okay to not get names right the first time but if you don't have time to fix bad naming during review, your team is probably in trouble.

I have documented a team's approach to code review more than once so maybe it's time to share my approach to code review as a blog post.

Why do code review?

Why is time spent on code review time spent well? Code review helps…

  • to transfer knowledge (both ways),
  • to keep code quality up and to prevent introduction of bugs,
  • to reduce bias on where to take shortcuts and where to go extra miles, and
  • to practice communication.

How to write code that is about to be reviewed?

My rules of thumb for making changes and commits are:

  • Make things that explain themselves.
  • Comment what cannot explain itself.
  • Resist adding unused features.
  • Add tests for every bug you fix.
  • Add tests for every feature you add.
  • Keep code coverage up.
  • Keep technical debt down.
  • Make refactorings, bugfixes, and whitespace changes go into separate commits.
  • Clean-up while going, in healthy amounts.

How to do review?

During review I am looking for…

  • introduction of new bugs,
  • things that were forgotten but should not be missing,
  • things that are misleading or easy to misunderstand,
  • places that cause more cognitive load than necessary,
  • things that do not explain themselves and are not commented; applies on all levels, e.g. variable names, class hierarchy, etc.,
  • unclear and/or lacking commit messages,
  • commits that do more than one thing or more than they should; good Git history helps understanding the past so it better be clean,
  • inconsistency with the existing code base,
  • common struggles, e.g. proper use of super() in overridden methods.

I am not, or at least a lot less, reviewing for and caring about:

  • code formatting — we mostly have CI and pre-commit checks for that —, and
  • enforcing the same style on everyone.

Challenges during code review

Things can go sideways when…

  • code review starts to replace proper testing,
  • authors expect waving through rather than proper review,
  • review becomes personal,
  • attitude "we do not have time to discuss this" meets someone who cares,
  • code quality remains low across review iterations so that the reviewer could have done the whole thing him- or herself better in a fraction of the time and energy,
  • pull requests are too big to begin with,
  • unreasonable time constraints exist,
  • there are larger gaps in knowledge on the side of the reviewer,
  • non-obvious claims are made but are not backed up with sources,
  • reviewers request clean-ups that exceed the scope of the change and the energy of the author, or when
  • multiple reviewers disagree and insist on their point.

Do you want to discuss your code review situation with me? Please get in touch.

Best, Sebastian

What is wrong with Monotype's font licensing?

Monotype operates a number of sites to license their fonts:

These sites offer the same fonts and the same licensing terms.

Monotype's font licensing

Depending on your intended use, you need one or more of their license types for each font:

  • Desktop for print, logos, product packaging, offline use; gets you .ttf or .otf files.
  • Web for use on websites outside of ads; gets you .woff and .woff2 files.
  • DigitalAds for use in ads on the web; gets you .woff and .woff2 files.
  • App for embedding fonts into an application for display; gets you .ttf or .otf files.
  • ePub for embedding fonts into ebook-like products; gets you .ttf or .otf files.
  • Server for use on websites where users can choose from a variety of fonts to appear on products about to be produced; gets you .ttf or .otf files.

While complicated, I do get that Monotype asks for a different price when use differs by more than just a bit. But that's where understanding ends.

So what's wrong with Monotype's licensing?

1. I should not pay multiple times to gain access to different desktop font file formats.

Let's take font Futura. I can choose between these downloads:

  • a) OpenType .otf - Pro
  • b) OpenType .ttf - Pro
  • c) OpenType .otf - Std

If I need more than one — e.g. both .ttf and .otf — I pay twice or thrice, not just slightly more than for a single format. To me as a customer that makes as little sense as paying twice for a .bmp and a .png download of the same picture. If some formats require more time hinting than others, that's still no excuse to have me pay plain multiple times. And things get worse by the fact that it's anything but easy to figure out if you need OpenType CFF or OpenType TTF and if you need Std or Pro font files. I spent more than an hour researching on the format matter and probably still made the wrong choice. I shouldn't have too.

2. Tracking users and increasing page load time is not acceptable

When it comes to licensing fonts for use on the web, Monotype licenses a specific maximum number of page views — 250,000, 2,500,000, 25,000,000 or 75,000,000 — and forces the customer to have the web page to load a file from their server so they can count the number of page impressions. As a result:

  • They can track users across websites (whether they do that or not).
  • The load time of the web page increases for nothing of value to the customer or the customers of the customer.
  • It is mistrusting the customer.
  • It's a problem in light of the Slashdot effect: You hardly know upfront what content is going to go viral.

3. License type "Server" makes you pay by CPU core

The "Server" license type comes in flavors for 1 to 10 CPU cores. I don't see how the number of cores running my servers would be any of Monotype's business, virtual machine or not.

To summarize

Monotype's fonts are quite expensive already but the company chooses to use their market position to further extract money from their customers in ways that are not legitimate.

How to get Debian, WiFi/WLAN and AMD graphics running on Lenovo ThinkPad E495

I was helping someone with a non-trivial installation of Debian on a Lenovo ThinkPad E495 recently. The installation was anything but straightforward so I'll quickly document what it took to get WiFi and graphics working eventually. I have an idea by now why the E495 did not receive a hardware certification from Ubuntu like other ThinkPads did.

I believe that E495 and E595 are very close with regard to hardware outside of screen size, so there is some chance that most to all of this article applies to E595 as well.

Relevant E495 hardware components

If you're wondering if your E495 is close enough to keep reading this article, our model has:

  • AMD Ryzen 7 3700U CPU with Radeon Vega Mobile Gfx "Picasso"
    Advanced Micro Devices, Inc. [AMD/ATI] Picasso (rev c1)
    Kernel modules: amdgpu
  • Realtek RTL8111/8168/8411 Ethernet Controller
    Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 10)
    Kernel modules: r8169
  • Realtek RTL8822BE WiFi adapter
    Realtek Semiconductor Co., Ltd. RTL8822BE 802.11a/b/g/n/ac WiFi adapter
    Kernel modules: rtwpci
  • Realtek RTL8822B Bluetooth Radio

The details were extracted from output of lspci -k in the running system. The Bluetooth hardware not attached through PCI but internal USB.

What made the installation difficult?

  • Out of the box Debian buster is too old for recent hardware like this one (so we went for bullseye)
  • Even bullseye firmware packages are to outdated to contain all needed files
  • The bullseye installer is buggy (so it takes installing buster first and doing a dist-upgrade)
  • We saw nasty rendering artifacts in XFCE (before we adjusted xfwm4 configuration)
  • Some menu entries would not open (but have XFCE complain about lack of qdbus)
  • With secure boot enabled, WiFi will list available networks and pretend to be connected but it won't serve any actual traffic. Ouch.

The approach to installation that worked

Before we start, please make sure you have secure boot disabled in the BIOS or at least WiFi will not be fully functional.

So we start by installing Debian buster; it takes a working Ethernet connection and using a USB stick flashed with the Debian buster netinst ISO, e.g. debian-10.3.0-amd64-netinst.iso. During installation we do not select any desktop environment so we do not end up in a broken Xorg session but the terminal, after reboot.

After the installation, we mass-replace buster by bullseye in /etc/apt/sources.list, e.g. with sed -i 's,buster,bullseye,g /etc/apt/sources.list' and do a dist upgrade, e.g. by apt update ; apt dist-upgrade -V. One of the downloads of apt update will fail because bullseye is not released yet but that's okay. On a side note, we went for bullseye rather than testing here because bullseye will stop moving at some point in the near future while testing will remain rolling all the time.

What more do we need now?

  • some firmware files
  • the right Xorg driver package
  • a desktop environment
  • (a config change of xfwm4 in case of XFCE)

Let's start with firmware.

Firmware

If the related packages in Debian were complete and recent, you would need:

How do I know which firmware files are needed? The kernel knows which hardware needs which firmware file and the kernel log tells us which firmware files it loads or failed loading. In working E495 system it looks like this:

# sudo dmesg | fgrep direct-loading
[    1.768139] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_gpu_info.bin
[    1.768281] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_sdma.bin
[    1.770323] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_asd.bin
[    1.770364] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_ta.bin
[    1.770391] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_pfp.bin
[    1.770412] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_me.bin
[    1.770427] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_ce.bin
[    1.770456] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_rlc.bin
[    1.770582] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_mec.bin
[    1.770687] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_mec2.bin
[    1.772470] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/raven_dmcu.bin
[    1.772592] amdgpu 0000:05:00.0: firmware: direct-loading firmware amdgpu/picasso_vcn.bin
[  116.581265] platform regulatory.0: firmware: direct-loading firmware regulatory.db
[  116.581427] platform regulatory.0: firmware: direct-loading firmware regulatory.db.p7s
[  116.828475] rtw_pci 0000:04:00.0: firmware: direct-loading firmware rtw88/rtw8822b_fw.bin
[  116.991964] bluetooth hci0: firmware: direct-loading firmware rtl_bt/rtl8822b_fw.bin
[  116.992154] bluetooth hci0: firmware: direct-loading firmware rtl_bt/rtl8822b_config.bin
[  117.098348] r8169 0000:02:00.0: firmware: direct-loading firmware rtl_nic/rtl8168g-3.fw

The source of all of these files is the linux-firmware upstream Git repository.

Let's see how recent these very firmware files are in the upstream Git today (as of 2020-05-04):

# git clone -c fetch.fsckObjects=False \
    https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
# cd linux-firmware
# for i in $(ls -1 amdgpu/picasso_* \
                   rtw88/rtw8822b_fw.bin \
                   rtl_bt/rtl8822b_*.bin \
                   rtl_nic/rtl8168g-3.fw); do \
    echo "$(git log --format=%as -n1 -- "$i") $i" ; done | sort
2013-04-23 rtl_nic/rtl8168g-3.fw
2017-04-14 rtl_bt/rtl8822b_config.bin
2017-04-14 rtl_bt/rtl8822b_fw.bin
2018-10-04 rtw88/rtw8822b_fw.bin
2018-12-13 amdgpu/picasso_gpu_info.bin
2018-12-13 amdgpu/picasso_sdma.bin
2019-04-26 amdgpu/picasso_rlc_am4.bin
2020-01-06 amdgpu/picasso_ce.bin
2020-01-06 amdgpu/picasso_mec2.bin
2020-01-06 amdgpu/picasso_mec.bin
2020-01-06 amdgpu/picasso_pfp.bin
2020-01-06 amdgpu/picasso_rlc.bin
2020-01-06 amdgpu/picasso_vcn.bin
2020-04-16 amdgpu/picasso_asd.bin
2020-04-16 amdgpu/picasso_me.bin
2020-04-16 amdgpu/picasso_ta.bin

For the AMD firmware files, all but amdgpu/picasso_ta.bin are contained in package firmware-amd-graphics but Debian most recent 20190717-2 does not have up-to-date versions. For the Realtek firmware files, all but rtw88/rtw8822b_fw.bin are included with latest version 20190717-2 of firmware-realtek in Debian.

So we'll copy files from Git into /lib/firmware/ manually. Make sure to keep the directory structure, e.g. copy amdgpu/picasso_ta.bin to /lib/firmware/amdgpu/picasso_ta.bin.

To get the firmware loaded, you either need to reload selected kernel modules — at least amdgpu, r8169, rtwpci, rtw88 — or just do a quick reboot. I recommend going for the latter, for simplicity.

Xorg and desktop environment

Now that we have the firmware files around, let's install packages xserver-xorg-video-amdgpu and libgl1-mesa-dri so that Xorg has the right drivers. In case you don't get a picture but a hanging Xorg later, inspecting log file /var/log/Xorg.0.log may help.

Adding a desktop environment can easily by done by sudo tasksel: you'll get the same experience as inside the Debian installer.

In case you choose XFCE like us and experience weird graphic artifacts like these, our fix was running…

# xfconf-query -c xfwm4 -p /general/vblank_mode -t string \
    -s xpresent --create
# pkill xfwm4

as learned from this forum post by mbod. The added pkill xfwm4 kills the current instance of xfwm4 so that XFCE starts a new one that respects our change in configuration.

To teach XFCE how to open desktop menu items just install qdbus-qt5 and it should work.

Done.

If you cannot get it to work, feel free to get in touch.

Some closing notes:

  • Stable distributions and recent hardware are not a good match.

  • Command line tool efibootmgr can be of great help with debugging or changing EFI boot entries and boot order from within a running system.

  • Command like tool mokutil can be of help checking if secure boot is enabled, without a reboot.

Best, Sebastian

How to get Debian and WiFi/WLAN running on Lenovo ThinkPad L13 without an RJ45 port

I installed Debian on a Lenovo ThinkPad L13 a few days ago. The system is working by now but the road to "it works!" was a lot more exhausting than expected. I learned a few things along the way so it was not in vain. Maybe I can save you some time and/or headache with your own installation or help getting the WiFi to work. Let's go!

Why even get a Lenovo ThinkPad L13?

Simplified, I was looking for the cheapest ThinkPad with:

  • a less-than-14-inch matt full-HD IPS display

  • at least 16GB RAM, i5 CPU, 512 GB SSD, Intel GPU

  • support for a docking station

  • decent support for Linux

Canonical has certified Lenovo ThinkPad L13 to work well with Ubuntu 18.04 so I was in good faith that feeding Debian to it would work well. Not right away but eventually.

What's nice about the L13?

  • I like the keyboard

  • It has a shutter, a small hardware switch to close the eye of the camera

  • It looks rather elegant

What's weird about the L13?

  • It does not have an RJ45 LAN port. That's a bug in my world.

  • The battery cannot be removed. Bug!

  • It comes with two USB-A slots, only. I'm used to three or more and need them, so that's an inconvenience to me.

  • The BIOS is graphical be default (but it can be changed back to curses-like by setting Config > Setup UI: Simple Text) and supports mouse navigation. Except when the cursor gets stuck, happened multiple times; definitely a bug.

  • By default the function keys F1 to F12 act as if the Fn key was hold. That's not very friendly to users of Linux (think Ctrl+Alt+F1); the fix is setting Config > Keyboard/Mouse > F1-F12 as Primary Function: Enabled in the BIOS before you first need one of those keys.

What made the installation difficult?

  • The laptop itself lacks a RJ45 LAN port but the Debian installer wants wired Ethernet.

  • The WiFi chip does not work with the firmware and Linux kernel packaged in Debian buster; it needs more recent versions of these packages or a more recent release of Debian.

  • Unlike the installer of Debian buster, the installer of Debian bullseye is in alpha stage and fails with an error half-way in.

  • Upgrading a Debian buster with XFCE to bullseye produces file/package conflicts that need manual fixing and apt install -f to continue.

  • There is EFI involved, so e.g. all your live media need to be built for booting with EFI.

The approach to installation that worked for me

The lack of a RJ45 port can be approached in at least three different ways:

  • (a) Buy a docking station with RJ45 and wait until you have it at hand

  • (b) Buy a USB-to-RJ45 external network card adapter and wait until you have it at hand

  • (c) Boot a live system shipping working WiFi drivers (e.g. Xubuntu 19.10) and use QEMU to make the Debian installer believe that the WLAN below its feet is plain LAN to them.

I didn't have a USB-to-RJ45 network adapter and no docking station handy. So it was waiting or… adventure. I decided for (c): no waiting, QEMU, adventure.

Installing Debian to the host SSD from inside QEMU

Before I continue: If you're using this as a manual I'll assume/expect that you already:

  • have set the boot order in the BIOS to boot off an external medium so we don't end up in some half-installed Windows; if you're looking for + and - keys on an L13 with a German layout try ß and Shift+´

  • have Security > Secure Boot: Disabled in the BIOS

  • have set Config > Keyboard/Mouse > F1-F12 as Primary Function: Enabled in the BIOS or know a way around it from within a running Linux

  • have an external live medium ready that

    • supports EFI (if you made something custom yourself) and
    • comes with recent iwlwifi firmware, e.g. Xubuntu 19.10.

Excellent — let's continue.

So you first boot that live medium with working WiFi. To install to the host SSD from within QEMU it takes:

  • Passing the SSD to QEMU

  • KVM virtualization to be fast

  • Allocating enough RAM to QEMU (for both swap size math and speed of installation)

  • An OVMF EFI image so that the installer detects an EFI environment and runs grub-install for platform efi-amd64 rather than pc, e.g. from package ovmf on Ubuntu

  • A small Debian buster network installer ISO download, e.g. debian-10.3.0-amd64-netinst.iso

You then boot QEMU with KVM with the EFI image for a BIOS and two drives: the installer medium and the host SSD to install to. Note that the installer ISO cannot be passed as a CD-ROM drive or EFI won't boot off it. My command looked something like this:

sudo qemu-system-x86_64 \
        -enable-kvm -m 12G \
        -bios /usr/share/OVMF/OVMF_CODE.fd \
        -drive file=debian-10.3.0-amd64-netinst.iso,format=raw \
        -drive file=/dev/vnme0n1,format=raw

While in the Debian installer, when asked for the software to install, do not enable "desktop environment" and do not enable any specific desktop environment like XFCE, either. This allows upgrading to bullseye after the installation without running into conflicts during upgrade. The trick is to add the desktop environment after upgrading, not before. You can run tasksel from the installed system later and it will not only install say XFCE for you but also present the very ASCII dialog that you were presented during installation.

When the installation is done, the installer asks to reboot and remove media. Do not worry about media removal: EFI will boot of the disk because the boot order in the NVRAM of the VM (not the host) has that order set by the installer. If you run into a situation where you want to restart the installation from the beginning but cannot stop QEMU from preferring the SSD over the Debian installer live media, for a workaround consider wiping the EFI-partition of the SSD — it will be re-written during installation anyway.

Back on track: The Debian installer finished, it's the second boot in QEMU and we need to do some finial adjustments to make the installation ready to boot off actual hardware. So log in as root, upgrade to bullseye using apt, enable package distribution non-free next to main to be able to install firmware-iwlwifi and then install it, install the meta package of the desktop environment that you want, e.g. package xfce or by running tasksel as root. Also install some WiFi management tool, e.g. network-manager-gnome (for command nm-applet) if you don't want to transfer .deb files with a USB stick later.

Do a last boot in QEMU to ensure that everything but already WiFi works, do a clean shutdown of the VM, run sync on the host to flush unwritten data to disk and boot on real hardware after.

Done.

If you cannot get it to work, feel free to get in touch.

Some closing notes:

  • Stable distributions and recent hardware are not a good match. It make sense now but I didn't expect that big of a disconnect.

  • The Debian installer makes full disk encryption with sane defaults, separate /home and /var very convenient by now. Very nice. It's ahead of Ubuntu in that regard.

  • Command line tool efibootmgr can be of great help with debugging or changing EFI boot entries and boot order from within a running system.

Best, Sebastian

Get QEMU to boot EFI

It took me some Googling and experiments before I saw it work the first time: an EFI boot inside QEMU. I was blown away.

What is an EFI boot in QEMU good for? Two things:

  1. To predict about future bootability on actual EFI hardware.

  2. To make a Linux installer work with WLAN as if it's LAN.

My case was a bit of both combined, but that's a story for another post.

To have QEMU do an EFI boot, besides QEMU it takes:

  • OVMF installed (e.g. package sys-firmware/edk2-ovmf on Gentoo)

  • An EFI-only test image for proof (e.g. MemTest86 5.x or later)

Then:

wget https://www.memtest86.com/downloads/memtest86-usb.zip
unzip memtest86-usb.zip

sudo qemu-system-x86_64 \
    -enable-kvm -m 2G \
    -bios /usr/share/edk2-ovmf/OVMF_CODE.fd \
    -drive file=memtest86-usb.img,format=raw

Nice!

Arguments -enable-kvm -m 2G are optional and just make things run faster. The location of BIOS file OVMF_CODE.fd depends on your Linux distrubution:

  • Debian: /usr/share/OVMF/OVMF_CODE.fd (not /usr/share/qemu/OVMF.fd)

  • Gentoo: /usr/share/edk2-ovmf/OVMF_CODE.fd

  • Ubuntu: /usr/share/OVMF/OVMF_CODE.fd (not /usr/share/qemu/OVMF.fd)

Enough EFI for today.

Best, Sebastian

Helvetica Neue: Integrating with Linux

First, this post is not sponsored by anyone and has no links or ads that make me any money. Let's go!

I grew quite a bit of a sympathy for Helvetica fonts recently and ended up buying font Helvetica Neue a few days ago eventually.

While buying and then integrating the font into my Linux setup I learned a few things… that I would like to share with you.

Buying process + choices I had to make

Precisely I bought bundle "Neue Helvetica Pro Basic Family", 8 weights, each italic and not italic, desktop license 1-5 computers, Pro OpenType TTF, 20% discount from a promo code from signing up for their newsletter prior to buying, totaling at 141.85 Euro including VAT.

I picked Helvetica Neue over other Helveticas because it seemed like one of the more modern options fit for 2020. Also I had seen it work very well before (with content of The Futur in particular), and it was not an experiment (like Helvetica Now would have been), and it was more affordable than some of the other options.

I picked OpenType TTF over OpenType CFF for the desktop download because in my local prior experiments, TTF fonts rendering looked different and better. I should not pay twice to get both formats though, that's not cool.

Out of all the font-selling websites run by Monotype with close to identical offerings — fontshop.com, fonts.com, linotype.com, myfonts.com — I went for FontShop for buying because I liked the arty feeling about the site and because they allow experimenting with a font prior to buying in a more fun way than the others.

What that go me was 16 .ttf files:

  • HelveticaNeueLTPro-It.ttf
  • HelveticaNeueLTPro-UltLtIt.ttf
  • HelveticaNeueLTPro-UltLt.ttf
  • HelveticaNeueLTPro-Roman.ttf
  • HelveticaNeueLTPro-Lt.ttf
  • HelveticaNeueLTPro-Md.ttf
  • HelveticaNeueLTPro-Hv.ttf
  • HelveticaNeueLTPro-HvIt.ttf
  • HelveticaNeueLTPro-Blk.ttf
  • HelveticaNeueLTPro-BdIt.ttf
  • HelveticaNeueLTPro-Bd.ttf
  • HelveticaNeueLTPro-Th.ttf
  • HelveticaNeueLTPro-ThIt.ttf
  • HelveticaNeueLTPro-LtIt.ttf
  • HelveticaNeueLTPro-MdIt.ttf
  • HelveticaNeueLTPro-BlkIt.ttf

Installation

Installing them was easy:

  1. create a folder like ~/.local/share/fonts/Monotype_Imaging/TrueType/Helvetica_Neue_LT_Pro/,

  2. put the 16 .tff files in, and

  3. ran fc-cache to update the Fontconfig cache.

Integrating with Linux more

But I wanted a bit more. I often saw website refer to plain Helvetica — e.g. through CSS like font-family: [..],Helvetica,Arial,sans-serif,[..]; — and I wanted browser to use my Helvetica Neue for that. Also, I wanted change the default choice for sans-serif to Helvetica Neue, at least to see how it would feel for a few days. To summarize I wanted:

  • to map Helvetica to Helvetica Neue for all applications and

  • to set Helvetica Neue as the default sans-serif font for all applications.

Took me a few takes to get that right but looking back it's actually not that hard.

Tool fc-match helped me understand where I was at. When I started out, sans-serif mapped to Liberation Sans and Helvetica mapped to TeX Gyre Heros as can be seen here:

# fc-match Helvetica
texgyreheros-regular.otf: "TeX Gyre Heros" "Regular"

# fc-match sans-serif
LiberationSans-Regular.ttf: "Liberation Sans" "Regular"

Those mappings are configured in .conf files below /etc/fonts/ and also ~/.config/fontconfig/. So I learned from the existing .conf files I found, put two more files into ~/.config/fontconfig/, one per task, and ran fc-cache again. This is how I named my files:

  • ~/.config/fontconfig/conf.d/01-helvetica-neue-aliases.conf

  • ~/.config/fontconfig/conf.d/02-neue-helvetica-default-sans-serif.conf

Let's a have a closer look at their content.

To make Helvetica Neue win over TeX Gyre Heros I came up with this:

# cat ~/.config/fontconfig/conf.d/01-helvetica-neue-aliases.conf
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <description>
    Serve Helvetica Neue when asked for Helvetica
  </description>

  <!-- needs binding="same" to win over TeX Gyre Heros -->
  <alias binding="same">
    <family>Helvetica</family>
    <prefer>
      <family>Helvetica Neue LT Pro</family>
    </prefer>
  </alias>

</fontconfig>

Mapping sans-serif to Helvetica Neue was similar, slightly easier:

# cat ~/.config/fontconfig/conf.d/02-neue-helvetica-default-sans-serif.conf
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>

  <description>
    Make Helvetica Neue the default sans-serif
  </description>

  <alias>
    <family>sans-serif</family>
    <prefer>
      <family>Helvetica Neue LT Pro</family>
    </prefer>
  </alias>

</fontconfig>

Using fc-match, it was easy to verify those files worked:

# fc-match Helvetica
HelveticaNeueLTPro-Roman.ttf: "Helvetica Neue LT Pro" "Regular"

# fc-match sans-serif
HelveticaNeueLTPro-Roman.ttf: "Helvetica Neue LT Pro" "Regular"

During my experiments, I played with a downloaded copy of Web Font Specimen to make sure that both Firefox and Chromium were doing what I expected. (I had one of the <prefer> tags be an <accept> earlier and that made Firefox and Chromium behave differently — I still need to figure out why, but use of <prefer> fixed things for me.)

I also got curious in which order fc-cache process the .conf files, in particular how user config and system config would blend together. Why not just spy on fc-cache while it does the work… using strace. I'll trim this down a bit to the interesting part:

# strace -F -efile fc-cache |& fgrep openat \
      | grep -Eo '"[^"]+\.conf"' | sed 's,",,g' | nl
     1  /etc/fonts/fonts.conf
     2  /etc/fonts/conf.avail/10-hinting-slight.conf
     3  /etc/fonts/conf.avail/10-scale-bitmap-fonts.conf
     4  /etc/fonts/conf.avail/20-unhint-small-vera.conf
[..]
    10  /etc/fonts/conf.avail/50-user.conf
    11  /home/user/.config/fontconfig/conf.d/01-helvetica-neue-aliases.conf
    12  /home/user/.config/fontconfig/conf.d/02-neue-helvetica-default-sans-serif.conf
    13  /home/user/.config/fontconfig/fonts.conf
    14  /etc/fonts/conf.avail/51-local.conf
[..]
    51  /etc/fonts/conf.avail/70-yes-bitmaps.conf

So that's where fonts.conf and user config come in. It's controlled by 50-user.conf, cut down to the interesting bits:

# cat /etc/fonts/conf.avail/50-user.conf
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <description>Load per-user customization files</description>
  <!--
    Load per-user customization files where stored on XDG Base Directory
    specification compliant places. it should be usually:
    $HOME/.config/fontconfig/conf.d
    $HOME/.config/fontconfig/fonts.conf
  -->
  <include ignore_missing="yes" prefix="xdg">fontconfig/conf.d</include>
  <include ignore_missing="yes" prefix="xdg">fontconfig/fonts.conf</include>
</fontconfig>

My personal summary

  • I'm very happy to have desktop access to Helvetica Neue now, e.g. for use with future presentation slides.

  • I will probably revert back to Liberation Sans or DejaVu Sans for a sans-serif default though, will see.

  • Mapping or re-mapping fonts on Linux is not that hard.

  • Fontconfig configuration can be adjusted without root permissions by putting a small XML file into directory ~/.config/fontconfig/conf.d/.

  • Tools fc-cache and fc-match are the Fontconfig commands needed to get user fonts and configuration to work.

  • Font Manager is great for browsing and previewing installed fonts on a Linux System.

  • Googling for differences between OpenType TFF and OpenType CFF for quite a while did not help me making the choice easier. Converting one font TTF-to-CFF and another CFF-to-TFF using FontForge and comparing results did.

Enough fonts for me today.

Best, Sebastian