rtcqs v0.6.1 released

A new version of rtcqs, a Linux audio performance analyzer, is now available. Most notable changes include:

  • Fixed inconsistent use of single and double quotes
  • Replaced audio group check with a group agnostic check (fixes #4)
  • Governor check can now deal with systems that have SMT disabled
  • Tickless check now deals with all CONFIG_NO_HZ* variants and with nohz being set on the kernel command line (fixes #8)
  • File systems check has been expanded
  • IRQ check now loops through /sys/kernel/irq instead of parsing /proc/interrupts
  • rtprio check now checks if a SCHED_FIFO priority can be set instead of a SCHED_RR priority
  • Improved preempt RT check, check if “preempt=full” is part of the kernel command line (fixes #7)
  • Refactoring, created separate classes for main app, resources and GUI
  • Moved all packaging directives into pyprojects.toml

While working on this release I found out PySimpleGUI is not open source anymore so rtcqs’ GUI has become a bit of a moving target. I’m looking at alternatives like pygubu or even popsicle but that will be something for in the long run. In the short run there are more improvements in the pipeline. The swappiness check needs some attention and same goes for the IRQ check. I’ve been working on a different project to automate prioritizing IRQs and I’m planning to to reuse some parts of that project for the IRQ check in rtcqs. The idea is to have rtcqs not only list the status of all audio related IRQs but also any audio devices attached to those IRQs.

rtcqs is available on Codeberg, PyPI and is also included in the AUR.

rtcqs v0.6.1 released

New notebook – The verdict

Well, Phoronix did a review of a similar machine and apparently it’s far from being a slouch and also has the fastest integrated GPU currently available. More on that GPU later. So no regret when it comes to those performance benchmarks. Actually no regret at all, so far the notebook performs really well.

There are some more things worth mentioning that add up to the positive verdict besides all the pros I already mentioned in my earlier posts. There is the battery life which is still pretty good given the performant and power greedy CPU. It can run for hours when idling. When running Ardour it’s done in about two hours though but then I work with all the sluices wide open. But it charges pretty fast. Another thing that struck me is that the notebook is so much quieter than the old one. And the keyboard is just really nice now that I got a bit more used to it. And I managed to map the last media key, the stop one, to something useful with my old friend xdotool. Mapped this media key to the “stop/cancel” keycode using udev and added a keyboard shortcut in XFCE that gets triggered by this keycode and that executes a small script that looks like this:

#!/bin/bash

xdotool search --classname ardour_ardour key --delay 100 h space h

Now if Ardour is running and I press the Stop media key, xdotool sends the key sequence “h space h” to Ardour with a delay of 100ms between the key strokes. “h” sets the playhead to start, “space” stops the transport and another “h” to be sure the playhead is at the start position. Only thing that I’d like to add is some kind of detection if transport is running or not.

Also managed to achieve an even lower latency with my USB interface by adding the option implicit_fb=1 to the snd-usb-audio kernel module. This not only gets rid of the kernel ring buffer getting flooded with warnings but it also results in clean audio at 32*3/48, so 2ms of systemic latency. So it’s on par with my old notebook, albeit with some more headroom. Lower doesn’t seem to be possible, it results in slowed down, distorted audio.

So would I advise everyone doing Linux audio to get this notebook or a similar spec’d one? Well, there’s this GPU that still seems to be a bit too new, too shiny and too fast for the kernel I’m currently running (6.7.2) so I’m getting reliable crashes with software like OwlPlug and occasional crashes when connected via HDMI to a second screen. But it’s tolerable and it will probably get sorted out sooner or later. Other than that, this thing flies and hopefully I can do another decade with this machine.

Edit: worked around the GPU crashing by copying /usr/share/X11/xorg.conf.d/10-amdgpu.conf to /etc/X11/xorg.conf.d/10-amdgpu.conf and disabling hardware acceleration by adding the line Option "Accel" "off" to it. Content looks like this:

Section "OutputClass"
        Identifier "AMDgpu"
        MatchDriver "amdgpu"
        Driver "amdgpu"
        Option "HotplugDriver" "amdgpu"
        Option "Accel" "off"
EndSection

After restarting Xorg I can now open OwlPlug without issues.

New notebook – The verdict

New notebook – Doing real-time audio

Configuring this new machine took some effort but it can now run reliably with a few milliseconds of latency. First thing I did was installing a liquorix kernel as I have a good experience with those. Added the threadirqs kernel option to /etc/default/grub and updated the Grub configuration with sudo update-grub. After a reboot I was greeted with threaded IRQ’s.

Next step was to prioritize the desired IRQ’s, highest prio for the USB bus to which my audio interface is connected and also a high prio for the onboard audio. I decided to go the custom route as the tool I normally use for this, rtirq, prioritizes all USB threads and I only want the USB threads prioritized that do the audio work. Additional challenge was that these IRQ’s change on every boot. Concocted the following script snippet.

#!/bin/bash

# Prioritize USB port with sound card attached
# Since IRQ's change on every boot figure out IRQ dynamically

# Set maximum priority of the IRQ thread
prio=90

# Next IRQ thread found will get a priority
# decreased with the value set below
prio_step=5

# System paths to look for information
proc_path=/proc/asound
sys_pci_bus_path=/sys/class/pci_bus

# The logic - A for loop that does the following:
# * Iterates through all cards that are set by ALSA
# * Determines if it's an USB card
# * Sets priority on IRQ thread if this is the case,
    lowest card number gets the highest priority
for card_number in $(awk '/\[.*\]/ {print $1}' /proc/asound/cards); do

  if [ -e "$proc_path/card$card_number/usbid" -a -e "$proc_path/card$card_number/stream0" ]; then
    snd_dev_card=card$card_number
    snd_dev_pci_bus_ref=$(grep -Eo "usb-[^[:space:],-]+" $proc_path/$snd_dev_card/stream0 | sed "s/usb-\(.*\)/\1/")
    snd_dev_pci_bus_ref_short=$(awk -F ':' '{print $1":"$2}' <<<$snd_dev_pci_bus_ref)
    snd_dev_irq=$(cat $sys_pci_bus_path/$snd_dev_pci_bus_ref_short/device/$snd_dev_pci_bus_ref/irq)
    snd_dev_irq_pid=$(pgrep $snd_dev_irq-xhci)

    chrt -f -p $prio $snd_dev_irq_pid

    prio=$((prio-prio_step))
  fi

done

This snippet assumes the card numbers are set properly by assigning each card their own index value through the snd-usb-audio kernel module. This can be done with a file in /etc/modprobe.d/, i.e. /etc/modprobe.d/audio.conf. For my USB devices the relevant line in this file looks like this:

# RME Babyface, Edirol UA-25, Akai MPK Mini, Arturia BeatStep Pro, Behringer BCR2000
options snd-usb-audio index=0,1,5,6,7 vid=0x0424,0x0582,0x09e8,0x1c75,0x1397 pid=0x3fb7,0x0074,0x007c,0x0287,0x00bc

So my RME Babyface gets the lowest index (card number) and thus the highest real-time priority.

For onboard audio the situation was a bit trickier. The Lenovo comes with three different audio devices:

  • Onboard audio, speakers and TRRS jack
  • Digital audio, HDMI
  • Onboard mic

I wanted to index all properly so they don’t get in the way of my USB devices. For the onboard audio and HDMI this was no problem, adding the following line to /etc/modprobe.d/audio.conf was enough:

# Onboard audio
options snd-hda-intel index=10,11

Unfortunately you can’t discern between multiple devices with the snd-hda-intel driver and in my case both cards also have no model name so they show up as HD-Audio Generic cards with ID names Generic and Generic_1. Not very helpful. Luckily you can assign ID names dynamically after boot so I used that to give each card a proper ID name:

# Assign proper ID's to onboard devices
# Device with vendor ID 1002 is HDMI
# Device with vendor ID 1022 is onboard audio
for card in card{10,11}; do

  if grep -q 1002 /sys/class/sound/$card/device/vendor; then
    echo -n HDMI > /sys/class/sound/$card/id

  elif grep -q 1022 /sys/class/sound/$card/device/vendor; then
    echo -n ALC257 > /sys/class/sound/$card/id

    # Prioritize IRQ of onboard audio
    snd_dev_irq=$(cat /sys/class/sound/$card/device/irq)
    snd_dev_irq_pid=$(pgrep $snd_dev_irq-snd_hda_intel)

    chrt -f -p $prio $snd_dev_irq_pid
  fi

done

Now both cards can be used with their ID names (so hw:ALC257 for instance) and the onboard audio device gets prioritized with the $prio value set earlier on. Now the only culprit remaining was the onboard mic.

The onboard mic, which gets ID name acp63, is driven by a kernel module with the name of snd-soc-ps-mach. Now this module doesn’t take the index parameter so enter the slots parameter for the top level snd kernel module. With this parameter you can set which slot gets assigned to a specific driver. So added the following to /etc/modprobe.d/audio.conf:

# Onboard mic
options snd slots=,,,,,,,,,,,,snd-soc-ps-mach

And voilà, this is what arecord -l now thinks of it:

$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: Babyface2359686 [Babyface (23596862)], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 11: ALC257 [HD-Audio Generic], device 0: ALC257 Analog [ALC257 Analog]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 12: acp63 [acp63], device 0: DMIC capture dmic-hifi-0 []
  Subdevices: 1/1
  Subdevice #0: subdevice #0

Awesome, full control again! Everything properly named and prioritized. But how does this perform?

With my RME Babyface I can go down to 64 frames/period and 3 periods at 48kHz, so that’s a nominal latency of 4ms. And it’s rock solid at this setting:

$ ./xruncounter -m

******************** SYSTEM CHECK *********************

    Sound Playback: USB-Audio - Babyface (23596862)
     Sound Capture: USB-Audio - Babyface (23596862)
      Graphic Card: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev c7)
Operating System: Debian GNU/Linux 12 (bookworm)
          Kernel: Linux 6.6.11-1-liquorix-amd64
    Architecture: x86-64
               CPU: AMD Ryzen 7 7840HS with Radeon 780M Graphics

***************** jackd start parameter ****************

    /usr/bin/jackd -P80 -S -dalsa -dhw:Babyface2359686 -r48000 -p64 -n3 -Xseq

********************** Pulseaudio **********************

    pulse is not active

********************** Test 8 Core *********************

Samplerate is 48000Hz 
Buffersize is 64 
Buffer/Periods  3
jack running with realtime priority 80
Xrun 1 at DSP load 83.76% use 3.56ms from 1.33ms jack cycle time
Xrun 2 at DSP load 92.76% use 3.32ms from 1.33ms jack cycle time
Xrun 3 at DSP load 87.17% use 2.48ms from 1.33ms jack cycle time
Xrun 4 at DSP load 92.11% use 1.15ms from 1.34ms jack cycle time
Xrun 5 at DSP load 95.91% use 3.29ms from 1.33ms jack cycle time
Xrun 6 at DSP load 97.95% use 1.85ms from 1.33ms jack cycle time
in complete 6 Xruns in 16809 cycles                                  
first Xrun happen at DSP load 83.76% in cycle 16112
process takes 3.56ms from total 1.34ms jack cycle time

On my old BTO I could go lower though, it would run at 32 frames/period and 3 periods at 48kHz with clean audio but the Lenovo is limited to 64 frames/period. If you try to go lower you will get distorted audio and the kernel ring buffer will fill up with messages like below:

[11883.067551] retire_capture_urb: 1338 callbacks suppressed
[11883.083397] xhci_hcd 0000:66:00.4: WARN Event TRB for slot 1 ep 5 with no TDs queued?

Eventually I’ll dive deeper into this but for now I’m OK with running at 64 frames/period. The onboard audio runs too at very low settings but provides way less room to do anything useful. But just about enough to run some soft synths and MIDI input. As a comparison, here’s the output of xruncounter for the onboard audio. In my case onboard audio likes period sizes higher than 3 best. And no full duplex at this setting, playback only.

$ ./xruncounter -m

******************** SYSTEM CHECK *********************

      Graphic Card: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev c7)
Operating System: Debian GNU/Linux 12 (bookworm)
          Kernel: Linux 6.6.11-1-liquorix-amd64
    Architecture: x86-64
               CPU: AMD Ryzen 7 7840HS with Radeon 780M Graphics

***************** jackd start parameter ****************

    /usr/bin/jackd -P80 -S -dalsa -dhw:ALC257 -r48000 -p32 -n6 -Xseq -P

********************** Pulseaudio **********************

    pulse is not active

********************** Test 8 Core *********************

Samplerate is 48000Hz 
Buffersize is 32 
Buffer/Periods  6
jack running with realtime priority 80
Xrun 1 at DSP load 97.32% use 0.40ms from 0.67ms jack cycle time
in complete 1 Xruns in 5781 cycles                                  
first Xrun happen at DSP load 97.32% in cycle 5527
process takes 0.40ms from total 0.67ms jack cycle time

So in my opinion the Lenovo performs pretty well. I do need to run these tests on my old BTO too to find out how much performance I’ve gained. And if I could find a way to work around or find a solution for those xhci_hcd warnings so that I can go even lower then that would be terrific. It could very well be a limitation of the USB implementation of this notebook but I can live with that as it runs really stable at sub 5ms latencies.

New notebook – Doing real-time audio

New notebook – The last bits

Sorted out the last bits that didn’t work smoothly yet.

WiFi

WiFi connection was intermittent and slow. lspci had this to say about it:

01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter

There are numerous reports about this controller not working, almost all firmware related. The Liquorix kernel I’m running has the latest firmware so that was not the issue. Then I stumbled on a report related to power saving. Disabled that through NetworkManager by adding an extra configuration file in /etc/NetworkManager/conf.d with the following lines in it:

[connection]
wifi.powersave = 2

WiFi is now stable and fast.

Keyboard

Backlight

This can be controlled through sysfs or D-Bus but the system thinks there are only three settings: off (0), dimmed (1) or bright (2). But my keyboard thinks there are four settings, the fourth being what I would call “responsive bright”. This setting disables the backlight after a minute of inactivity and responds to key strokes by enabling backlight again to the bright setting. You can toggle this setting on the keyboard itself with Fn+Space but unfortunately the system does not know about this. I’ve worked around this by using a tiny daemon that listens for key strokes and enables the backlight and disabling it again when there’s no activity for a minute. There are several solutions for this, I settled for kbd_backlight_ctrl because it works and it’s just a few lines of C. Can’t set the brightness though but since it’s just a few lines of code I see this as an opportunity to improve my non-existent C skills.

Media keys

The keyboard has four dedicated media keys, play/pause, stop, previous and next. These work perfectly with a media player like VLC but Ardour does not recognize them. By creating supplementary udev hardware database entries I remapped the media keys so they’re now also functioning in Ardour which is quite neat. Only the stop key has no purpose yet as Ardour has no shortcut for what I want to map it to, stopping and going back to the start marker. Maybe I could create something myself through Ardour’s Lua scripting engine.

The hardware database entries look like this:

evdev:input:b0011v0001p0001*
 KEYBOARD_KEY_a2=space
 KEYBOARD_KEY_90=left
 KEYBOARD_KEY_99=right

These went into /etc/udev/hwdb.d/99-media-keys.hwdb. After running systemd-hwdb update && udevadm trigger as root Ardour now sees the play/pause button as space, previous as left and next as right. I used Remapping Keyboard Keys in Ubuntu with udev / evdev as a reference.

Audio devices

There are three audio devices on this machine:

$ lspci | grep -i audio
64:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller
64:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor (rev 63)
64:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller
$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: Babyface2359686 [Babyface (23596862)], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 10: Generic [HD-Audio Generic], device 3: HDMI 0 [HDMI 0]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 10: Generic [HD-Audio Generic], device 7: HDMI 1 [HDMI 1]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 10: Generic [HD-Audio Generic], device 8: HDMI 2 [HDMI 2]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 11: Generic_1 [HD-Audio Generic], device 0: ALC257 Analog [ALC257 Analog]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: Babyface2359686 [Babyface (23596862)], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 1: acp63 [acp63], device 0: DMIC capture dmic-hifi-0 []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 11: Generic_1 [HD-Audio Generic], device 0: ALC257 Analog [ALC257 Analog]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

So onboard audio, HDMI and a capture device acp63. As you can see I’ve managed to get onboard and HDMI out of the way by indexing them but unfortunately the kernel module for the capture device, snd-soc-ps-mach, does not seem to support indexing. I’d like to index it as card 12 because now that acp63 device claims 0 when my Babyface is not attached. Something for later investigation.

I’ll dedicate a separate post to doing real-time audio on this machine. I need to do some more stress testing but Ardour runs without complaining at 48kHz, 64 frames/period and 3 periods/buffer, so 4ms system latency. No weird things there.

New notebook – The last bits

New notebook – The day after

Everything seems to work, network, audio, external monitor, display brightness. And this is the first notebook I’ve ever come across to that has a touch pad with an actual real middle click! Love it! Keyboard is nice too, backlight works fine and it has a nice array of function keys that are almost all mapped now. Overall feel of this machine is really nice, aluminum housing and the chiclet keyboard feels stable and solid. Startup time is twice as fast as the BTO. Compared to my new notebook the old one feels strangely completely outdated.

Now what is in there? CPU is an AMD Ryzen 7 7840HS, an 8-core CPU with SMT. Disabled SMT though, Ardour seems to run slightly better on 8 real cores. 32GB of RAM which should be more than enough, my old notebook had 16GB and never had any issues with that. Storage is a 1TB SK hynix BC901 NVMe drive with more than decent throughput. Almost all USB ports sit on their own bus which is quite nice. The only bus that shares its IRQ with something else (the WiFi module) is bus 001. USB ports that use this bus are both the USB-C and USB Type-A connectors closest to the screen. So better not use those.

The only thing that is a bit flaky is the WiFi connection but came across more mentions about that. The connection sometimes randomly reconnects. Can live with it, when making music I often completely disable WiFi. Did have to use systemd automount instead of an fstab entry to mount my NAS as the WiFi connection apparently takes a bit too long to come up.

Didn’t have to tweak a lot so far. As I already mentioned I did disable SMT (Simultaneous Multi-Threading) and I’ve prioritized the IRQ of the USB bus my audio interface sits on. Not with rtirq or udev-rtirq (which doesn’t work properly on this machine, the wrong IRQ seems to get prioritized) but with a small script to fetch the IRQ number that seems to change on every boot and feeding the outcome to chrt. Installed a liquorix kernel on it and enabled threaded IRQ’s, disabled mitigations and USB autosuspend. Also allowed my user to set CPU DMA latency.

BTO vs Lenovo

It’s running Debian 12 with XFCE again. Simple, fast and looks good enough to me with the Greybird theme and elementary icons. And no more notebook that takes off into orbit, the new one is way more quiet.

On with making music!

Edit: the script I threw together could be useful for others too with USB audio interfaces connected to a machine that uses MSI (Message-Signaled Interrupts) enabled USB controllers. You can find the snd_dev_id of your interface with aplay -l.

#!/bin/bash

prio=90
prio_step=5
proc_path=/proc/asound
sys_pci_bus_path=/sys/class/pci_bus
snd_dev_ids=( Babyface2359686 UA25 )

for snd_dev_id in ${snd_dev_ids[@]}; do
  snd_dev_card_number=$(awk '/'$snd_dev_id'/ {print $1}' $proc_path/cards)

  if [ -n "$snd_dev_card_number" ]; then
    snd_dev_card=card$snd_dev_card_number
    snd_dev_pci_bus_ref=$(grep -Eo "usb-[^[:space:],-]+" $proc_path/$snd_dev_card/stream0 | sed "s/usb-\(.*\)/\1/")
    snd_dev_pci_bus_ref_short=$(awk -F ':' '{print $1":"$2}' <<<$snd_dev_pci_bus_ref)
    snd_dev_irq=$(cat $sys_pci_bus_path/$snd_dev_pci_bus_ref_short/device/$snd_dev_pci_bus_ref/irq)
    snd_dev_irq_pid=$(pgrep $snd_dev_irq-xhci)

    chrt -f -p $prio $snd_dev_irq_pid
    
    prio=$((prio-prio_step))
  fi

done
New notebook – The day after

New notebook – Lenovo IdeaPad Pro 5

After long consideration I ended up with a Lenovo IdeaPad Pro 5 16APH8 (83AR0045MH). For a while I was eying a BTO P•BOOK 16P1390 that got a good review from an old colleague of mine from the UvA. That’s actually a TongFang ID6H2 but by the time I made up my mind the version I actually wanted with an i7-13700H CPU was already sold out. Why BTO? And why replace it? Well, my current audio workstation is a BTO from 2013 and it has served me well, actually, it still runs amazingly well but it’s showing its age here and there. No support for bigger external monitors for instance, and I’m also hitting full DSP load in Ardour a bit too early.

So I did some more research and concluded that it might be a good thing to switch to AMD. Not only because Linus is on AMD now but I just want 8 cores that perform the same. Add to that that quite some Linux music producers favor AMD too and that the Ryzen 7 7840HS I went for doesn’t differ that much performance wise from its Intel i7-13700H equivalent. But it’s cheaper in the configuration I chose compared to a BTO alternative. I also looked at other vendors like Laptop With Linux and Tuxedo but the Clevo’s from Laptop With Linux are too bulky for my taste and are a bit behind spec wise and Tuxedo only offers keyboards with ISO layouts and they’re also quite pricey. I quickly let go of the idea of getting a Framework notebook, that would easily get twice as expensive as the notebook I have now and despite how much I love their sustainability approach that is simply too much. And not buying a new notebook every two or three years is pretty sustainable too I guess.

I verified beforehand if Linux would run on my Lenovo and found some references that confirmed Linux would run well on it. Given the track record Lenovo has with Linux on its notebooks this was no surprise. Just picked it up from a local store which I find more convenient and safer than having it sent over from an online vendor or even Lenovo itself. Lenovo does offer an option without an OS though but a dual boot option can come in handy, especially when having to update firmware of external devices or the firmware of the notebook itself.

Installing Debian 12 on it as we speak and the initial setup went very smoothly. I’m surprised at how small and light it is. And I’ll have to get used to the resolution, but the WQXGA resolution was a conscious decision. I work with two screens, Ardour editor on my main screen and the mixer on my notebook screen and coming from 1920×1080 @ 60Hz the Lenovo with its 2560×1600 @ 120 Hz screen will definitely be an improvement.

On with setting up my new machine!

New notebook – Lenovo IdeaPad Pro 5

Balancing act

At the moment everything seems a bit like a balancing act. First on a physical level, as I’m currently recovering from surgery one of the things I have to learn again is to find my balance, literally. While recovering I can’t do very much hobby stuff in the analogue domain so I swayed a bit to the digital domain again.

About 6 years ago I bought a Mixbus 32C license but found myself using Ardour more and more. During the pandemic I took a subscription and from then on I basically started using Ardour exclusively. Still remember the first time I opened up Ardour back in the 00’s, to me it was intimidating, daunting, what did all those buttons and sliders do? But like with more things in life, sometimes you just fathom the seeming complexity of something, call it an eye opener, and then you’re like, why didn’t I start using Ardour right from the beginning?

Now Ardour is my DAW of choice. It’s running on Debian 12 with a Liquorix kernel on my old, trusted BTO and I’ve never had such a stable setup before. Yes, Debian, after 14 years of Ubuntu that has become a balancing act too. The more applications are moved into Snap the more it alienates me from the OS. While I understand the concept of self-contained applications, it’s part of my job, I don’t think this concept has a real purpose on a desktop OS. It adds another layer of complexity and makes communication between applications harder. Whole different story for another time.

Ardour 8.0 has just been released and I can wholeheartedly recommend it. Installing and setting it up is a breeze and even on my old BTO it runs like a charm. The only restriction is that I can’t use too many Dragonfly Reverb plugins within a project but once I give in to my GAS to get a Framework notebook that will be resolved too.

Balancing act

rtcqs v0.4.2 released

This release comes with a new Power Management check which checks if the audio group has read/write permissions on /dev/cpu_dma_latency. If your user is a member of the audio group and permissions are set for this group then DAW’s like Ardour and Reaper can open this file as your user, keep it open and control power management this way. This allows a user to prevent CPU sleep states for example so your CPUs are always on and instantly available which could lower the chance running into xruns.

Ardour CPU DMA latency setting Under Edit – Preferences – Performance

This release also introduces a new basic and simple tkinter-based GUI. The Qt GUI does look fancy but to use it it also needs a fancy amount of dependencies. When building binaries with PyInstaller the result of the Qt GUI is a whopping 130MB package while the tkinter version stays below 12MB.

rtcqs tkinter GUI

Future plans are to get rid of some checks:

  • Max user watches as it’s not related to the overall performance of your system
  • System timer as it’s not relevant anymore, rtcqs already checks for the more relevant stuff (high res timers and tickless kernel)
  • Background processes as it’s merely a placeholder which checks for two processes that don’t exist anymore on modern systems

I’m having my doubts about swappiness too as it’s not really applicable anymore for modern machines. But I’m curious if it still applies for smaller systems like RPi’s for example. I’d like to add a filesystem mount option check, for Ext it would check if the filesystem is mounted at least with the relatime option or even noatime for example. And maybe a disk scheduler check but I’m not conviced yet that it really makes a difference.

The new release and binary packages of rtcqs and rtcqs_simple_gui can be found on the Codeberg repo: https://codeberg.org/rtcqs/rtcqs/releases/tag/v0.4.2

rtcqs v0.4.2 released

RPi 3 and the real time kernel

As a beta tester for MOD I thought it would be cool to play around with netJACK which is supported on the MOD Duo. The MOD Duo can run as a JACK master and you can connect any JACK slave to it as long as it runs a recent version of JACK2. This opens a plethora of possibilities of course. I’m thinking about building a kind of sidecar device to offload some stuff to using netJACK, think of synths like ZynAddSubFX or other CPU greedy plugins like fat1.lv2. But more on that in a later blog post.

So first I need to set up a sidecar device and I sacrificed one of my RPi’s for that, an RPi 3. Flashed an SD card with Raspbian Jessie Lite and started to do some research on the status of real time kernels and the Raspberry Pi because I’d like to use a real time kernel to get sub 5ms system latency. I compiled real time kernels for the RPi before but you had to jump through some hoops to get those running so I hoped things would have improved somewhat. Well, that’s not the case so after having compiled a first real time kernel the RPi froze as soon as I tried to runapt-get install rt-tests. After having applied a patch to fix how the RPi folks implemented the FIQ system the kernel compiled without issues:

Linux raspberrypi 4.9.33-rt23-v7+ #2 SMP PREEMPT RT Sun Jun 25 09:45:58 CEST 2017 armv7l GNU/Linux

And the RPi seems to run stable with acceptable latencies:

Histogram of the latency on the RPi with a real time kernel during 300000 cyclictest loops
Histogram of the latency on the RPi with a real time kernel during 300000 cyclictest loops

So that’s a maximum latency of 75 µs, not bad. I also spotted some higher values around 100 but that’s still okay for this project. The histogram was created with mklatencyplot.bash. I used a different invocation of cyclictest though:

cyclictest -Sm -p 80 -n -i 500 -l 300000

And I ran hackbench in the background to create some load on the RPi:

(while true; do hackbench > /dev/null; done) &

Compiling a real time kernel for the RPi is still not a trivial thing to do and it doesn’t help that the few howto’s on the interwebs are mostly copy-paste work, incomplete and contain routines that are unclear or even unnecessary. One thing that struck me too is that the howto’s about building kernels for RPi’s running Raspbian don’t mention the make deb-pkg routine to build a real time kernel. This will create deb packages that are just so much easier to transfer and install then rsync’ing the kernel image and modules. Let’s break down how I built a real time kernel for the RPi 3.

First you’ll need to git clone the Raspberry Pi kernel repository:

git clone -b 'rpi-4.9.y' --depth 1 https://github.com/raspberrypi/linux.git

This will only clone the rpi-4.9.y branch into a directory called linux without any history so you’re not pulling in hundreds of megs of data. You will also need to clone the tools repository which contains the compiler we need to build a kernel for the Raspberry Pi:

git clone https://github.com/raspberrypi/tools.git

This will end up in the tools directory. Next step is setting some environment variables so subsequent make commands pick those up:

export KERNEL=kernel7
export ARCH=arm
export CROSS_COMPILE=/path/to/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
export CONCURRENCY_LEVEL=$(nproc)

The KERNEL variable is needed to create the initial kernel config. The ARCH variable is to indicate which architecture should be used. The CROSS_COMPILE variable indicates where the compiler can be found. The CONCURRENCY_LEVEL variable is set to the number of cores to speed up certain make routines like cleaning up or installing the modules (not the number of jobs, that is done with the -j option of make).

Now that the environment variables are set we can create the initial kernel config:

cd linux
make bcm2709_defconfig

This will create a .config inside the linux directory that holds the initial kernel configuration. Now download the real time patch set and apply it:

cd ..
wget https://www.kernel.org/pub/linux/kernel/projects/rt/4.9/patch-4.9.33-rt23.patch.xz
cd linux
xzcat ../patch-4.9.33-rt23.patch.xz | patch -p1

Most howto’s now continue with building the kernel but that will result in a kernel that will freeze your RPi because of the FIQ system implementation that causes lock ups of the RPi when using threaded interrupts which is the case with real time kernels. That part needs to be patched so download the patch and dry-run it:

cd ..
wget https://www.osadl.org/monitoring/patches/rbs3s/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch
cd linux
patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1 --dry-run

You will notice one hunk will fail, you will have to add that stanza manually so note which hunk it is for which file and at which line it should be added. Now apply the patch:

patch -i ../usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch -p1

And add the failed hunk manually with your favorite editor. With the FIQ patch in place we’re almost set for compiling the kernel but before we can move on to that step we need to modify the kernel configuration to enable the real time patch set. I prefer doing that with make menuconfig. You will need the libncurses5-dev package to run this commando so install that with apt-get install libncurses5-dev. Then select Kernel Features - Preemption Model - Fully Preemptible Kernel (RT) and select Exit twice. If you’re asked if you want to save your config then confirm. In the Kernel features menu you could also set the the timer frequency to 1000 Hz if you wish, apparently this could improve USB throughput on the RPi (unconfirmed, needs reference). For real time audio and MIDI this setting is irrelevant nowadays though as almost all audio and MIDI applications use the hr-timer module which has a way higher resolution.

With our configuration saved we can start compiling. Clean up first, then disable some debugging options which could cause some overhead, compile the kernel and finally create ready to install deb packages:

make clean
scripts/config --disable DEBUG_INFO
make -j$(nproc) deb-pkg

Sit back, enjoy a cuppa and when building has finished without errors deb packages should be created in the directory above the linux one. Copy the deb packages to your RPi and install them on the RPi with dpkg -i. Open up /boot/config.txt and add the following line to it:

kernel=vmlinuz-4.9.33-rt23-v7+

Now reboot your RPi and it should boot with the realtime kernel. You can check with uname -a:

Linux raspberrypi 4.9.33-rt23-v7+ #2 SMP PREEMPT RT Sun Jun 25 09:45:58 CEST 2017 armv7l GNU/Linux

Since Rasbian uses almost the same kernel source as the one we just built it is not necessary to copy any dtb files. Also running mkknlimg is not necessary anymore, the RPi boot process can handle vmlinuz files just fine.

The basis of the sidecar unit is now done. Next up is tweaking the OS and setting up netJACK.

Edit: there’s a thread on LinuxMusicians referring to this article which already contains some very useful additional information.

RPi 3 and the real time kernel

Lekker sleutelen

Dit wordt mijn nieuwe Linux blogje waarin ik me vooral focus op het maken van muziek met GNU/Linux. Dit vergt namelijk nog wel enig gesleutel aan je distro. Ik wil hier op dit blog dan ook alle info zetten die ik bij elkaar gescharreld heb en mijn ervaringen delen die ik heb opgedaan tijdens het sleutelen aan mijn systemen en het produceren van muziek met GNU/Linux.

Lekker sleutelen