vediksagar
← writing

· 3 min read · #linux #debugging #file-descriptors

Three file descriptors at a time

My status bar went quietly stale after about two hours. The cause was an accessibility socket leaking pidfds, three per connection, until the process hit its open-file limit.

My desktop runs a small status bar I wrote in Python for the i3 window manager. One day it started showing a second, stale copy of itself, and after that it stopped reacting to anything i3 told it. No crash, no traceback. It just went quiet.

This is the story of how that turned out to be a file-descriptor leak, three descriptors at a time, and the one environment variable that stopped it.

The symptom

The bar asks i3 for a snapshot of the screen layout whenever something changes. After a couple of hours of uptime, those queries began failing silently. The bar kept its old picture of the world, so when a monitor went away it never noticed, and a second bar was left drawn where the old one had been.

When a long-running process works fine at first and breaks after hours, the first question is: what is it accumulating?

Counting what the process holds

On Linux every open file, socket and pipe a process holds is listed under /proc/PID/fd. Counting them is cheap:

ls -l /proc/PID/fd | wc -l

The bar was sitting at 1024, which is exactly the default soft limit on open files. Once a process is at that ceiling, every new open() or socket() fails with EMFILE, and any code that does not check for that error just stops doing its job. That was the silent failure.

The interesting part was what the 1024 descriptors were. Most of them were not files or sockets but pidfds, file descriptors that refer to a process:

ls -l /proc/PID/fd | grep -c pidfd

Finding who opens them

A pidfd leak needs something that keeps creating processes, or keeps talking to them. On my desktop that was a separate tool of mine, a window finder that scans the accessibility tree of open apps (the AT-SPI bus that screen readers use). It started a fresh scanner process for every scan.

So I ran scans by hand and counted. Every scan left the bar holding three more pidfds, and they never went away. At the rate I used the finder, the bar reached 1024 in roughly two hours, which matched when it went stale.

The three descriptors are opened when an accessibility client connects to the app’s private, peer-to-peer AT-SPI socket. I traced the descriptors to SO_PEERPIDFD credentials on that socket. On this Ubuntu machine the D-Bus library involved is libdbus 1.16.2; I am inferring that it is the layer that keeps them, I have not read its source to prove it.

The fix

AT-SPI can skip the private socket and route everything through the shared accessibility bus instead. One environment variable turns the peer-to-peer path off:

import os
os.environ["ATSPI_DISABLE_P2P"] = "1"  # must run before GTK/AT-SPI initialise

With that line at the top of the bar, four scans went from 12 leaked pidfds to 0, and the finder still saw all 96 accessibility nodes of the bar, so nothing it depended on broke.

I also set it for the whole session, so every app started after login gets it:

# ~/.config/environment.d/90-atspi-no-p2p.conf
ATSPI_DISABLE_P2P=1

Making it fail loudly next time

The leak was the cause, but the reason it took hours to notice was that the failure was silent. So the bar now logs every failed snapshot together with its current open-file count, and retries three times with backoff (1, 2 and 4 seconds). If anything like this happens again, the log will say “snapshot failed, 1024 fds open” instead of nothing.

There is also a regression test that runs a few scans against the bar and asserts that the pidfd count does not grow.

What I took from it

  • When something breaks only after hours, count what it holds before reading any code. /proc/PID/fd answers in a second.
  • 1024 is a number worth recognising on sight: it is the default open-file soft limit.
  • The real bug was not the leak, it was code that hit EMFILE and kept going as if nothing happened. Errors you cannot see are the expensive ones.

Other long-running apps on the same desktop (the browser, the Bluetooth applet and a couple of GTK helpers) had also climbed toward their limits. They pick up the fix on their next start.