Blog posts for tags/api-design

  1. Removing Process.memory_maps() on macOS

    This is part of the psutil 5.6.0 release (see the full release notes).

    As of 5.6.0, Process.memory_maps() is no longer defined on macOS.

    The bug

    #1291: on macOS, Process.memory_maps() would either raise OSError: [Errno 22] Invalid argument or segfault the whole Python process! Both triggered from code as simple as psutil.Process().as_dict(), since Process.as_dict() iterates every attribute, and Process.memory_maps() is one of them.

    The root cause was inside Apple's undocumented proc_regionfilename() syscall. On some memory regions it returns EINVAL. On others it takes the process down. Which regions? Nobody figured out. Arnon Yaari (@wiggin15) did most of the investigation: he wrote a standalone C reproducer and walked me through what he'd tried.

    In PR-1436 I attempted a fix by reverse-engineering vmmap(1) but it didn't work. The fundamental problem is that vmmap is closed source and proc_regionfilename is undocumented. Neither my virtualized macOS (10.11.6) nor Travis CI (10.12.1) could reproduce the bug, which reproduced reliably only on 10.14.3.

    Why remove outright

    While removing the C code I noticed that the macOS unit test had been disabled long ago, presumably by me after recurring flaky Travis runs. Meaning that the method had been broken on some macOS versions far longer than the 2018 bug report suggested.

    Deprecating for a cycle didn't help either: raising AccessDenied breaks code that relied on a successful return, returning an empty list does the same silently, and leaving the method in place doesn't stop the segfault. Basically there was no sane solution, so since 5.6 is a major version I decided to just remove Process.memory_maps() for good.

    On macOS it never supported other processes anyway. Calling it on any PID other than the current one (or its children) raised AccessDenied, even as root.

    If someone finds a Mach API path that works, the method can return. Nobody has found one so far.

  2. Fixing Unicode across Python 2 and 3

    This one took a while. Adding proper Unicode support to psutil took four months of auditing, design decisions, and rewriting nearly every API that returned a string. The full journey is documented in #1040, and what follows is a summary.

    This can serve as a case study for any Python library with a C extension that needs to support both Python 2 and Python 3, as it will encounter the exact same set of problems.

    What was broken

    psutil has different APIs returning a string, many of which misbehaved when it came to unicode. There were three distinctive problems (#1040). Each API could:

    • A: raise a decoding error for non-ASCII strings (Python 3).
    • B: return unicode instead of str (Python 2).
    • C: return incorrect / invalid encoded data for non-ASCII strings (both).

    Process.memory_maps() hit all three on various OSes. disk_partitions() raised decoding errors on every UNIX except Linux. Windows service methods leaked unicode into Python 2 return values. The C extension had accumulated years of ad-hoc encode/decode decisions, with no single rule covering all of them.

    It was a mess.

    Filesystem or locale encoding?

    First problem was that the C extension was using 2 approaches when it came to decoding and returning a string: PyUnicode_DecodeFSDefault (filesystem encoding) for path-like APIs, and PyUnicode_DecodeLocale (user locale) for non-path strings like Process.username().

    It appeared clear that I had to use PyUnicode_DecodeFSDefault for all filesystem-related APIs like Process.exe() and Process.open_files().

    It was less clear, though, when to use PyUnicode_DecodeLocale.

    After some back and forth, I decided to use a single encoding for all APIs: the filesystem encoding (PyUnicode_DecodeFSDefault). This makes the encoding choice an implementation detail of psutil, not something the user has to care about.

    Error handling

    Second question was what to do in case the string cannot be correctly decoded (because invalid, corrupted or whatever). On Python 3 + UNIX the natural choice was 'surrogateescape', which is also the default for PyUnicode_DecodeFSDefault. On Windows the default is 'surrogatepass' (Python 3.6) or 'replace' as per PEP 529.

    And here come the troubles: Python 2 is different. To correctly handle all kinds of strings on Python 2 we should return unicode instead of str, but I didn't want to do that, nor have APIs which return two different types depending on the circumstance.

    Since unicode support is already broken in Python 2 and its stdlib (see bpo-18695), I was happy to always return str, use 'replace' as the error handler, and simply consider unicode support in psutil + Python 2 broken.

    Final behavior

    Starting from 5.3.0, psutil behaves consistently across all APIs that return a string. The rules are intentionally simple, even if the underlying implementation is not.

    The notes below apply to any method returning a string such as Process.exe() or Process.cwd(), including non-filesystem-related methods such as Process.username():

    • all strings are encoded using the OS filesystem encoding (PyUnicode_DecodeFSDefault), which varies depending on the platform you're on (e.g. 'UTF-8' on Linux, 'mbcs' on Windows).

    • no API call is supposed to crash with UnicodeDecodeError.

    • in case of badly encoded data returned by the OS, the following error handlers are used to replace the bad characters in the string:

      • Python 2: 'replace'.
      • Python 3: 'surrogateescape' on POSIX, 'replace' on Windows.

    • on Python 2 all APIs return bytes (str type), never unicode.

    • on Python 2 you can go back to unicode by doing:

      >>> unicode(proc.exe(), sys.getdefaultencoding(), errors="replace")

    The full journey was implemented in PR-1052, and shipped in 5.3.0 (see the changelog).

  3. Improved process_iter()

    This is part of the psutil 5.3.0 release (see the changelog for the full list of changes).

    The old pattern

    Iterating over processes and collecting attributes requires more boilerplate than it should. A process returned by psutil.process_iter() may disappear before you access it, or require elevated privileges, so every lookup has to be guarded with a try / except:

    >>> import psutil
>>> for proc in psutil.process_iter():
...     try:
...         pinfo = proc.as_dict(attrs=['pid', 'name'])
...     except (psutil.NoSuchProcess, psutil.AccessDenied):
...         pass
...     else:
...         print(pinfo)
...
{'pid': 1, 'name': 'systemd'}
{'pid': 2, 'name': 'kthreadd'}
{'pid': 3, 'name': 'ksoftirqd/0'}

    This is not decorative. It's necessary to avoid the race condition.

    The new pattern

    5.3.0 adds attrs and ad_value parameters to psutil.process_iter(). With these, the loop body becomes:

    >>> import psutil
>>> for proc in psutil.process_iter(attrs=['pid', 'name']):
...     print(proc.info)
...
{'pid': 1, 'name': 'systemd'}
{'pid': 2, 'name': 'kthreadd'}
{'pid': 3, 'name': 'ksoftirqd/0'}

    Internally, process_iter() attaches an info dict to the Process instance. The attributes are pre-fetched in one shot. Processes that disappear during iteration are silently skipped, and attributes that would raise AccessDenied get assigned ad_value, which defaults to None:

    for p in psutil.process_iter(['name', 'username'], ad_value="N/A"):
    print(p.name(), p.username())

    Performance

    Beyond the syntactic win, the new syntax is also faster than calling individual methods in a loop. process_iter(attrs=[...]) is equivalent to using Process.oneshot() on each process (see Making psutil twice as fast for how that works): attributes that share a syscall or a /proc file are fetched together instead of re-read on every method call, which is a lot faster.

    Comprehensions

    With the exception boilerplate out of the way, comprehensions finally work cleanly. E.g. getting processes owned by the current user can be written as:

    >>> import getpass
>>> from pprint import pprint as pp
>>> pp([(p.pid, p.info['name']) for p in psutil.process_iter(attrs=['name', 'username']) if p.info['username'] == getpass.getuser()])
[(16832, 'bash'),
 (19772, 'ssh'),
 (20492, 'python')]

  4. Proper zombie process handling

    This is part of the psutil 3.0 release (see the full release notes).

    Except on Linux and Windows (which does not have them), support for zombie processes was broken. The full story is in #428.

    The problem

    Say you create a zombie process and instantiate a Process for it:

    import os, time

def create_zombie():
    pid = os.fork()  # the zombie
    if pid == 0:
        os._exit(0)  # child exits immediately
    else:
        time.sleep(1000)  # parent does NOT call wait()

pid = create_zombie()
p = psutil.Process(pid)

    Up until psutil 2.X, every time you tried to query it you'd get a NoSuchProcess exception:

    >>> p.name()
  File "psutil/__init__.py", line 374, in _init
    raise NoSuchProcess(pid, None, msg)
psutil.NoSuchProcess: no process found with pid 123

    This was misleading, because the PID technically still existed:

    >>> psutil.pid_exists(p.pid)
True

    Depending on the platform, some process information could still be retrieved:

    >>> p.cmdline()
['python']

    Worst of all, psutil.process_iter() didn't return zombies at all. That was a real problem, because identifying them is a legitimate use case: a zombie usually indicates a bug where a parent process spawns a child, kills it, but never calls wait() to reap it.

    What changed

    • A new ZombieProcess exception is raised whenever a process cannot be queried because it is a zombie.
    • It replaces NoSuchProcess, which was incorrect and misleading.
    • ZombieProcess inherits from NoSuchProcess, so existing code keeps working.
    • psutil.process_iter() now correctly includes zombie processes, so you can reliably identify them:
    import psutil

zombies = []
for p in psutil.process_iter():
    try:
        if p.status() == psutil.STATUS_ZOMBIE:
            zombies.append(p)
    except psutil.NoSuchProcess:
        pass

  5. Announcing psutil 2.0

    psutil 2.0 is out. This is a major rewrite and reorganization of both the Python and C extension modules. It costed me four months of work and more than 22,000 lines (the diff against old 1.2.1). Many of the changes are not backward compatible; I'm sure this will cause some pain, but I think it's for the better and needed to be done.

    API changes

    I already wrote a detailed blog post about this, so use that as the official reference on how to port your code.

    RST documentation

    I've never been happy with the old doc hosted on Google Code. The markup language provided by Google is pretty limited, plus it's not under revision control. The new doc is more detailed, uses reStructuredText as the markup language, lives in the same code repository as psutil, and is hosted on the excellent Read the Docs: http://psutil.readthedocs.org/

    Physical CPUs count

    You're now able to distinguish between logical and physical CPUs. The full story is in #427.

    >>> psutil.cpu_count()  # logical
4
>>> psutil.cpu_count(logical=False)  # physical cores only
2

    Process instances are hashable

    psutil.Process instances can now be compared for equality and used in sets and dicts. The most useful application is diffing process snapshots:

    >>> before = set(psutil.process_iter())
>>> # ... some time passes ...
>>> after = set(psutil.process_iter())
>>> new_procs = after - before  # processes spawned in between

    Equality is not just PID-based. It also includes the process creation time, so a Process whose PID got reused by the kernel won't be mistaken for the original. The full story is in #452.

    Speedups

    • #477: Process.cpu_percent() is about 30% faster.
    • #478: (Linux) almost all APIs are about 30% faster on Python 3.X.

    Other improvements and bugfixes

    • #424: published Windows installers for Python 3.X 64-bit.
    • #447: the psutil.wait_procs() timeout parameter is now optional.
    • #459: a Makefile is now available for running tests and other repetitive tasks (also on Windows).
    • #463: the timeout parameter of cpu_percent* functions defaults to 0.0, because the previous default was a common source of slowdowns.
    • #340: (Windows) Process.open_files() no longer hangs.
    • #448: (Windows) fixed a memory leak affecting Process.children() and Process.ppid().
    • #461: namedtuples are now pickle-able.
    • #474: (Windows) Process.cpu_percent() is no longer capped at 100%.

Social

Feed