Blog posts for tags/release

  1. Letting go of Python 2.7

    About dropping Python 2.7 support in psutil, 3 years ago I stated (#2014):

    Not a chance, for many years to come. [Python 2.7] currently represents 7-10% of total downloads, meaning around 70k / 100k downloads per day.

    Only 3 years later, and to my surprise, downloads for Python 2.7 dropped to 0.36%! As such, as of psutil 7.0.0, I finally decided to drop support for Python 2.7!

    The numbers

    These are downloads per month:

    $ pypinfo --percent psutil pyversion
    Served from cache: False
    Data processed: 4.65 GiB
    Data billed: 4.65 GiB
    Estimated cost: $0.03
    
    | python_version | percent | download_count |
    | -------------- | ------- | -------------- |
    | 3.10           |  23.84% |     26,354,506 |
    | 3.8            |  18.87% |     20,862,015 |
    | 3.7            |  17.38% |     19,217,960 |
    | 3.9            |  17.00% |     18,798,843 |
    | 3.11           |  13.63% |     15,066,706 |
    | 3.12           |   7.01% |      7,754,751 |
    | 3.13           |   1.15% |      1,267,008 |
    | 3.6            |   0.73% |        803,189 |
    | 2.7            |   0.36% |        402,111 |
    | 3.5            |   0.03% |         28,656 |
    | Total          |         |    110,555,745 |
    

    According to pypistats.org Python 2.7 downloads represent 0.28% of the total, around 15,000 downloads per day.

    The pain

    Keeping 2.7 alive had become increasingly difficult, but still possible: tests ran via old PyPI backports and a tweaked GitHub Actions workflow on Linux and macOS, plus a separate third-party service (Appveyor) for Windows. But the workarounds in the source kept piling up:

    • A Python compatibility layer (psutil/_compat.py) plus #if PY_MAJOR_VERSION <= 3 branches in C, with constant str-vs-unicode juggling on both sides.
    • No f-strings, and no free use of enum for constants (which ended up with a different API shape than on Python 3).
    • An outdated pip and outdated deps.
    • 4 extra CI jobs per commit (Linux, macOS, Windows 32-bit and 64-bit), making the pipeline slower and flakier.
    • 7 wheels specific to Python 2.7 to ship on every release:
    psutil-6.1.1-cp27-cp27m-macosx_10_9_x86_64.whl
    psutil-6.1.1-cp27-none-win32.whl
    psutil-6.1.1-cp27-none-win_amd64.whl
    psutil-6.1.1-cp27-cp27m-manylinux2010_i686.whl
    psutil-6.1.1-cp27-cp27m-manylinux2010_x86_64.whl
    psutil-6.1.1-cp27-cp27mu-manylinux2010_i686.whl
    psutil-6.1.1-cp27-cp27mu-manylinux2010_x86_64.whl
    

    The removal

    The removal was done in PR-2481, which dropped around 1500 lines of code (nice!). It felt liberating. In doing so, in the doc I still made the promise that the 6.1.* series will keep supporting Python 2.7 and will receive critical bug-fixes only (no new features). It will be maintained in a specific python2 branch. I explicitly kept the setup.py script compatible with Python 2.7 in terms of syntax, so that, when the tarball is fetched from PyPI, it will emit an informative error message on pip install psutil. The user trying to install psutil on Python 2.7 will see:

    $ pip2 install psutil
    As of version 7.0.0 psutil no longer supports Python 2.7.
    Latest version supporting Python 2.7 is psutil 6.1.X.
    Install it with: "pip2 install psutil==6.1.*".
    
  2. System load average on Windows in Python

    psutil 5.6.2 is out. It implements an emulation of os.getloadavg() on Windows, kindly contributed by Ammar Askar, who originally implemented it for CPython's test suite.

    This idea has been floating around for quite a while. The first proposal dates back to 2010, when psutil was still hosted on Google Code, and it popped up multiple times over the years. There's a bunch of info online mentioning the pieces you'd theoretically use (the so-called System Processor Queue Length), but I couldn't find any real implementation. A quick search suggests there's real demand for this, but very few tools provide it natively (the only ones I could find are sFlowTrend and Zabbix). So I'm glad this finally landed in psutil / Python.

    Other improvements and bugfixes in psutil 5.6.2

    The full list is in the changelog. A couple worth mentioning:

    • #1476: ability to set a process's high I/O priority on Windows.
    • #1458: colorized test output. Nobody will use this directly, but it's nice and I'm porting it to other projects I maintain (e.g. pyftpdlib). Good candidate for a small PyPI module that could also include the unittest extensions I've been re-implementing piece by piece:
      • #1478: re-running failed tests.
      • display test timings / durations. This is something I'm also contributing to CPython: BPO-4080 and PR-12271.

    About me

    I'm currently in China (Shenzhen) for a mix of vacation and work, and I will likely take a break from Open Source for a while (about 2.5 months), during which I'll also go to the Philippines and Japan.

    External

  3. Announcing psutil 5.6.0

    psutil 5.6.0 is out. Highlights: a new Process.parents() method, several important Windows improvements, and the removal of Process.memory_maps() on macOS.

    Process parents()

    The new method returns the parents of a process as a list of Process instances. If no parents are known, an empty list is returned.

    >>> import psutil
    >>> p = psutil.Process(5312)
    >>> p.parents()
    [psutil.Process(pid=4699, name='bash', started='09:06:44'),
     psutil.Process(pid=4689, name='gnome-terminal-server', started='09:06:44'),
     psutil.Process(pid=1, name='systemd', started='05:56:55')]
    

    Nothing fundamentally new here, since this is a convenience wrapper around Process.parent(), but it's still nice to have it built in. It pairs well with Process.children() when working with process trees. The idea was proposed by Ghislain Le Meur.

    Windows

    Certain Windows APIs that need to be dynamically loaded from DLLs are now loaded only once at startup, instead of on every function call. This makes some operations 50% to 100% faster; see benchmarks in PR-1422.

    Process.suspend() and Process.resume() previously iterated over all process threads via CreateToolhelp32Snapshot(), which was unorthodox and broke when the process had been suspended by Process Hacker. They now call the undocumented NtSuspendProcess() / NtResumeProcess() NT APIs, same as Process Hacker and Sysinternals tools. Discussed in #1379, implemented in PR-1435.

    SE DEBUG is a privilege bit set on the Python process at startup so psutil can query processes owned by other users (Administrator, Local System), meaning fewer AccessDenied exceptions for low-PID processes. The code setting it had presumably been broken for years and is now finally fixed in PR-1429.

    Removal of Process.memory_maps() on macOS

    Process.memory_maps() is gone on macOS (#1291). The underlying Apple API would randomly raise EINVAL or segfault the host process, and no amount of reverse-engineering produced a safe fix. So I removed it. This is covered in a separate post.

    Improved exceptions

    One problem that affected psutil maintenance over the years was receiving bug reports whose tracebacks did not indicate which syscall had actually failed. This was especially painful on Windows, where a single routine may invoke multiple Windows APIs. Now the OSError (or WindowsError) exception includes the syscall from which the error originated. See PR-1428.

    Other changes

    See the changelog.

  4. Removing Process.memory_maps() on macOS

    This is part of the psutil 5.6.0 release (see the full release notes).

    As of 5.6.0, Process.memory_maps() is no longer defined on macOS.

    The bug

    #1291: on macOS, Process.memory_maps() would either raise OSError: [Errno 22] Invalid argument or segfault the whole Python process! Both triggered from code as simple as psutil.Process().as_dict(), since Process.as_dict() iterates every attribute, and Process.memory_maps() is one of them.

    The root cause was inside Apple's undocumented proc_regionfilename() syscall. On some memory regions it returns EINVAL. On others it takes the process down. Which regions? Nobody figured out. Arnon Yaari (@wiggin15) did most of the investigation: he wrote a standalone C reproducer and walked me through what he'd tried.

    In PR-1436 I attempted a fix by reverse-engineering vmmap(1) but it didn't work. The fundamental problem is that vmmap is closed source and proc_regionfilename is undocumented. Neither my virtualized macOS (10.11.6) nor Travis CI (10.12.1) could reproduce the bug, which reproduced reliably only on 10.14.3.

    Why remove outright

    While removing the C code I noticed that the macOS unit test had been disabled long ago, presumably by me after recurring flaky Travis runs. Meaning that the method had been broken on some macOS versions far longer than the 2018 bug report suggested.

    Deprecating for a cycle didn't help either: raising AccessDenied breaks code that relied on a successful return, returning an empty list does the same silently, and leaving the method in place doesn't stop the segfault. Basically there was no sane solution, so since 5.6 is a major version I decided to just remove Process.memory_maps() for good.

    On macOS it never supported other processes anyway. Calling it on any PID other than the current one (or its children) raised AccessDenied, even as root.

    If someone finds a Mach API path that works, the method can return. Nobody has found one so far.

  5. AIX support

    After a long wait psutil finally supports a new exotic platform: AIX!

    Honestly I'm not sure how many AIX Python users are out there (probably not many), but here it is.

    For this we have to thank Arnon Yaari, who started working on the port a couple of years ago (#605). I was skeptical at first, because AIX is the only platform I can't virtualize and test on my laptop, so that made me a bit nervous. Arnon did a great job. The final PR-1123 is huge: it required a considerable amount of work on his part, and a review of more than 140 messages exchanged between us over about a month, during which I was travelling through China.

    The end result is very good: almost all original unit tests pass, and code quality is awesome, which (I must say) is fairly unusual for an external contribution like this. Kudos to you, Arnon! ;-)

    Other changes

    Besides AIX support, release 5.4.0 also includes a couple of important bug fixes for psutil.sensors_temperatures() and psutil.sensors_fans() on Linux, and a fix for a bug on macOS that could cause a segmentation fault when using Process.open_files(). The complete list of bug fixes is in the changelog.

    The future

    Looking ahead at other exotic, still-unsupported platforms, two contributions are worth mentioning: a (still incomplete) PR for Cygwin which looks promising (PR-998), and Mingw32 compiler support on Windows (PR-845).

    psutil is gradually reaching a point where adding new features is becoming rarer, so it's a good moment to welcome new platforms while the API is mature and stable.

    Future work along these lines could also include Android and (hopefully) iOS support. Now that would be really awesome to have.

    Stay tuned.

    Discussion

Social

Feeds