Blog posts for tags/community

  1. Wheels for free-threaded Python now available

    With the release of psutil 7.1.2, wheels for free-threaded Python are now available. This milestone was achieved largely through a community effort, as several internal refactorings to the C code were required to make it possible (see #2565). Many of these changes were contributed by Lysandros Nikolaou. Thanks to him for the effort and for bearing with me in code reviews! ;-)

    What is free-threaded Python?

    Free-threaded Python (available since Python 3.13) refers to Python builds that are compiled with the GIL (Global Interpreter Lock) disabled, allowing true parallel execution of Python bytecodes across multiple threads. This is particularly beneficial for CPU-bound applications, as it enables better utilization of multi-core processors.

    The state of free-threaded wheels

    According to Hugo van Kemenade's free-threaded wheels tracker, the adoption of free-threaded wheels among the top 360 most-downloaded PyPI packages with C extensions is still limited. Only 128 out of these 360 packages provide wheels compiled for free-threaded Python, meaning they can run on Python builds with the GIL disabled. This shows that, while progress has been made, most popular packages with C extensions still do not offer ready-made wheels for free-threaded Python.

    What it means for users

    When a library author provides a wheel, users can install a pre-compiled binary package without having to build it from source. This is especially important for packages with C extensions, like psutil, which is largely written in C. Such packages often have complex build requirements and require installing a C compiler. On Windows, that means installing Visual Studio or the Build Tools, which can take several gigabytes and a significant setup effort. Providing wheels spares users from this hassle, makes installation far simpler, and is effectively essential for the users of that package. You basically pip install psutil and you're done.

    What it means for library authors

    Currently, universal wheels for free-threaded Python do not exist. Each wheel must be built specifically for a Python version. Right now authors must create separate wheels for Python 3.13 and 3.14. Which means distributing a lot of files already:

    psutil-7.1.2-cp313-cp313t-macosx_10_13_x86_64.whl
    psutil-7.1.2-cp313-cp313t-macosx_11_0_arm64.whl
    psutil-7.1.2-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl
    psutil-7.1.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl
    psutil-7.1.2-cp313-cp313t-win_amd64.whl
    psutil-7.1.2-cp313-cp313t-win_arm64.whl
    psutil-7.1.2-cp314-cp314t-macosx_10_15_x86_64.whl
    psutil-7.1.2-cp314-cp314t-macosx_11_0_arm64.whl
    psutil-7.1.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl
    psutil-7.1.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl
    psutil-7.1.2-cp314-cp314t-win_amd64.whl
    psutil-7.1.2-cp314-cp314t-win_arm64.whl
    

    This also multiplies CI jobs and slows down the test matrix (see build.yml). A true universal wheel would greatly reduce this overhead, allowing a single wheel to support multiple Python versions and platforms. Hopefully, Python 3.15 will simplify this process. Two competing proposals, PEP 803 and PEP 809, aim to standardize wheel naming and metadata to allow producing a single wheel that covers multiple Python versions. That would drastically reduce distribution complexity for library authors, and it's fair to say it's essential for free-threaded CPython to truly succeed.

    How to install free-threaded psutil

    You can now install psutil for free-threaded Python directly via pip:

    pip install psutil --only-binary=:all:
    

    This ensures you get the pre-compiled wheels without triggering a source build.

    Discussion

  2. AIX support

    After a long wait psutil finally supports a new exotic platform: AIX!

    Honestly I'm not sure how many AIX Python users are out there (probably not many), but here it is.

    For this we have to thank Arnon Yaari, who started working on the port a couple of years ago (#605). I was skeptical at first, because AIX is the only platform I can't virtualize and test on my laptop, so that made me a bit nervous. Arnon did a great job. The final PR-1123 is huge: it required a considerable amount of work on his part, and a review of more than 140 messages exchanged between us over about a month, during which I was travelling through China.

    The end result is very good: almost all original unit tests pass, and code quality is awesome, which (I must say) is fairly unusual for an external contribution like this. Kudos to you, Arnon! ;-)

    Other changes

    Besides AIX support, release 5.4.0 also includes a couple of important bug fixes for psutil.sensors_temperatures() and psutil.sensors_fans() on Linux, and a fix for a bug on macOS that could cause a segmentation fault when using Process.open_files(). The complete list of bug fixes is in the changelog.

    The future

    Looking ahead at other exotic, still-unsupported platforms, two contributions are worth mentioning: a (still incomplete) PR for Cygwin which looks promising (PR-998), and Mingw32 compiler support on Windows (PR-845).

    psutil is gradually reaching a point where adding new features is becoming rarer, so it's a good moment to welcome new platforms while the API is mature and stable.

    Future work along these lines could also include Android and (hopefully) iOS support. Now that would be really awesome to have.

    Stay tuned.

    Discussion

  3. NetBSD support

    Roughly two months have passed since I last announced that psutil added support for OpenBSD. Today I'm happy to announce that it's the turn of NetBSD! This was contributed by Thomas Klausner, Ryo Onodera and myself in PR-557.

    Differences with FreeBSD (and OpenBSD)

    The NetBSD implementation has limitations similar to the ones I encountered with OpenBSD. Again, FreeBSD remains the BSD variant with the best support in terms of kernel functionality.

    • Process.memory_maps() is not implemented. The kernel provides the necessary pieces, but I haven't done this yet (hopefully later).
    • Process.num_ctx_switches()'s involuntary field is always 0. The kinfo_proc() syscall provides this info, but it's always set to 0.
    • Process.cpu_affinity() (get and set) is not supported.
    • psutil.cpu_count(logical=False) always returns None.

    As for the rest: it is all there. All memory, disk, network and process APIs are fully supported and functioning.

    Other enhancements available in this psutil release

    Besides NetBSD support, this release has a couple of interesting enhancements:

    • #708: [Linux] psutil.net_connections() and Process.connections() can be up to 3x faster when there are many connections.
    • #718: psutil.process_iter() is now thread safe.

    You can read the rest in the changelog, as usual.

    Move to Prague

    As a personal note I'd like to add that I'm currently in Prague (Czech Republic) and I'm thinking about moving down here for a while.

    Discussion

  4. Real process memory and environ in Python

    psutil 4.0.0 is out, with some interesting news about process memory metrics. I'll get straight to the point and describe what's new.

    "Real" process memory info

    Determining how much memory a process really uses is not an easy matter (see this and this). RSS (Resident Set Size), which most people rely on, is misleading because it includes both memory unique to the process and memory shared with others. What's more interesting for profiling is the memory that would be freed if the process were terminated right now. In the Linux world this is called USS (Unique Set Size), the major feature introduced in psutil 4.0.0 (not only for Linux but also for Windows and macOS).

    USS memory

    The USS (Unique Set Size) is the memory unique to a process, that would be freed if the process were terminated right now. On Linux it can be determined by parsing the "private" blocks in /proc/PID/smaps. The Firefox team pushed this further and got it working on macOS and Windows too.

    >>> psutil.Process().memory_full_info()
    pfullmem(rss=101990, vms=521888, shared=38804, text=28200, lib=0, data=59672, dirty=0, uss=81623, pss=91788, swap=0)
    

    PSS and swap

    On Linux there are two additional metrics that can also be determined via /proc/PID/smaps: PSS and swap.

    pss, aka "Proportional Set Size", represents the amount of memory shared with other processes, accounted so that the amount is divided evenly between the processes that share it. I.e. if a process has 10 MBs all to itself (USS) and 10 MBs shared with another process, its PSS will be 15 MBs.

    swap is simply the amount of memory that has been swapped out to disk. With Process.memory_full_info() it is possible to implement a tool like procsmem.py, similar to smem on Linux, which provides a list of processes sorted by uss. It's interesting to see how rss differs from uss:

    ~/svn/psutil$ ./scripts/procsmem.py
    PID     User    Cmdline                            USS     PSS    Swap     RSS
    ==============================================================================
    ...
    3986    giampao /usr/bin/python3 /usr/bin/indi   15.3M   16.6M      0B   25.6M
    3906    giampao /usr/lib/ibus/ibus-ui-gtk3       17.6M   18.1M      0B   26.7M
    3991    giampao python /usr/bin/hp-systray -x    19.0M   23.3M      0B   40.7M
    3830    giampao /usr/bin/ibus-daemon --daemoni   19.0M   19.0M      0B   21.4M
    20529   giampao /opt/sublime_text/plugin_host    19.9M   20.1M      0B   22.0M
    3990    giampao nautilus -n                      20.6M   29.9M      0B   50.2M
    3898    giampao /usr/lib/unity/unity-panel-ser   27.1M   27.9M      0B   37.7M
    4176    giampao /usr/lib/evolution/evolution-c   35.7M   36.2M      0B   41.5M
    20712   giampao /usr/bin/python -B /home/giamp   45.6M   45.9M      0B   49.4M
    3880    giampao /usr/lib/x86_64-linux-gnu/hud/   51.6M   52.7M      0B   61.3M
    20513   giampao /opt/sublime_text/sublime_text   65.8M   73.0M      0B   87.9M
    3976    giampao compiz                          115.0M  117.0M      0B  130.9M
    32486   giampao skype                           145.1M  147.5M      0B  149.6M
    

    Implementation

    To get these values (uss, pss and swap) we need to walk the whole process address space. This usually requires higher privileges and is considerably slower than Process.memory_info(), which is probably why tools like ps and top show RSS/VMS instead of USS. A big thanks goes to the Mozilla team for figuring this out on Windows and macOS, and to Eric Rahm who put the psutil PRs together (see PR-744, PR-745 and PR-746). If you don't use Python and want to port the code to another language, here are the interesting parts:

    Memory type percent

    After reorganizing the process memory APIs (PR-744), I added a new memtype parameter to Process.memory_percent(). You can now compare a specific memory type (not only RSS) against the total physical memory. E.g.

    >>> psutil.Process().memory_percent(memtype='pss')
    0.06877466326787016
    

    Process environ

    The second biggest improvement in psutil 4.0.0 is the ability to read a process's environment variables. This opens up interesting possibilities for process recognition and monitoring. For instance, you can start a process with a custom environment variable, then iterate over all processes to find the one of interest:

    import psutil
    for p in psutil.process_iter():
        try:
            env = p.environ()
        except psutil.Error:
            pass
        else:
            if 'MYAPP' in env:
                ...
    

    Process environ was a long-standing issue (#52, from 2009) that I gave up on because the Windows implementation only worked for the current process. Frank Benkstein solved that (PR-747), and it now works on Linux, Windows and macOS for all processes (you may still hit AccessDenied for processes owned by another user):

    >>> import psutil
    >>> from pprint import pprint as pp
    >>> pp(psutil.Process().environ())
    {...
     'CLUTTER_IM_MODULE': 'xim',
     'COLORTERM': 'gnome-terminal',
     'COMPIZ_BIN_PATH': '/usr/bin/',
     'HOME': '/home/giampaolo',
     'PWD': '/home/giampaolo/svn/psutil',
      }
    >>>
    

    Note that the resulting dict usually doesn't reflect changes made after the process started (e.g. os.environ['MYAPP'] = '1'). Again, for anyone porting this to other languages, here are the interesting parts:

    Extended disk IO stats

    psutil.disk_io_counters() now reports additional metrics on Linux and FreeBSD:

    • busy_time: the time spent doing actual I/Os (in milliseconds).
    • read_merged_count and write_merged_count (Linux only): the number of merged reads and writes (see the iostats doc).

    These give a better picture of actual disk utilization (#756), similar to the iostat command on Linux.

    OS constants

    Given the growing number of platform-specific metrics, I added a set of constants to tell which platform you're on: psutil.LINUX, psutil.WINDOWS, etc.

    Other fixes

    The complete list of changes is available in the changelog.

    Porting code

    Since 4.0.0 is a major version, I took the chance to (lightly) change / break some APIs.

    • Process.memory_info() no longer returns just an (rss, vms) namedtuple. It returns a variable-length namedtuple that varies by platform (rss and vms are always present, even on Windows). Essentially the same result as the old Process.memory_info_ex(). This shouldn't break your code unless you were doing rss, vms = p.memory_info().
    • Process.memory_info_ex() is deprecated. It still works as an alias for Process.memory_info(), issuing a DeprecationWarning.
    • psutil.disk_io_counters() on NetBSD and OpenBSD no longer returns write_count and read_count because the kernel doesn't provide them (we were returning the busy time instead). Should be a small issue given NetBSD and OpenBSD support is very recent.

    Discussion

  5. OpenBSD support

    Starting from version 3.3.0 (released just now) psutil officially supports OpenBSD. This was contributed by Landry Breuil and myself in PR-615.

    Differences with FreeBSD

    As expected, the OpenBSD implementation is very similar to FreeBSD's, so I merged most of it into a single C file (_psutil_bsd.c), with 2 separate files (freebsd.c, openbsd.c) for the parts that differ. Here are the functional differences with FreeBSD:

    • Process.memory_maps() is not implemented. The kernel provides the necessary pieces but I haven't done this yet (hopefully later).
    • Process.num_ctx_switches()'s involuntary field is always 0. kinfo_proc provides this info but it is always set to 0.
    • Process.cpu_affinity() (get and set) is not supported.
    • Process.exe() is determined by inspecting the command line, so it may not always be available (returns None).
    • psutil.swap_memory() sin and sout (swap in and swap out) values are not available, and are therefore always set to 0.
    • psutil.cpu_count(logical=False) always returns None.

    As with FreeBSD, Process.open_files() can't return file paths (FreeBSD can sometimes). Otherwise everything is there and I'm satisfied with the result.

    Considerations about BSD platforms

    psutil has supported FreeBSD since the beginning (year 2009). At the time, it made sense to prefer it as the preferred BSD variant as it's the most popular.

    Compared to FreeBSD, OpenBSD appears to be more "minimal", both in terms of kernel facilities and the number of CLI tools available. One thing I particularly appreciate about FreeBSD is that the source code for all CLI tools is available under /usr/src, which was a big help when implementing psutil APIs.

    OpenBSD source code is also available, but it uses CVS and I am not sure it includes the source code for all CLI tools.

    There are still two more BSD variants worth supporting: NetBSD and DragonFlyBSD (in this order). About a year ago, someone provided a patch adding basic NetBSD support, so that will likely happen sooner or later.

    Other enhancements available in this release

    The only other enhancement is #558, which allows specifying a different location for the /proc filesystem on Linux.

    Discussion

Social

Feed