Blog posts for tags/travel

  1. System load average on Windows in Python

    New psutil 5.6.2 release implements an emulation of os.getloadavg() on Windows which was kindly contributed by Ammar Askar who originally implemented it for cPython's test suite. This idea has been floating around for quite a while. The first proposal dates back to 2010, when psutil was still hosted on Google Code, and it popped up multiple times throughout the years. There was/is a bunch of info on internet mentioning the bits with which it's theoretically possible to do this (the so called System Processor Queue Length), but I couldn't find any real implementation. A Google search tells there is quite some demand for this, but very few tools out there providing this natively (the only one I could find is this sFlowTrend tool and Zabbix), so I'm very happy this finally landed into psutil / Python.

    Other improvements and bugfixes in psutil 5.6.2

    The full list is here but I would like to mention a couple:

    • 1476: the possibility to set process' high I/O priority on Windows
    • 1458: colorized test output. I admit nobody will use this directly but it's very cool and I'm porting it to a bunch of other projects I work on (e.g. pyftpdlib). Also, perhaps this could be a good candidate for a small module to put on PYPI which can also include some functionalities taken from pytest and which I'm gradually re-implementing in unittest module amongst which:
      • 1478: re-running failed tests
      • display test timings/durations: this is something I'm contributing to cPython, see BPO-4080 and and PR-12271

    About me

    I'm currently in China (Shenzhen) for a mix of vacation and work, and I will likely take a break from Open Source for a while (likely 2.5 months, during which I will also go to Philippines and Japan - I love Asia ;-)).


  2. psutil 5.3.0 and full unicode support

    psutil 5.3.0 is finally out. This release is a major one, as it includes tons of improvements and bugfixes, probably like no other previous release. It is interesting to notice how huge the diff between 5.2.2 and 5.3.0 is. This is due to the fact that I've been travelling quite a lot this year, so I kept postponing it. It may sound weird but I consider publishing a new release and write a blog post about more stressful than working on the release itself. =). Anyway, here goes.

    Full Unicode support

    This is the biggest change. In order to achieve this I had to refactor all functions and internals either returning or accepting a string. Incidentally this helped me having a better understanding of how Unicode works and how it should be handled at the C level in terms of differences between Python 2 and 3. Issue #1040 includes all the reasonings I've been through and potentially serves as a documentation for people who are facing a similar task (handling Unicode in C for both Python 2 and 3). Up until version 5.2.x psutil functions returning a string had different problems as they could:

    • raise decoding error on Python 3 in case of non-ASCII string
    • return unicode instead of str (Python 2)
    • return incorrect / invalid encoded data in case of non-ASCII string

    5.3.0 fixes these three issues and consolidates the correct handling of Unicode strings. On Windows this was achieved by using Unicode-specific Windows APIs. The notes below describe how Unicode and strings in general are handled internally by psutil and they apply to any API returning a string such as Process.exe or Process.cwd, including non-filesystem related methods such as Process.username or WindowsService.description:

    • all strings are encoded by using the OS filesystem encoding (sys.getfilesystemencoding()) which varies depending on the platform (e.g. "UTF-8" on OSX, "mbcs" on Win)
    • no API call is supposed to crash with UnicodeDecodeError
    • instead, in case of badly encoded data returned by the OS, the following error handlers are used to replace the corrupted characters in the string:
    • on Python 2 all APIs return bytes (str type), never unicode
    • on Python 2 you can go back to unicode by doing:
    >>> unicode(proc.exe(), sys.getdefaultencoding(), errors="replace")

    Improved process_iter() function

    process_iter() accepts two new parameters in order to invoke Process.as_dict() internally: "attrs" and "ad_value". With this you can iterate over all processes in one shot without having to catch NoSuchProcess explicitly. Before:

    >>> import psutil
    >>> for proc in psutil.process_iter():
    ...     try:
    ...         pinfo = proc.as_dict(attrs=['pid', 'name'])
    ...     except psutil.NoSuchProcess:
    ...         pass
    ...     else:
    ...         print(pinfo)
    {'pid': 1, 'name': 'systemd'}
    {'pid': 2, 'name': 'kthreadd'}
    {'pid': 3, 'name': 'ksoftirqd/0'}


    >>> import psutil
    >>> for proc in psutil.process_iter(attrs=['pid', 'name']):
    ...     print(
    {'pid': 1, 'name': 'systemd'}
    {'pid': 2, 'name': 'kthreadd'}
    {'pid': 3, 'name': 'ksoftirqd/0'}

    This improves expressiveness as it makes it possible to use nice list/dict comprehensions. Here's some examples.

    Processes having "python" in their name:

    >>> from pprint import pprint as pp
    >>> pp([ for p in psutil.process_iter(attrs=['pid', 'name']) if 'python' in['name']])
    [{'name': 'python3', 'pid': 21947},
    {'name': 'python', 'pid': 23835}]

    Processes owned by user:

    >>> import getpass
    >>> pp([(,['name']) for p in psutil.process_iter(attrs=['name', 'username']) if['username'] == getpass.getuser()])
    (16832, 'bash'),
    (19772, 'ssh'),
    (20492, 'python')]

    Processes actively running:

    >>> pp([(, for p in psutil.process_iter(attrs=['name', 'status']) if['status'] == psutil.STATUS_RUNNING])
    [(1150, {'name': 'Xorg', 'status': 'running'}),
    (1776, {'name': 'unity-panel-service', 'status': 'running'}),
    (20492, {'name': 'python', 'status': 'running'})]

    Automatic overflow handling of numbers

    On very busy or long-lived system systems numbers returned by disk_io_counters() and net_io_counters() functions may wrap (restart from zero). Up to version 5.2.x you had to take this into account while now this is automatically handled by psutil (see: #802). If a "counter" restarts from 0 psutil will add the value from the previous call for you so that numbers will never decrease. This is crucial for applications monitoring disk or network I/O in real time. Old behavior can be resumed by passing nowrap=True argument.

    SunOS Process environ()

    Process.environ() is now available also on SunOS (see #1091).

    Other improvements and bug fixes

    Amongst others, here's a couple of important bug fixes I'd like to mention:

    • #1044: on OSX different Process methods could incorrectly raise AccessDenied for zombie processes. This was due to poor proc_pidpath OSX API.
    • #1094: on Windows, pid_exists() may lie due to the poor OpenProcess Windows API which can return a handle even when a process PID no longer exists. This had repercussions for many Process methods such as cmdline(), environ(), cwd(), connections() and others which could have unpredictable behaviors such as returning empty data or erroneously raise NoSuchProcess exceptions. For the same reason (broken OpenProcess API), processes could unexpectedly stick around after using terminate() and wait().

    BSD systems also received some love (NetBSD and OpenBSD in particular). Different memory leaks were fixed and functions returning connected sockets were partially rewritten. The full list of enhancement and bug fixes can be seen here.

    About me

    I would like to spend a couple more words about my current situation. Last year (2016) I relocated to Prague and remote worked from there the whole year (it's been cool - great city!). This year I have mainly been resting in Turin (Italy) due to some health issues and travelling across Asia once I started to recover. I am currently in Shenzhen, China, and unless the current situation with North Korea gets worse I'm planning to continue my trip until November and visit Taiwan, South Korea and Japan. Once I'm finished the plan is to briefly return to Turin (Italy) and finally return to Prague. By then I will probably be looking for a new (remote) gig again, so if you have anything for me by November feel free to send me a message. ;-)

  3. psutil 3.0, aka how I reimplemented ifconfig in Python

    Here we are. It's been a long time since my last blog post and my last psutil release. The reason? I've been travelling! I mean... a lot. I've spent 3 months in Berlin, 3 weeks in Japan and 2 months in New York City. While I was there I finally had the chance to meet my friend Jay Loden in person. Jay and I originally started working on psutil together 7 years ago.

    Back then I didn't know any C (and I still am a terrible C developer) so he's been crucial to develop the initial psutil skeleton including OSX and Windows support. I'm back home now (but not for long ;-)), so I finally have some time to write this blog post and tell you about the new psutil release. Let's see what happened.


    In a few words, we're now able to list network interface addresses similarly to "ifconfig" command on UNIX:

    >>> import psutil
    >>> from pprint import pprint
    >>> pprint(psutil.net_if_addrs())
    {'ethernet0': [snic(family=<AddressFamily.AF_INET: 2>,
                   snic(family=<AddressFamily.AF_PACKET: 17>,
     'localhost': [snic(family=<AddressFamily.AF_INET: 2>,
                   snic(family=<AddressFamily.AF_PACKET: 17>,

    This is limited to AF_INET (IPv4), AF_INET6 (IPv6) and AF_LINK (Ethernet) address families. If you want something more poweful (e.g. AF_BLUETOOTH) you can take a look at netifaces extension. And here's the code which does these tricks on POSIX and Windows:

    Also, here's some doc.


    This will return a bunch of information about network interface cards:

    >>> import psutil
    >>> from pprint import pprint
    >>> pprint(psutil.net_if_stats())
    {'ethernet'0: snicstats(isup=True,
                            duplex=<NicDuplex.NIC_DUPLEX_FULL: 2>,
     'localhost': snicstats(isup=True,
                            duplex=<NicDuplex.NIC_DUPLEX_UNKNOWN: 0>,

    Again, here's the code for each platform:

    ...and the doc.


    Enums are a nice new feature introduced in Python 3.4. Very briefly (or at least, this is what I appreciate the most about them), they help you write an API with human-readable constants. If you use Python 2 you'll see something like this:

    >>> import psutil
    >>> psutil.IOPRIO_CLASS_IDLE

    On Python 3.4 you'll see a more informative:

    >>> import psutil
    >>> psutil.IOPRIO_CLASS_IDLE
    <IOPriority.IOPRIO_CLASS_IDLE: 3>

    They are backward compatible, meaning if you're sending serialized data produced with psutil through the network you can safely use comparison operators and so on. The psutil APIs returning enums (on Python >=3.4) are:

    • psutil.net_connections() (the address families):
    • psutil.Process.connections() (same as above)
    • psutil.net_if_stats() (all NIC_DUPLEX_* constants)
    • psutil.Process.nice() on Windows (for all the *_PRIORITY_CLASS constants)
    • psutil.Process.ionice() on Linux (for all the IOPRIO_CLASS_* constants)

    All the other existing constants remained plain strings (STATUS_*) or integers (CONN_*).

    Zombie processes

    This is a big one. The full story is here but basically the support for zombie processes on UNIX was broken (except on Linux, and Windows doesn't have zombie processes). Up until psutil 2.X we could instantiate a zombie process:

    >>> pid = create_zombie()
    >>> p = psutil.Process(pid)

    ...but every time we queried it we got a NoSuchProcess exception:

      File "psutil/", line 374, in _init
        raise NoSuchProcess(pid, None, msg)
    psutil.NoSuchProcess: no process found with pid 123

    That was misleading though because the PID technically still existed:

    >>> psutil.pid_exists(

    Furthermore, depending on what platform you were on, certain process stats could still be queried (instead of raising NoSuchProcess):

    >>> psutil.cmdline()

    Also process_iter() did not return zombie processes at all. This was probably the worst aspect because being able to identify them is an important use case, as they signal an issue with process: if a parent process spawns a child, terminates it (via kill()), but doesn't wait() for it it will create a zombie. Long story short, the way this changed in psutil 3.0 is that:

    • we now have a new ZombieProcess exception, raised every time we're not able to query a process because it's a zombie
    • it is raised instead of NoSuchProcess (which was incorrect and misleading)
    • it is still backward compatible (meaning you won't have to change your old code) because it inherits from NoSuchProcess
    • process_iter() finally works, meaning you can safely identify zombie processes like this:
    import psutil
    zombies = []
    for p in psutil.process_iter():
            if p.status() == psutil.STATUS_ZOMBIE:
        except NoSuchProcess:

    Removal of deprecated APIs

    This is another big one, probably the biggest. In a previous blog post I already talked about deprecated APIs. What I did back then (January 2014) was to rename and officially deprecate different APIs and provide aliases for them so that people wouldn't yell at me because I broke their existent code. The most interesting deprecation was certainly the one affecting module constants and the hack which was used in order to provide "module properties". With this new release I decided to get rid of all those aliases. I'm sure this will cause problems but hey! This is a new major release, right? =). Plus the amount of crap which was removed is impressive (see the commit). Here's the old aliases which are now gone for good (or bad, depending on how much headache they will cause you):

    Removed module functions and constants

    Already deprecated name New name
    psutil.BOOT_TIME() psutil.boot_time()
    psutil.NUM_CPUS() psutil.cpu_count()
    psutil.TOTAL_PHYMEM() psutil.virtual_memory().total
    psutil.avail_phymem() psutil.virtual_memory().free
    psutil.avail_virtmem() psutil.swap_memory().free
    psutil.cached_phymem() psutil.virtual_memory().cached
    psutil.get_pid_list() psutil.pids().cached
    psutil.get_users() psutil.users()
    psutil.network_io_counters() psutil.net_io_counters()
    psutil.phymem_buffers() psutil.virtual_memory().buffers
    psutil.phymem_usage() psutil.virtual_memory()
    psutil.total_virtmem() psutil.swap_memory().total
    psutil.used_virtmem() psutil.swap_memory().used
    psutil.used_phymem() psutil.virtual_memory().used
    psutil.virtmem_usage() psutil.swap_memory()

    Process methods (assuming p = psutil.Process()):

    Already deprecated name New name
    p.get_children() p.children()
    p.get_connections() p.connections()
    p.get_cpu_affinity() p.cpu_affinity()
    p.get_cpu_percent() p.cpu_percent()
    p.get_cpu_times() p.cpu_times()
    p.get_io_counters() p.io_counters()
    p.get_ionice() p.ionice()
    p.get_memory_info() p.memory_info()
    p.get_ext_memory_info() p.memory_info_ex()
    p.get_memory_maps() p.memory_maps()
    p.get_memory_percent() p.memory_percent()
    p.get_nice() p.nice()
    p.get_num_ctx_switches() p.num_ctx_switches()
    p.get_num_fds() p.num_fds()
    p.get_num_threads() p.num_threads()
    p.get_open_files() p.open_files()
    p.get_rlimit() p.rlimit()
    p.get_threads() p.threads()
    p.getcwd() p.cwd()
    p.set_cpu_affinity() p.cpu_affinity()
    p.set_ionice() p.ionice()
    p.set_nice() p.nice()
    p.set_rlimit() p.rlimit()

    If your code suddenly breaks with AttributeError after you upgraded psutil it means you were using one of those deprecated aliases. In that case just take a look at the table above and rename stuff in accordance.

    Bug fixes

    I fixed a lot of stuff (full list here), but here's the list of things which I think are worth mentioning:

    • #512: [FreeBSD] fix segfault in net_connections().
    • #593: [FreeBSD] Process.memory_maps() segfaults.
    • #606: Process.parent() may swallow NoSuchProcess exceptions.
    • #614: [Linux]: cpu_count(logical=False) return the number of physical CPUs instead of physical cores.
    • #628: [Linux] truncates process name in case it contains spaces or parentheses.

    Ease of development

    These are not enhancements you will directly benefit from but I put some effort into making my life easier every time I work on psutil.

    • I care about psutil code being fully PEP8 compliant so I added a pre-commit GIT hook which runs flake8 on every commit and rejects it if the coding style is not compliant. The way I install this is via make install-git-hooks.
    • I added a make install-dev-deps command which installs all deps and stuff which is useful for testing (ipdb, coverage, etc).
    • A new make coverage command which runs coverage. With this I discovered some of parts in the code which weren't covered by tests and I fixed that.
    • I started using tox to easily test psutil against all supported Python versions (from 2.6 to 3.4) in one shot.
    • I reorganized tests so that now they can be easily executed with py.test and nose (before, only unittest runner was fully supported)

    Final words

    I must say I'm pretty satisfied with how psutil is going and the satisfaction I still get every time I work on it. Right now it gets almost 800.000 download a month, which is pretty great for a Python library. As of right now I consider psutil almost "completed" in terms of features, meaning I'm basically running out of ideas on what I should add next (see TODO). From now on the future development will probably focus on adding support for more exotic platforms (OpenBSD, NetBSD, Android). There also have been some discussions on python-ideas mailing list about including psutil into Python stdlib but, assuming that will ever happen, it's still far away in the future as it would require a lot of time which I currently don't have. That should be all. I hope you will all enjoy this new release.

  4. psutil 2.1.2 and Python wheels

    psutil 2.1.2 is out. This release has been cooking for a while now, and that's because I've been travelling for the past 3 months between Spain, Japan and Germany. Hopefully I will be staying in Berlin for a while now, so I will have more time to dedicate to the project. The main new "feature" of this release is that other than the exe files, Windows users can now also benefit of Python wheels (full story is here) which are available on PYPI. Frankly I don't know much about the new wheels packaging system but long story short is that Windows users can now install psutil via pip and therefore also include it as a dependency into requirements.txt. Other than this 2.1.2 can basically be considered a bug-fix release, including some important fixes amongst which:

    • #506: restored Python 2.4 compatibility
    • #340: Process.get_open_files() no longer hangs on Windows (this was a very old and high-priority issue)
    • #501: disk_io_counters() may return negative values on Windows
    • #504: (Linux) couldn't build RPM packages via

    The list of all fixes can be found here. For the next release I plan to drop support for Python 2.4 and 2.5 and hopefully network interfaces information similarly to ifconfig.