Blog posts for tags/psutil

  1. psutil 3.0, aka how I reimplemented ifconfig in Python

    Here we are. It's been a long time since my last blog post and my last psutil release. The reason? I've been travelling! I mean... a lot. I've spent 3 months in Berlin, 3 weeks in Japan and 2 months in New York City. While I was there I finally had the chance to meet my friend Jay Loden in person. Jay and I originally started working on psutil together 7 years ago.

    Back then I didn't know any C (and I still am a terrible C developer) so he's been crucial to develop the initial psutil skeleton including OSX and Windows support. I'm back home now (but not for long ;-)), so I finally have some time to write this blog post and tell you about the new psutil release. Let's see what happened.

    net_if_addrs()

    In a few words, we're now able to list network interface addresses similarly to "ifconfig" command on UNIX:

    >>> import psutil
    >>> from pprint import pprint
    >>> pprint(psutil.net_if_addrs())
    {'ethernet0': [snic(family=<AddressFamily.AF_INET: 2>,
                        address='10.0.0.4',
                        netmask='255.0.0.0',
                        broadcast='10.255.255.255'),
                   snic(family=<AddressFamily.AF_PACKET: 17>,
                        address='9c:eb:e8:0b:05:1f',
                        netmask=None,
                        broadcast='ff:ff:ff:ff:ff:ff')],
     'localhost': [snic(family=<AddressFamily.AF_INET: 2>,
                        address='127.0.0.1',
                        netmask='255.0.0.0',
                        broadcast='127.0.0.1'),
                   snic(family=<AddressFamily.AF_PACKET: 17>,
                        address='00:00:00:00:00:00',
                        netmask=None,
                        broadcast='00:00:00:00:00:00')]}
    

    This is limited to AF_INET (IPv4), AF_INET6 (IPv6) and AF_LINK (Ethernet) address families. If you want something more poweful (e.g. AF_BLUETOOTH) you can take a look at netifaces extension. And here's the code which does these tricks on POSIX and Windows:

    Also, here's some doc.

    net_if_stats()

    This will return a bunch of information about network interface cards:

    >>> import psutil
    >>> from pprint import pprint
    >>> pprint(psutil.net_if_stats())
    {'ethernet'0: snicstats(isup=True,
                            duplex=<NicDuplex.NIC_DUPLEX_FULL: 2>,
                            speed=100,
                            mtu=1500),
     'localhost': snicstats(isup=True,
                            duplex=<NicDuplex.NIC_DUPLEX_UNKNOWN: 0>,
                            speed=0,
                            mtu=65536)}
    

    Again, here's the code for each platform:

    ...and the doc.

    Enums

    Enums are a nice new feature introduced in Python 3.4. Very briefly (or at least, this is what I appreciate the most about them), they help you write an API with human-readable constants. If you use Python 2 you'll see something like this:

    >>> import psutil
    >>> psutil.IOPRIO_CLASS_IDLE
    3
    

    On Python 3.4 you'll see a more informative:

    >>> import psutil
    >>> psutil.IOPRIO_CLASS_IDLE
    <IOPriority.IOPRIO_CLASS_IDLE: 3>
    

    They are backward compatible, meaning if you're sending serialized data produced with psutil through the network you can safely use comparison operators and so on. The psutil APIs returning enums (on Python >=3.4) are:

    • psutil.net_connections() (the address families):
    • psutil.Process.connections() (same as above)
    • psutil.net_if_stats() (all NIC_DUPLEX_* constants)
    • psutil.Process.nice() on Windows (for all the *_PRIORITY_CLASS constants)
    • psutil.Process.ionice() on Linux (for all the IOPRIO_CLASS_* constants)

    All the other existing constants remained plain strings (STATUS_*) or integers (CONN_*).

    Zombie processes

    This is a big one. The full story is here but basically the support for zombie processes on UNIX was broken (except on Linux, and Windows doesn't have zombie processes). Up until psutil 2.X we could instantiate a zombie process:

    >>> pid = create_zombie()
    >>> p = psutil.Process(pid)
    

    ...but every time we queried it we got a NoSuchProcess exception:

    >>> psutil.name()
      File "psutil/__init__.py", line 374, in _init
        raise NoSuchProcess(pid, None, msg)
    psutil.NoSuchProcess: no process found with pid 123
    

    That was misleading though because the PID technically still existed:

    >>> psutil.pid_exists(p.pid)
    True
    

    Furthermore, depending on what platform you were on, certain process stats could still be queried (instead of raising NoSuchProcess):

    >>> psutil.cmdline()
    ['python']
    

    Also process_iter() did not return zombie processes at all. This was probably the worst aspect because being able to identify them is an important use case, as they signal an issue with process: if a parent process spawns a child, terminates it (via kill()), but doesn't wait() for it it will create a zombie. Long story short, the way this changed in psutil 3.0 is that:

    • we now have a new ZombieProcess exception, raised every time we're not able to query a process because it's a zombie
    • it is raised instead of NoSuchProcess (which was incorrect and misleading)
    • it is still backward compatible (meaning you won't have to change your old code) because it inherits from NoSuchProcess
    • process_iter() finally works, meaning you can safely identify zombie processes like this:
    import psutil
    zombies = []
    for p in psutil.process_iter():
        try:
            if p.status() == psutil.STATUS_ZOMBIE:
                zombies.append(p)
        except NoSuchProcess:
            pass
    

    Removal of deprecated APIs

    This is another big one, probably the biggest. In a previous blog post I already talked about deprecated APIs. What I did back then (January 2014) was to rename and officially deprecate different APIs and provide aliases for them so that people wouldn't yell at me because I broke their existent code. The most interesting deprecation was certainly the one affecting module constants and the hack which was used in order to provide "module properties". With this new release I decided to get rid of all those aliases. I'm sure this will cause problems but hey! This is a new major release, right? =). Plus the amount of crap which was removed is impressive (see the commit). Here's the old aliases which are now gone for good (or bad, depending on how much headache they will cause you):

    Removed module functions and constants

    Already deprecated name New name
    psutil.BOOT_TIME() psutil.boot_time()
    psutil.NUM_CPUS() psutil.cpu_count()
    psutil.TOTAL_PHYMEM() psutil.virtual_memory().total
    psutil.avail_phymem() psutil.virtual_memory().free
    psutil.avail_virtmem() psutil.swap_memory().free
    psutil.cached_phymem() psutil.virtual_memory().cached
    psutil.get_pid_list() psutil.pids().cached
    psutil.get_process_list()  
    psutil.get_users() psutil.users()
    psutil.network_io_counters() psutil.net_io_counters()
    psutil.phymem_buffers() psutil.virtual_memory().buffers
    psutil.phymem_usage() psutil.virtual_memory()
    psutil.total_virtmem() psutil.swap_memory().total
    psutil.used_virtmem() psutil.swap_memory().used
    psutil.used_phymem() psutil.virtual_memory().used
    psutil.virtmem_usage() psutil.swap_memory()

    Process methods (assuming p = psutil.Process()):

    Already deprecated name New name
    p.get_children() p.children()
    p.get_connections() p.connections()
    p.get_cpu_affinity() p.cpu_affinity()
    p.get_cpu_percent() p.cpu_percent()
    p.get_cpu_times() p.cpu_times()
    p.get_io_counters() p.io_counters()
    p.get_ionice() p.ionice()
    p.get_memory_info() p.memory_info()
    p.get_ext_memory_info() p.memory_info_ex()
    p.get_memory_maps() p.memory_maps()
    p.get_memory_percent() p.memory_percent()
    p.get_nice() p.nice()
    p.get_num_ctx_switches() p.num_ctx_switches()
    p.get_num_fds() p.num_fds()
    p.get_num_threads() p.num_threads()
    p.get_open_files() p.open_files()
    p.get_rlimit() p.rlimit()
    p.get_threads() p.threads()
    p.getcwd() p.cwd()
    p.set_cpu_affinity() p.cpu_affinity()
    p.set_ionice() p.ionice()
    p.set_nice() p.nice()
    p.set_rlimit() p.rlimit()

    If your code suddenly breaks with AttributeError after you upgraded psutil it means you were using one of those deprecated aliases. In that case just take a look at the table above and rename stuff in accordance.

    Bug fixes

    I fixed a lot of stuff (full list here), but here's the list of things which I think are worth mentioning:

    • #512: [FreeBSD] fix segfault in net_connections().
    • #593: [FreeBSD] Process.memory_maps() segfaults.
    • #606: Process.parent() may swallow NoSuchProcess exceptions.
    • #614: [Linux]: cpu_count(logical=False) return the number of physical CPUs instead of physical cores.
    • #628: [Linux] Process.name() truncates process name in case it contains spaces or parentheses.

    Ease of development

    These are not enhancements you will directly benefit from but I put some effort into making my life easier every time I work on psutil.

    • I care about psutil code being fully PEP8 compliant so I added a pre-commit GIT hook which runs flake8 on every commit and rejects it if the coding style is not compliant. The way I install this is via make install-git-hooks.
    • I added a make install-dev-deps command which installs all deps and stuff which is useful for testing (ipdb, coverage, etc).
    • A new make coverage command which runs coverage. With this I discovered some of parts in the code which weren't covered by tests and I fixed that.
    • I started using tox to easily test psutil against all supported Python versions (from 2.6 to 3.4) in one shot.
    • I reorganized tests so that now they can be easily executed with py.test and nose (before, only unittest runner was fully supported)

    Final words

    I must say I'm pretty satisfied with how psutil is going and the satisfaction I still get every time I work on it. Right now it gets almost 800.000 download a month, which is pretty great for a Python library. As of right now I consider psutil almost "completed" in terms of features, meaning I'm basically running out of ideas on what I should add next (see TODO). From now on the future development will probably focus on adding support for more exotic platforms (OpenBSD, NetBSD, Android). There also have been some discussions on python-ideas mailing list about including psutil into Python stdlib but, assuming that will ever happen, it's still far away in the future as it would require a lot of time which I currently don't have. That should be all. I hope you will all enjoy this new release.

  2. psutil 2.1.2 and Python wheels

    psutil 2.1.2 is out. This release has been cooking for a while now, and that's because I've been travelling for the past 3 months between Spain, Japan and Germany. Hopefully I will be staying in Berlin for a while now, so I will have more time to dedicate to the project. The main new "feature" of this release is that other than the exe files, Windows users can now also benefit of Python wheels (full story is here) which are available on PYPI. Frankly I don't know much about the new wheels packaging system but long story short is that Windows users can now install psutil via pip and therefore also include it as a dependency into requirements.txt. Other than this 2.1.2 can basically be considered a bug-fix release, including some important fixes amongst which:

    • #506: restored Python 2.4 compatibility
    • #340: Process.get_open_files() no longer hangs on Windows (this was a very old and high-priority issue)
    • #501: disk_io_counters() may return negative values on Windows
    • #504: (Linux) couldn't build RPM packages via setup.py

    The list of all fixes can be found here. For the next release I plan to drop support for Python 2.4 and 2.5 and hopefully network interfaces information similarly to ifconfig.

  3. psutil 2.0

    The time has finally come: psutil 2.0 is out! This is a release which took me a considerable amount of effort and careful thinking during the past 4 months as I went through a major rewrite and reorganization of both python and C extension modules. To get a sense of how much has changed you can compare the differences with old 1.2.1 version by running "hg diff -r release-1.2.1:release-2.0.0" which will produce more than 22,000 lines of output! In those 22k lines I tried to nail down all the quirks the project had accumulated since its start 4 years ago and the resulting code base is now cleaner than ever, more manageable and fully compliant with PEP-7 and PEP-8 guidelines. There were some difficult decisions because many of the changes I introduced are not backward compatible so I was concerned with the pain this may cause existing users. I kind of still am, but I'm sure the transition will be well perceived on the long run as it will result in more manageable user code. OK, enough with the preface and let's see what changed.

    API changes

    I already wrote a detailed blog post about what changed so I recommend you to use that as the official reference on how to port your code.

    RST documentation

    I've never been happy with old doc hosted on Google Code. The markup language provided by Google is pretty limited, plus it's not put under revision control. New doc is more detailed, it uses reStructuredText as the markup language, it lives in the same code repository as psutil's and it is hosted on the excellent readthedocs web site: http://psutil.readthedocs.org/

    Physical CPUs count

    You're now able to distinguish between logical and physical CPUs:

    >>> psutil.cpu_count()  # logical
    4
    >>> psutil.cpu_count(logical=False)  # physical cores only
    2
    

    Full story is in issue 427.

    Process instances are hashable

    Basically this means process instances can now be checked for equality and can be used with set()s:

    >>> p1 = psutil.Process()
    >>> p2 = psutil.Process()
    >>> p1 == p2
    True
    >>> set((p1, p2))
    set([<psutil.Process(pid=8217, name='python') at 140007043550608>])
    

    Full story is in issue 452.

    Speedups

    • #477: process cpu_percent() is about 30% faster.
    • #478: (Linux) almost all APIs are about 30% faster on Python 3.X.

    Other improvements and bugfixes

    • #424: Windows installers for Python 3.X 64-bit
    • #447: psutil.wait_procs() timeout parameter is now optional.
    • #459: a Makefile is now available for running tests and other repetitive tasks (also on Windows)
    • #463: timeout parameter of cpu_percent* functions default to 0.0 because it was a common trap to introduce slowdowns.
    • #340: (Windows) process open_files() no longer hangs.
    • #448: (Windows) fixed a memory leak affecting children() and ppid() methods.
    • #461: namedtuples are now pickle-able.
    • #474: (Windows) Process.cpu_percent() is no longer capped at 100%

    OK, that's all folks. I hope you will enjoy this new version and report your feedback.

  4. psutil 2.0 API redesign

    This my second blog post is going to be about psutil 2.0, a major release in which I decided to reorganize the existing API for the sake of consistency. At the time of writing psutil 2.0 is still under development and the intent of this blog post is to serve as an official reference which describes how you should port your existent code base. In doing so I will also explain why I decided to make these changes. Despite many APIs will still be available as aliases pointing to the newer ones, the overall changes are numerous and many of them are not backward compatible. I'm sure many people will be sorely bitten but I think this is for the better and it needed to be done, hopefully for the first and last time. OK, here we go now.

    Module constants turned into functions

    What changed

    Old name Replacement
    psutil.BOOT_TIME psutil.boot_time()
    psutil.NUM_CPUS psutil.cpu_count()
    psutil.TOTAL_PHYMEM psutil.virtual_memory().total

    Why I did it

    I already talked about this more extensively in this blog post. In short: other than introducing unnecessary slowdowns, calculating a module level constant at import time is dangerous because in case of error the whole app will crash. Also, the represented values may be subject to change (think about the system clock) but the constant cannot be updated. Thanks to this hack accessing the old constants still works and produces a DeprecationWarning.

    Module functions renamings

    What changed

    Old name Replacement
    psutil.get_boot_time() psutil.boot_time()
    psutil.get_pid_list() psutil.pids()
    psutil.get_users() psutil.users()

    Why I did it

    They were the only module level functions having a get_ prefix. All others do not.

    Process class' methods renamings

    All methods lost their get_ and set_ prefixes. A single method can now be used for both getting and setting (if a value is passed). Assuming p = psutil.Process():

    Old name Replacement
    p.get_children() p.children()
    p.get_connections() p.connections()
    p.get_cpu_affinity() p.cpu_affinity()
    p.get_cpu_percent() p.cpu_percent()
    p.get_cpu_times() p.cpu_times()
    p.get_io_counters() p.io_counters()
    p.get_ionice() p.ionice()
    p.get_memory_info() p.memory_info()
    p.get_ext_memory_info() p.memory_info_ex()
    p.get_memory_maps() p.memory_maps()
    p.get_memory_percent() p.memory_percent()
    p.get_nice() p.nice()
    p.get_num_ctx_switches() p.num_ctx_switches()
    p.get_num_fds() p.num_fds()
    p.get_num_threads() p.num_threads()
    p.get_open_files() p.open_files()
    p.get_rlimit() p.rlimit()
    p.get_threads() p.threads()
    p.getcwd() p.cwd()

    ...as for set_* methods:

    Old name Replacement
    p.set_cpu_affinity() p.cpu_affinity(cpus)
    p.set_ionice() p.ionice(ioclass, value=None)
    p.set_nice() p.nice(value)
    p.set_rlimit() p.rlimit(resource, limits=None)

    Why I did it

    I wanted to be consistent with system-wide module level functions which have no get_ prefix. After I got rid of get_ prefixes removing also set_ seemed natural and helped diminish the number of methods.

    Process properties are now methods

    What changed

    Assuming p = psutil.Process():

    Old name Replacement
    p.cmdline p.cmdline()
    p.create_time p.create_time()
    p.exe p.exe()
    p.gids p.gids()
    p.name p.name()
    p.parent p.parent()
    p.ppid p.ppid()
    p.status p.status()
    p.uids p.uids()
    p.username p.username()

    Why I did it

    Different reasons:

    • Having a mixed API which uses both properties and methods for no particular reason is confusing and messy as you don't know whether to use "()" or not (see here).
    • It is usually expected from a property to not perform many computations internally whereas psutil actually invokes a function every time it is accessed. This has two drawbacks: * you may get an exception just by accessing the property (e.g. "p.name" may raise NoSuchProcess or AccessDenied) * you may erroneously think properties are cached but this is true only for name, exe, and create_time.

    CPU percent intervals

    What changed

    The timeout parameter of cpu_percent* functions now defaults to 0.0 instead of 0.1. The functions affected are:

    • psutil.Process.cpu_percent()
    • psutil.cpu_percent()
    • psutil.cpu_times_percent()

    Why I did it

    I originally set 0.1 as the default timeout because in order to get a meaningful percent value you need to wait some time. Having an API which "sleeps" by default is risky though, because it's easy to forget it does so. That is particularly problematic when calling cpu_percent() for all processes: it's very easy to forget about specifying timeout=0 resulting in dramatic slowdowns which are hard to spot. For example, this code snippet might take different seconds to complete depending on the number of active processes:

    >>> # this will be slow
    >>> for p in psutil.process_iter():
    ...    print(p.cpu_percent())
    

    Migration strategy

    Except for Process properties (name, exe, cmdline, etc.) all the old APIs are still available as aliases pointing to the newer names and raising DeprecationWarning. psutil will be very clear on what you should use instead of the deprecated API as long you start the interpreter with the "-Wd" option. This will enable deprecation warnings which were silenced in Python 2.7 (IMHO, from a developer standpoint this was a bad decision).

    giampaolo@ubuntu:/tmp$ python -Wd
    Python 2.7.3 (default, Sep 26 2013, 20:03:06)
    [GCC 4.6.3] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import psutil
    >>> psutil.get_pid_list()
    __main__:1: DeprecationWarning: psutil.get_pid_list is deprecated; use psutil.pids() instead
    [1, 2, 3, 6, 7, 13, ...]
    >>>
    >>>
    >>> p = psutil.Process()
    >>> p.get_cpu_times()
    __main__:1: DeprecationWarning: get_cpu_times() is deprecated; use cpu_times() instead
    pcputimes(user=0.08, system=0.03)
    >>>
    

    If you have a solid test suite you can run tests and fix the warnings one by one. As for the the Process properties which were turned into methods it's more difficult because whereas psutil 1.2.1 returns the actual value, psutil 2.0.0 will return the bound method:

    # psutil 1.2.1
    >>> psutil.Process().name
    'python'
    >>>
    
    # psutil 2.0.0
    >>> psutil.Process().name
    <bound method Process.name of psutil.Process(pid=19816, name='python') at 139845631328144>
    >>>
    

    What I would recommend if you want to drop support with 1.2.1 is to grep for ".name", ".exe" etc. and just replace them with ".exe()" and ".name()" one by one. If on the other hand you want to write a code which works with both versions I see two possibilities:

    • #1 check version info, like this:
    >>> PSUTIL2 = psutil.version_info >= (2, 0)
    >>> p = psutil.Process()
    >>> name = p.name() if PSUTIL2 else p.name
    >>> exe = p.exe() if PSUTIL2 else p.exe
    
    • #2 get rid of all ".name", ".exe" occurrences you have in your code and use as_dict() instead:
    >>> p = psutil.Process()
    >>> pinfo = p.as_dict(attrs=["name", "exe"])
    >>> pinfo
    {'exe': '/usr/bin/python2.7', 'name': 'python'}
    >>> name = pinfo['name']
    >>> exe = pinfo['exe']
    

    New features introduced in 2.0.0

    Ok, enough with the bad news. =) psutil 2.0.0 is not only about code breakage. I also had the chance to integrate a bunch of interesting features.

    • #427: you're now able to distinguish between the number of logical and physical CPUs:
    >>> psutil.cpu_count()  # logical
    4
    >>> psutil.cpu_count(logical=False)  # physical cores only
    2
    
    • #452: process classes are now hashable and can be checked for equality. That means you can use Process objects with sets (finally!).
    • #447: psutil.wait_procs() "timeout" parameter is now optional
    • #461: functions returning namedtuples are now pickle-able
    • #459: a Makefile is now available to automatize repetitive tasks such as build, install, running tests etc. There's also a make.bat for Windows.
    • introduced unittest2 module as a requirement for running tests

Social

Feeds