Differences with FreeBSD¶

As expected, OpenBSD implementation is very similar to FreeBSD's (which was already in place), that is why I decided to merge most of it in a single C file (_psutil_bsd.c) and use 2 separate C files for when the two implementations differed too much: freebsd.c and openbsd.c. In terms of functionality here's the differences with FreeBSD. Unless specified, these differences are due to the kernel which does not provide the information natively (meaning we can't do anything about it).

• Process.memory_maps() is not implemented. The kernel provides the necessary pieces but I didn't do this yet (hopefully later).
• Process.num_ctx_switches()'s involuntary field is always 0. kinfo_proc provides this info but it is always set to 0.
• Process.cpu_affinity() (get and set) is not supported.
• Process.exe() is determined by inspecting the command line so it may not always be available (return None).
• psutil.swap_memory() sin and sout (swap in and swap out) values are not available and hence are always set to 0.
• psutil.cpu_count(logical=False) always return None.

Similarly to FreeBSD, also OpenBSD implementation of Process.open_files() is problematic as it is not able to return file paths (FreeBSD can sometimes). Other than these differences the functionalities are all there and pretty much the same, so overall I'm pretty satisfied with the result.

Considerations about BSD platforms¶

psutil has been supporting FreeBSD basically since the beginning (year 2009). At the time it made sense to support FreeBSD instead of other BSD variants because it is the most popular, followed by OpenBSD and NetBSD. Compared to FreeBSD, OpenBSD appears to be more "minimal" both in terms of facilities provided by the kernel and the number of system administration tools available. One thing which I appreciate a lot about FreeBSD is that the source code of all CLI tools installed on the system is available under /usr/bin/src, which was a big help for implementing all psutil APIs. OpenBSD source code is also available but it uses CSV and I am not sure it includes the source code for all CLI tools. There are still two more BSD variants for which it may be worth to add support for: NetBSD and DragonflyBSD (in this order). About a year ago some guy provided a patch for adding basic NetBSD support so it is likely that will happen sooner or later.

Other enhancements available in this release¶

The only other enhancement is issue #558, which allows specifying a different location of /proc filesystem on Linux.

External discussions¶

• Reddit
• Hacker news

net_if_addrs()¶

In a few words, we're now able to list network interface addresses similarly to "ifconfig" command on UNIX:

>>> import psutil
>>> from pprint import pprint
>>> pprint(psutil.net_if_addrs())
{'ethernet0': [snic(family=<AddressFamily.AF_INET: 2>,
                    address='10.0.0.4',
                    netmask='255.0.0.0',
                    broadcast='10.255.255.255'),
               snic(family=<AddressFamily.AF_PACKET: 17>,
                    address='9c:eb:e8:0b:05:1f',
                    netmask=None,
                    broadcast='ff:ff:ff:ff:ff:ff')],
 'localhost': [snic(family=<AddressFamily.AF_INET: 2>,
                    address='127.0.0.1',
                    netmask='255.0.0.0',
                    broadcast='127.0.0.1'),
               snic(family=<AddressFamily.AF_PACKET: 17>,
                    address='00:00:00:00:00:00',
                    netmask=None,
                    broadcast='00:00:00:00:00:00')]}

This is limited to AF_INET (IPv4), AF_INET6 (IPv6) and AF_LINK (Ethernet) address families. If you want something more poweful (e.g. AF_BLUETOOTH) you can take a look at netifaces extension. And here's the code which does these tricks on POSIX and Windows:

• POSIX
• Windows

Also, here's some doc.

net_if_stats()¶

This will return a bunch of information about network interface cards:

>>> import psutil
>>> from pprint import pprint
>>> pprint(psutil.net_if_stats())
{'ethernet'0: snicstats(isup=True,
                        duplex=<NicDuplex.NIC_DUPLEX_FULL: 2>,
                        speed=100,
                        mtu=1500),
 'localhost': snicstats(isup=True,
                        duplex=<NicDuplex.NIC_DUPLEX_UNKNOWN: 0>,
                        speed=0,
                        mtu=65536)}

Again, here's the code for each platform:

• Windows
• Linux
• OSX & FreeBSD
• SunOS

...and the doc.

Enums¶

Enums are a nice new feature introduced in Python 3.4. Very briefly (or at least, this is what I appreciate the most about them), they help you write an API with human-readable constants. If you use Python 2 you'll see something like this:

>>> import psutil
>>> psutil.IOPRIO_CLASS_IDLE
3

On Python 3.4 you'll see a more informative:

>>> import psutil
>>> psutil.IOPRIO_CLASS_IDLE
<IOPriority.IOPRIO_CLASS_IDLE: 3>

They are backward compatible, meaning if you're sending serialized data produced with psutil through the network you can safely use comparison operators and so on. The psutil APIs returning enums (on Python >=3.4) are:

• psutil.net_connections() (the address families):
• psutil.Process.connections() (same as above)
• psutil.net_if_stats() (all NIC_DUPLEX_* constants)
• psutil.Process.nice() on Windows (for all the *_PRIORITY_CLASS constants)
• psutil.Process.ionice() on Linux (for all the IOPRIO_CLASS_* constants)

All the other existing constants remained plain strings (STATUS_*) or integers (CONN_*).

Zombie processes¶

This is a big one. The full story is here but basically the support for zombie processes on UNIX was broken (except on Linux, and Windows doesn't have zombie processes). Up until psutil 2.X we could instantiate a zombie process:

>>> pid = create_zombie()
>>> p = psutil.Process(pid)

...but every time we queried it we got a NoSuchProcess exception:

>>> psutil.name()
  File "psutil/__init__.py", line 374, in _init
    raise NoSuchProcess(pid, None, msg)
psutil.NoSuchProcess: no process found with pid 123

That was misleading though because the PID technically still existed:

>>> psutil.pid_exists(p.pid)
True

Furthermore, depending on what platform you were on, certain process stats could still be queried (instead of raising NoSuchProcess):

>>> psutil.cmdline()
['python']

Also process_iter() did not return zombie processes at all. This was probably the worst aspect because being able to identify them is an important use case, as they signal an issue with process: if a parent process spawns a child, terminates it (via kill()), but doesn't wait() for it it will create a zombie. Long story short, the way this changed in psutil 3.0 is that:

• we now have a new ZombieProcess exception, raised every time we're not able to query a process because it's a zombie
• it is raised instead of NoSuchProcess (which was incorrect and misleading)
• it is still backward compatible (meaning you won't have to change your old code) because it inherits from NoSuchProcess
• process_iter() finally works, meaning you can safely identify zombie processes like this:

import psutil
zombies = []
for p in psutil.process_iter():
    try:
        if p.status() == psutil.STATUS_ZOMBIE:
            zombies.append(p)
    except NoSuchProcess:
        pass

Removal of deprecated APIs¶

This is another big one, probably the biggest. In a previous blog post I already talked about deprecated APIs. What I did back then (January 2014) was to rename and officially deprecate different APIs and provide aliases for them so that people wouldn't yell at me because I broke their existent code. The most interesting deprecation was certainly the one affecting module constants and the hack which was used in order to provide "module properties". With this new release I decided to get rid of all those aliases. I'm sure this will cause problems but hey! This is a new major release, right? =). Plus the amount of crap which was removed is impressive (see the commit). Here's the old aliases which are now gone for good (or bad, depending on how much headache they will cause you):

Removed module functions and constants¶

Process methods (assuming p = psutil.Process()):

If your code suddenly breaks with AttributeError after you upgraded psutil it means you were using one of those deprecated aliases. In that case just take a look at the table above and rename stuff in accordance.

Bug fixes¶

I fixed a lot of stuff (full list here), but here's the list of things which I think are worth mentioning:

• #512: [FreeBSD] fix segfault in net_connections().
• #593: [FreeBSD] Process.memory_maps() segfaults.
• #606: Process.parent() may swallow NoSuchProcess exceptions.
• #614: [Linux]: cpu_count(logical=False) return the number of physical CPUs instead of physical cores.
• #628: [Linux] Process.name() truncates process name in case it contains spaces or parentheses.

Ease of development¶

These are not enhancements you will directly benefit from but I put some effort into making my life easier every time I work on psutil.

• I care about psutil code being fully PEP8 compliant so I added a pre-commit GIT hook which runs flake8 on every commit and rejects it if the coding style is not compliant. The way I install this is via make install-git-hooks.
• I added a make install-dev-deps command which installs all deps and stuff which is useful for testing (ipdb, coverage, etc).
• A new make coverage command which runs coverage. With this I discovered some of parts in the code which weren't covered by tests and I fixed that.
• I started using tox to easily test psutil against all supported Python versions (from 2.6 to 3.4) in one shot.
• I reorganized tests so that now they can be easily executed with py.test and nose (before, only unittest runner was fully supported)

Final words¶

I must say I'm pretty satisfied with how psutil is going and the satisfaction I still get every time I work on it. Right now it gets almost 800.000 download a month, which is pretty great for a Python library. As of right now I consider psutil almost "completed" in terms of features, meaning I'm basically running out of ideas on what I should add next (see TODO). From now on the future development will probably focus on adding support for more exotic platforms (OpenBSD, NetBSD, Android). There also have been some discussions on python-ideas mailing list about including psutil into Python stdlib but, assuming that will ever happen, it's still far away in the future as it would require a lot of time which I currently don't have. That should be all. I hope you will all enjoy this new release.

socket.sendfile()¶

I recently contributed a patch for Python's socket module which adds a high-level socket.sendfile() method (see full discussion at BPO-17552). socket.sendfile() will transmit a file until EOF is reached by attempting to use os.sendfile(), if available, else it falls back on using plain socket.send(). Internally, it takes care of handling socket timeouts and provides two optional parameters to move the file offset or to send only a limited amount of bytes. I came up with this idea because getting all of that right is a bit tricky, so a generic wrapper seemed to be convenient to have. socket.sendfile() will make its appearance in Python 3.5.

sendfile and Python¶

sendfile(2) made its first appearance into the Python stdlib kind of late: Python 3.3. It was contributed by Ross Lagerwall and me in BPO-10882. Since the patch didn't make it into python 2.X and I wanted to use sendfile() in pyftpdlib I later decided to release it as a stand alone module working with older (2.5+) Python versions (see pysendfile project). Starting with version 3.5, Python will hopefully start using sendfile() more extensively, in details:

• BPO-13563: ftplib
• BPO-13559: httplib
• asyncio: there are some plans for this even though no actual patch yet, see discussion and BDFL involvement.

Also, Windows provides something similar to sendfile(2): TransmitFile. Now that socket.sendfile() is in place it seems natural to add support for it as well (see BPO-21721).

Backport to Python 2.6 and 2.7¶

For those of you who are interested in using socket.sendfile() with older Python 2.6 and 2.7 versions here's a backport. It requires pysendfile module to be installed. Full code including tests is hosted here.

#!/usr/bin/env python

"""
This is a backport of socket.sendfile() for Python 2.6 and 2.7.
socket.sendfile() will be included in Python 3.5:
http://bugs.python.org/issue17552
Usage:

>>> import socket
>>> file = open("somefile.bin", "rb")
>>> sock = socket.create_connection(("localhost", 8021))
>>> sendfile(sock, file)
42319283
>>>
"""

import errno
import io
import os
import select
import socket
try:
    memoryview  # py 2.7 only
except NameError:
    memoryview = lambda x: x

if os.name == 'posix':
    import sendfile as pysendfile  # requires "pip install pysendfile"
else:
    pysendfile = None


_RETRY = frozenset((errno.EAGAIN, errno.EALREADY, errno.EWOULDBLOCK,
                    errno.EINPROGRESS))


class _GiveupOnSendfile(Exception):
    pass


if pysendfile is not None:

    def _sendfile_use_sendfile(sock, file, offset=0, count=None):
        _check_sendfile_params(sock, file, offset, count)
        sockno = sock.fileno()
        try:
            fileno = file.fileno()
        except (AttributeError, io.UnsupportedOperation) as err:
            raise _GiveupOnSendfile(err)  # not a regular file
        try:
            fsize = os.fstat(fileno).st_size
        except OSError:
            raise _GiveupOnSendfile(err)  # not a regular file
        if not fsize:
            return 0  # empty file
        blocksize = fsize if not count else count

        timeout = sock.gettimeout()
        if timeout == 0:
            raise ValueError("non-blocking sockets are not supported")
        # poll/select have the advantage of not requiring any
        # extra file descriptor, contrarily to epoll/kqueue
        # (also, they require a single syscall).
        if hasattr(select, 'poll'):
            if timeout is not None:
                timeout *= 1000
            pollster = select.poll()
            pollster.register(sockno, select.POLLOUT)

            def wait_for_fd():
                if pollster.poll(timeout) == []:
                    raise socket._socket.timeout('timed out')
        else:
            # call select() once in order to solicit ValueError in
            # case we run out of fds
            try:
                select.select([], [sockno], [], 0)
            except ValueError:
                raise _GiveupOnSendfile(err)

            def wait_for_fd():
                fds = select.select([], [sockno], [], timeout)
                if fds == ([], [], []):
                    raise socket._socket.timeout('timed out')

        total_sent = 0
        # localize variable access to minimize overhead
        os_sendfile = pysendfile.sendfile
        try:
            while True:
                if timeout:
                    wait_for_fd()
                if count:
                    blocksize = count - total_sent
                    if blocksize <= 0:
                        break
                try:
                    sent = os_sendfile(sockno, fileno, offset, blocksize)
                except OSError as err:
                    if err.errno in _RETRY:
                        # Block until the socket is ready to send some
                        # data; avoids hogging CPU resources.
                        wait_for_fd()
                    else:
                        if total_sent == 0:
                            # We can get here for different reasons, the main
                            # one being 'file' is not a regular mmap(2)-like
                            # file, in which case we'll fall back on using
                            # plain send().
                            raise _GiveupOnSendfile(err)
                        raise err
                else:
                    if sent == 0:
                        break  # EOF
                    offset += sent
                    total_sent += sent
            return total_sent
        finally:
            if total_sent > 0 and hasattr(file, 'seek'):
                file.seek(offset)
else:
    def _sendfile_use_sendfile(sock, file, offset=0, count=None):
        raise _GiveupOnSendfile(
            "sendfile() not available on this platform")


def _sendfile_use_send(sock, file, offset=0, count=None):
    _check_sendfile_params(sock, file, offset, count)
    if sock.gettimeout() == 0:
        raise ValueError("non-blocking sockets are not supported")
    if offset:
        file.seek(offset)
    blocksize = min(count, 8192) if count else 8192
    total_sent = 0
    # localize variable access to minimize overhead
    file_read = file.read
    sock_send = sock.send
    try:
        while True:
            if count:
                blocksize = min(count - total_sent, blocksize)
                if blocksize <= 0:
                    break
            data = memoryview(file_read(blocksize))
            if not data:
                break  # EOF
            while True:
                try:
                    sent = sock_send(data)
                except OSError as err:
                    if err.errno in _RETRY:
                        continue
                    raise
                else:
                    total_sent += sent
                    if sent < len(data):
                        data = data[sent:]
                    else:
                        break
        return total_sent
    finally:
        if total_sent > 0 and hasattr(file, 'seek'):
            file.seek(offset + total_sent)


def _check_sendfile_params(sock, file, offset, count):
    if 'b' not in getattr(file, 'mode', 'b'):
        raise ValueError("file should be opened in binary mode")
    if not sock.type & socket.SOCK_STREAM:
        raise ValueError("only SOCK_STREAM type sockets are supported")
    if count is not None:
        if not isinstance(count, int):
            raise TypeError(
                "count must be a positive integer (got %s)" % repr(count))
        if count <= 0:
            raise ValueError(
                "count must be a positive integer (got %s)" % repr(count))


def sendfile(sock, file, offset=0, count=None):
    """sendfile(sock, file[, offset[, count]]) -> sent

    Send a *file* over a connected socket *sock* until EOF is
    reached by using high-performance sendfile(2) and return the
    total number of bytes which were sent.
    *file* must be a regular file object opened in binary mode.
    If sendfile() is not available (e.g. Windows) or file is
    not a regular file socket.send() will be used instead.
    *offset* tells from where to start reading the file.
    If specified, *count* is the total number of bytes to transmit
    as opposed to sending the file until EOF is reached.
    File position is updated on return or also in case of error in
    which case file.tell() can be used to figure out the number of
    bytes which were sent.
    The socket must be of SOCK_STREAM type.
    Non-blocking sockets are not supported.
    """
    try:
        return _sendfile_use_sendfile(sock, file, offset, count)
    except _GiveupOnSendfile:
        return _sendfile_use_send(sock, file, offset, count)

What I will miss the most about GC¶

First of all I must say that despite the unfortunate situation of GC I'm also sad for abandoning it. It started as a really great hosting platform, and it still has some peculiar aspects which I know I will be missing. In order of importance:

• The bug tracker: it is much more powerful than github's, especially for the extremely customizable labeling system which is pure gold, the excellent searching system and the grid view. GC bug tracker seriously kicks some ass so kudos to whoever was behind its design! By comparison github bug tracker is too minimalistic and it has no good way to order issues or list them in a more compact form. I'm totally gonna miss psutil bug tracker.
• Mercurial: I'm a big fan of Mercurial and I consider it way more pleasant to work with compared to GIT. I don't know exactly why GIT ended up being so much more used than Mercurial (probably because of github?) but I'm sure that many other guys like me who know both systems will agree that Mercurial is simply so much easier to use. Unfortunately once you decide to stick with github you have no other choice. Mercurial, I'm gonna miss you too!
• GC layout: it is much simpler than github's! Everything is easy to find, even for a non-geek person. The home page alone is perfect to summarize what the project is about and doesn't have tens of icons all over the place. github layout is more complicated and needs some time to get used to, even for a programmer. If these projects (psutil and others) weren't about programming I wouldn't have chosen github because it's "not for the masses".

What I appreciate about github¶

• Travis integration: there's this totally awesome free continuos integration service called Travis which given a configuration file like this it will automatically run tests on multiple python versions every time a commit is pushed. They recently added OSX support and Windows support is on the way. This way I will finally be able to quickly test psutil on Linux, OSX and Windows without using virtualized systems except for FreeBSD and Solaris! To me this is like the ultimate Christmas gift and I couldn't ask for any better. Note: as of today Travis only works with github.
• forks and pull requests: honestly I'm not a big fan of them (yet?), probably because I'm used to the python-dev development workflow consisting in uploading patches on the bug tracker and reviewing them (see this for example). Nevertheless to my understanding most people use pull requests in order to contribute to open source projects so basically this is a service I'm glad to offer to my users who hopefully will be able to contribute back more easily. GC has a cloning system but isn't anywhere near github's and I'm not even sure how it works (never cared).
• the "social" side of github including the fact that you can "star" developers and receive notifications about their activity was another big incentive for migrating. The personal landing page collecting all your contributions to different projects is absolutely cool. GC had something similar but they stupidly removed it all of the sudden and never reintroduced it back. A lot of people were angry but again, they didn't care. Actually this was the feature I appreciated the most about GC after the bug tracker and that is when I seriously started thinking about flipping off GC for good.
• the enormous user base: the fact that github is the most used code hosting platform out there will hopefully help me and my projects have a little more visibility. Also, in many job interviews I've been asked what my github profile was so it seems github also became an active part in getting jobs.
• gists: gists are "a simple way to share code snippets and pastes with others. All gists are Git repositories, so they are automatically versioned, forkable and usable from Git". Seriously, they are beautiful. In order to share my code snippets I've always used Active State but I think I will eventually migrate them as well in order to have everything in one place.
• the fact that if you mention an issue number as part of your commit message that specific issue will automatically be updated. As I said GC bug tracker is superior in basically any aspect but since I always took care of updating issues by mentioning the specific cset which fixed them (see for example here) having this little extra feature will save me some time.
• SSH keys: using Mercurial on GC means using password based authentication. Incredibly they still do not support SSH key based auth. Simply "git push"ing without entering any password when I'm not on my laptop is nice, and of course, it is also much more secure.

Migration¶

For those of you who are interested in knowing how I did it, here goes: as for moving the issue from GC bug tracker to github's I used this tool. I managed to preserve the issue IDs but unfortunately not the real owners nor the real issue dates, which kind of sucks. As for migrating the code from mercurial to git I just used this. The Mercurial -> GIT transition was perfect and I also managed to preserve the original Mercurial named branches and tags, which for me it was crucial. In conclusion, psutil is a 5 years, medium sized project with hundreds of issues: the transition in this case is definitively possible but not painless so if you plan on migrating, the sooner you do it the better.