Blog posts for tags/psutil

1. Announcing psutil 5.3.0

   psutil 5.3.0 is finally out. This release is a major one, bigger than any release before it in terms of improvements and bugfixes. It is interesting to notice how huge the diff between 5.2.2 and 5.3.0 is. This is because I've been travelling quite a lot this year, so I kept postponing it. It may sound weird but I consider publishing a new release and writing a blog post about it more stressful than working on the release itself. =). Anyway, here goes.

   Full Unicode support¶

   String-returning APIs (Process.exe(), Process.cwd(), Process.username(), etc.) are now Unicode-correct on both Python 2 and 3 (#1040), see detailed separate blog post.

   Improved process_iter()¶

   process_iter() now accepts attrs and ad_value parameters, letting you pre-fetch process attributes in one shot and skip the try/except NoSuchProcess boilerplate, see detailed separate blog post.

   Automatic overflow handling of numbers¶

   On very busy or long-lived systems, numbers returned by disk_io_counters() and net_io_counters() functions may wrap (restart from zero). Up to version 5.2.x you had to take this into account, while now this is automatically handled by psutil (see: #802). If a "counter" restarts from 0 psutil will add the value from the previous call for you so that numbers will never decrease. This is crucial for applications monitoring disk or network I/O in real time. Old behavior can be resumed by passing the nowrap=True argument.

   SunOS Process environ()¶

   Process.environ() is now available also on SunOS (see #1091).

   Other improvements and bug fixes¶

   Amongst others, here are a couple of important bug fixes I'd like to mention:

   • #1044: on OSX different Process methods could incorrectly raise AccessDenied for zombie processes. This was due to the poor proc_pidpath OSX API.
   • #1094: on Windows, pid_exists() may lie due to the poor OpenProcess Windows API which can return a handle even when a process PID no longer exists. This had repercussions for many Process methods such as cmdline(), environ(), cwd(), connections() and others which could have unpredictable behaviors such as returning empty data or erroneously raising NoSuchProcess exceptions. For the same reason (broken OpenProcess API), processes could unexpectedly stick around after using terminate() and wait().

   BSD systems also received some love (NetBSD and OpenBSD in particular). Different memory leaks were fixed and functions returning connected sockets were partially rewritten. The full list of enhancements and bug fixes can be seen here.

   About me¶

   I would like to spend a couple more words about my current situation. Last year (2016) I relocated to Prague and remote worked from there the whole year (it's been cool - great city!). This year I have mainly been resting in Turin (Italy) due to some health issues and travelling across Asia once I started to recover. I am currently in Shenzhen, China, and unless the current situation with North Korea gets worse I'm planning to continue my trip until November and visit Taiwan, South Korea and Japan. Once I'm finished, the plan is to briefly return to Turin (Italy) and finally return to Prague. By then I will probably be looking for a new (remote) gig again, so if you have anything for me by November feel free to send me a message. ;-)

2. Fixing Unicode across Python 2 and 3

   This one took a while. Adding proper Unicode support to psutil took four months of auditing, design decisions, and rewriting nearly every API that returned a string. The full journey is documented in #1040, and what follows is a summary.

   This can serve as a case study for any Python library with a C extension that needs to support both Python 2 and Python 3, as it will encounter the exact same set of problems.

   What was broken¶

   psutil has different APIs returning a string, many of which misbehaved when it came to unicode. There were three distinctive problems (#1040). Each API could:

   • A: raise a decoding error for non-ASCII strings (Python 3).
   • B: return unicode instead of str (Python 2).
   • C: return incorrect / invalid encoded data for non-ASCII strings (both).

   Process.memory_maps() hit all three on various OSes. disk_partitions() raised decoding errors on every UNIX except Linux. Windows service methods leaked unicode into Python 2 return values. The C extension had accumulated years of ad-hoc encode/decode decisions, with no single rule covering all of them.

   It was a mess.

   Filesystem or locale encoding?¶

   First problem was that the C extension was using 2 approaches when it came to decoding and returning a string: PyUnicode_DecodeFSDefault (filesystem encoding) for path-like APIs, and PyUnicode_DecodeLocale (user locale) for non-path strings like Process.username().

   It appeared clear that I had to use PyUnicode_DecodeFSDefault for all filesystem-related APIs like Process.exe() and Process.open_files().

   It was less clear, though, when to use PyUnicode_DecodeLocale.

   After some back and forth, I decided to use a single encoding for all APIs: the filesystem encoding (PyUnicode_DecodeFSDefault). This makes the encoding choice an implementation detail of psutil, not something the user has to care about.

   Error handling¶

   Second question was what to do in case the string cannot be correctly decoded (because invalid, corrupted or whatever). On Python 3 + UNIX the natural choice was 'surrogateescape', which is also the default for PyUnicode_DecodeFSDefault. On Windows the default is 'surrogatepass' (Python 3.6) or 'replace' as per PEP 529.

   And here come the troubles: Python 2 is different. To correctly handle all kinds of strings on Python 2 we should return unicode instead of str, but I didn't want to do that, nor have APIs which return two different types depending on the circumstance.

   Since unicode support is already broken in Python 2 and its stdlib (see bpo-18695), I was happy to always return str, use 'replace' as the error handler, and simply consider unicode support in psutil + Python 2 broken.

   Final behavior¶

   Starting from 5.3.0, psutil behaves consistently across all APIs that return a string. The rules are intentionally simple, even if the underlying implementation is not.

   The notes below apply to any method returning a string such as Process.exe() or Process.cwd(), including non-filesystem-related methods such as Process.username():

   • all strings are encoded using the OS filesystem encoding (PyUnicode_DecodeFSDefault), which varies depending on the platform you're on (e.g. 'UTF-8' on Linux, 'mbcs' on Windows).

   • no API call is supposed to crash with UnicodeDecodeError.

   • in case of badly encoded data returned by the OS, the following error handlers are used to replace the bad characters in the string:

     • Python 2: 'replace'.
     • Python 3: 'surrogateescape' on POSIX, 'replace' on Windows.

   • on Python 2 all APIs return bytes (str type), never unicode.

   • on Python 2 you can go back to unicode by doing:

     >>> unicode(proc.exe(), sys.getdefaultencoding(), errors="replace")
     

   The full journey was implemented in PR-1052, and shipped in 5.3.0 (see the changelog).

3. Improved process_iter()

   This is part of the psutil 5.3.0 release (see the changelog for the full list of changes).

   The old pattern¶

   Iterating over processes and collecting attributes requires more boilerplate than it should. A process returned by psutil.process_iter() may disappear before you access it, or require elevated privileges, so every lookup has to be guarded with a try / except:

   >>> import psutil
   >>> for proc in psutil.process_iter():
   ...     try:
   ...         pinfo = proc.as_dict(attrs=['pid', 'name'])
   ...     except (psutil.NoSuchProcess, psutil.AccessDenied):
   ...         pass
   ...     else:
   ...         print(pinfo)
   ...
   {'pid': 1, 'name': 'systemd'}
   {'pid': 2, 'name': 'kthreadd'}
   {'pid': 3, 'name': 'ksoftirqd/0'}
   

   This is not decorative. It's necessary to avoid the race condition.

   The new pattern¶

   5.3.0 adds attrs and ad_value parameters to psutil.process_iter(). With these, the loop body becomes:

   >>> import psutil
   >>> for proc in psutil.process_iter(attrs=['pid', 'name']):
   ...     print(proc.info)
   ...
   {'pid': 1, 'name': 'systemd'}
   {'pid': 2, 'name': 'kthreadd'}
   {'pid': 3, 'name': 'ksoftirqd/0'}
   

   Internally, process_iter() attaches an info dict to the Process instance. The attributes are pre-fetched in one shot. Processes that disappear during iteration are silently skipped, and attributes that would raise AccessDenied get assigned ad_value, which defaults to None:

   for p in psutil.process_iter(['name', 'username'], ad_value="N/A"):
       print(p.name(), p.username())
   

   Performance¶

   Beyond the syntactic win, the new syntax is also faster than calling individual methods in a loop. process_iter(attrs=[...]) is equivalent to using Process.oneshot() on each process (see Making psutil twice as fast for how that works): attributes that share a syscall or a /proc file are fetched together instead of re-read on every method call, which is a lot faster.

   Comprehensions¶

   With the exception boilerplate out of the way, comprehensions finally work cleanly. E.g. getting processes owned by the current user can be written as:

   >>> import getpass
   >>> from pprint import pprint as pp
   >>> pp([(p.pid, p.info['name']) for p in psutil.process_iter(attrs=['name', 'username']) if p.info['username'] == getpass.getuser()])
   [(16832, 'bash'),
    (19772, 'ssh'),
    (20492, 'python')]
   

4. Sensors: temperatures, battery, CPU frequency

   psutil 5.1.0 is out. This release introduces new APIs to retrieve hardware temperatures, battery status, and CPU frequency information.

   Temperatures¶

   You can now retrieve hardware temperatures (PR-962). This is currently available on Linux only.

   • On Windows it's hard to do in a hardware-agnostic way. I ran into 3 WMI-based approaches, none of which worked with my hardware, so I gave up.
   • On macOS it seems relatively easy, but my virtualized macOS box doesn't support sensors, so I gave up for lack of hardware. If someone wants to give it a try, be my guest.

   >>> import psutil
   >>> psutil.sensors_temperatures()
   {'acpitz': [shwtemp(label='', current=47.0, high=103.0, critical=103.0)],
    'asus': [shwtemp(label='', current=47.0, high=None, critical=None)],
    'coretemp': [shwtemp(label='Physical id 0', current=52.0, high=100.0, critical=100.0),
                 shwtemp(label='Core 0', current=45.0, high=100.0, critical=100.0),
                 shwtemp(label='Core 1', current=52.0, high=100.0, critical=100.0),
                 shwtemp(label='Core 2', current=45.0, high=100.0, critical=100.0),
                 shwtemp(label='Core 3', current=47.0, high=100.0, critical=100.0)]}
   

   Battery status¶

   Battery status information is now available on Linux, Windows and FreeBSD (PR-963).

   >>> import psutil
   >>>
   >>> def secs2hours(secs):
   ...     mm, ss = divmod(secs, 60)
   ...     hh, mm = divmod(mm, 60)
   ...     return "%d:%02d:%02d" % (hh, mm, ss)
   ...
   >>> battery = psutil.sensors_battery()
   >>> battery
   sbattery(percent=93, secsleft=16628, power_plugged=False)
   >>> print("charge = %s%%, time left = %s" % (battery.percent, secs2hours(battery.secsleft)))
   charge = 93%, time left = 4:37:08
   

   CPU frequency¶

   Available on Linux, Windows and macOS (PR-952). Only Linux reports the real-time value (always changing); other platforms return the nominal "fixed" value.

   >>> import psutil
   >>> psutil.cpu_freq()
   scpufreq(current=931.42925, min=800.0, max=3500.0)
   >>> psutil.cpu_freq(percpu=True)
   [scpufreq(current=2394.945, min=800.0, max=3500.0),
    scpufreq(current=2236.812, min=800.0, max=3500.0),
    scpufreq(current=1703.609, min=800.0, max=3500.0),
    scpufreq(current=1754.289, min=800.0, max=3500.0)]
   

   What CPU a process is on¶

   Tells you which CPU a process is currently running on, somewhat related to Process.cpu_affinity() (PR-954). It's interesting for visualizing how the OS scheduler keeps evenly reassigning processes across CPUs (see the cpu_distribution.py script).

   CPU affinity¶

   A new shorthand is available to set affinity against all eligible CPUs:

   Process().cpu_affinity([])
   

   This was added because on Linux (#956) it is not always possible to set affinity against all CPUs directly. It is equivalent to:

   psutil.Process().cpu_affinity(list(range(psutil.cpu_count())))
   

   Other bug fixes¶

   See the full list in the changelog.

5. Making psutil twice as fast

   Starting from psutil 5.0.0 you can query multiple Process fields around twice as fast as before (see #799 and Process.oneshot() doc). It took 7 months, 108 commits, and a massive refactoring of psutil internals (PR-937), and I think it's one of the best improvements ever shipped in a psutil release.

   The problem¶

   How process information is retrieved varies by OS. Sometimes it means reading a file in /proc (Linux), other times calling C (Windows, BSD, macOS, SunOS), but it's always done differently. Psutil abstracts this away: you call Process.name() without worrying about what happens under the hood or which OS you're on.

   Internally, multiple pieces of process info (e.g. Process.name(), Process.ppid(), Process.uids(), Process.create_time()) are fetched by the same syscall. On Linux we read /proc/PID/stat to get the process name, terminal, CPU times, creation time, status and parent PID, but only one value is returned: the others are discarded. On Linux this code reads /proc/PID/stat 6 times:

   >>> import psutil
   >>> p = psutil.Process()
   >>> p.name()
   >>> p.cpu_times()
   >>> p.create_time()
   >>> p.ppid()
   >>> p.status()
   >>> p.terminal()
   

   On BSD most process metrics can be fetched with a single sysctl(), yet psutil was invoking it for each process method (e.g. see here and here).

   Do it in one shot¶

   It's clear that this approach is inefficient, especially in tools like top or htop, where process info is continuously fetched in a loop. psutil 5.0.0 introduces a new Process.oneshot() context manager. Inside it, the internal routine runs once (in the example, on the first Process.name() call) and the other values are cached. Subsequent calls sharing the same internal routine (read /proc/PID/stat, call sysctl() or whatever) return the cached value. The code above can now be rewritten like this, and on Linux it runs 2.4 times faster:

   >>> import psutil
   >>> p = psutil.Process()
   >>> with p.oneshot():
   ...     p.name()
   ...     p.cpu_times()
   ...     p.create_time()
   ...     p.ppid()
   ...     p.status()
   ...     p.terminal()
   

   Implementation¶

   One great thing about psutil's design is its abstraction. It is divided into 3 "layers". Layer 1 is represented by the main Process class (Python), which exposes the high-level API. Layer 2 is the OS-specific Python module, which is a thin wrapper on top of the OS-specific C extension module (layer 3).

   Because the code was organized this way (modular), the refactoring was reasonably smooth. I first refactored those C functions that collect multiple pieces of info and grouped them into a single function (e.g. see BSD implementation). Then I wrote a decorator that enables the cache only when requested (when entering the context manager), and decorated the "grouped functions" with it. The caching mechanism is controlled by the Process.oneshot() context manager, which is the only thing exposed to the end user. Here's the decorator:

   def memoize_when_activated(fun):
       """A memoize decorator which is disabled by default. It can be
       activated and deactivated on request.
       """
       @functools.wraps(fun)
       def wrapper(self):
           if not wrapper.cache_activated:
               return fun(self)
           else:
               try:
                   ret = cache[fun]
               except KeyError:
                   ret = cache[fun] = fun(self)
               return ret
   
       def cache_activate():
           """Activate cache."""
           wrapper.cache_activated = True
   
       def cache_deactivate():
           """Deactivate and clear cache."""
           wrapper.cache_activated = False
           cache.clear()
   
       cache = {}
       wrapper.cache_activated = False
       wrapper.cache_activate = cache_activate
       wrapper.cache_deactivate = cache_deactivate
       return wrapper
   

   To measure the speedup I wrote a benchmark script (well, two actually), and kept tuning until I was sure the change actually made psutil faster. The scripts report the speedup for calling all the "grouped" methods together (best-case scenario).

   Linux: +2.56x speedup¶

   The Linux implementation is mostly Python, reading files in /proc. These files typically expose multiple pieces of info per process; /proc/PID/stat and /proc/PID/status are the perfect example. We aggregate them into three groups. See the relevant code here.

   Windows: from +1.9x to +6.5x speedup¶

   Windows is an interesting one. For a process owned by our user, we group only Process.num_threads(), Process.num_ctx_switches() and Process.num_handles(), for a +1.9x speedup if we access those methods in one shot.

   Windows is special though, because certain methods have a dual implementation (#304): a "fast method" is tried first, but if the process is owned by another user it fails with AccessDenied. psutil then falls back to a second, "slower" method (see here for example).

   It's slower because it iterates over all PIDs, but unlike the "plain" Windows APIs it can still retrieve multiple pieces of information in one shot: number of threads, context switches, handles, CPU times, create time, and I/O counters.

   That's why querying processes owned by other users results in an impressive +6.5x speedup.

   macOS: +1.92x speedup¶

   On macOS we can get 2 groups of information. With sysctl() we get process parent PID, uids, gids, terminal, create time, name. With proc_info() we get CPU times (for PIDs owned by another user), memory metrics and ctx switches. Not bad.

   BSD: +2.18x speedup¶

   On BSD we gather tons of process info just by calling sysctl() (see implementation): process name, ppid, status, uids, gids, IO counters, CPU and create times, terminal and ctx switches.

   SunOS: +1.37x speedup¶

   SunOS is like Linux (it reads files in /proc), but the code is in C. Here too, we group different metrics together (see here and here).

   Discussion¶

   • Reddit