Blog posts for tags/psutil

  1. Improved Linux memory metrics

    OK, another psutil release. The headline of 4.4.0 is more accurate memory metrics on Linux, plus a pile of macOS fixes I'd been sitting on for years.

    Linux virtual memory

    People had been complaining for a while that virtual_memory() didn't match what free reported on Linux (#862, #685, #538). I finally dug into it (#887, PR-890) and, funny enough, it turns out free itself was doing it wrong until about two years ago, when somebody got tired of everyone guessing and moved the calculation into the kernel.

    Starting with Linux 3.14, /proc/meminfo has a MemAvailable column. That's what psutil now uses, and the available / used fields match free exactly. On older kernels (< 3.14) psutil falls back to the same formula that kernel commit introduced.

    free's source code also inspired a fix that prevents available memory from overflowing total memory on LXC containers.

    macOS fixes

    For years I did psutil's macOS development on an old 10.5 install emulated via VirtualBox, running iDeneb (a hacked macOS). I finally got access to a more recent version (El Capitan) via VirtualBox + Vagrant, and could address a pile of long-standing bugs:

    • #514: Process.memory_maps() segfault (critical).
    • #783: Process.status() could return "running" for zombie processes.
    • #908: several methods could mask the real error for high-privileged PIDs, raising NoSuchProcess / AccessDenied instead of OSError / RuntimeError.
    • #909: Process.open_files() and Process.connections() could raise OSError with no exception set when the process was gone.
    • #916: fixed many compilation warnings.

    NIC netmask on Windows

    Small but nice: net_if_addrs() on Windows now returns the netmask too.

    Improved procinfo.py

    scripts/procinfo.py is my kitchen-sink demo script. I taught it a bunch of new tricks, so it now dumps pretty much everything psutil knows about a process:

    $ python scripts/procinfo.py
pid           4600
name          chrome
parent        4554 (bash)
exe           /opt/google/chrome/chrome
cwd           /home/giampaolo
cmdline       /opt/google/chrome/chrome
started       2016-09-19 11:12
cpu-tspent    27:27.68
cpu-times     user=8914.32, system=3530.59,
              children_user=1.46, children_system=1.31
cpu-affinity  [0, 1, 2, 3, 4, 5, 6, 7]
memory        rss=520.5M, vms=1.9G, shared=132.6M, text=95.0M, lib=0B,
              data=816.5M, dirty=0B
memory %      3.26
user          giampaolo
uids          real=1000, effective=1000, saved=1000
terminal      /dev/pts/2
status        sleeping
nice          0
ionice        class=IOPriority.IOPRIO_CLASS_NONE, value=0
num-threads   47
num-fds       379
I/O           read_count=96.6M, write_count=80.7M,
              read_bytes=293.2M, write_bytes=24.5G
ctx-switches  voluntary=30426463, involuntary=460108
children      PID    NAME
              4605   cat
              4606   cat
              4609   chrome
              4669   chrome
open-files    PATH
              /opt/google/chrome/icudtl.dat
              /opt/google/chrome/snapshot_blob.bin
              /opt/google/chrome/natives_blob.bin
              /opt/google/chrome/chrome_100_percent.pak
              [...]
connections   PROTO LOCAL ADDR            REMOTE ADDR               STATUS
              UDP   10.0.0.3:3693         *:*                       NONE
              TCP   10.0.0.3:55102        172.217.22.14:443         ESTABLISHED
              UDP   10.0.0.3:35172        *:*                       NONE
              TCP   10.0.0.3:32922        172.217.16.163:443        ESTABLISHED
              UDP   :::5353               *:*                       NONE
              UDP   10.0.0.3:59925        *:*                       NONE
threads       TID              USER          SYSTEM
              11795             0.7            1.35
              11796            0.68            1.37
              15887            0.74            0.03
              19055            0.77            0.01
              [...]
              total=47
res-limits    RLIMIT                     SOFT       HARD
              virtualmem             infinity   infinity
              coredumpsize                  0   infinity
              cputime                infinity   infinity
              datasize               infinity   infinity
              filesize               infinity   infinity
              locks                  infinity   infinity
              memlock                   65536      65536
              msgqueue                 819200     819200
              nice                          0          0
              openfiles                  8192      65536
              maxprocesses              63304      63304
              rss                    infinity   infinity
              realtimeprio                  0          0
              rtimesched             infinity   infinity
              sigspending               63304      63304
              stack                   8388608   infinity
mem-maps      RSS      PATH
              381.4M   [anon]
              62.8M    /opt/google/chrome/chrome
              15.8M    /home/giampaolo/.config/google-chrome/Default/History
              6.6M     /home/giampaolo/.config/google-chrome/Default/Favicons
              [...]

    Other changes

    The full list is in the changelog.

  2. Windows services support

    New psutil 4.2.0 is out. The highlight of this release is the support for Windows services (executables that run at system startup, similar to UNIX init scripts):

    >>> import psutil
>>> list(psutil.win_service_iter())
[<WindowsService(name='AeLookupSvc', display_name='Application Experience') at 38850096>,
 <WindowsService(name='ALG', display_name='Application Layer Gateway Service') at 38850128>,
 <WindowsService(name='APNMCP', display_name='Ask Update Service') at 38850160>,
 <WindowsService(name='AppIDSvc', display_name='Application Identity') at 38850192>,
 ...]
>>> s = psutil.win_service_get('alg')
>>> s.as_dict()
{'binpath': 'C:\\Windows\\System32\\alg.exe',
 'description': 'Provides support for 3rd party protocol plug-ins for Internet Connection Sharing',
 'display_name': 'Application Layer Gateway Service',
 'name': 'alg',
 'pid': None,
 'start_type': 'manual',
 'status': 'stopped',
 'username': 'NT AUTHORITY\\LocalService'}

    I decided to do this mainly because I find pywin32 APIs too low levelish. Having something like this in psutil can be useful to discover and monitor services more easily. The code was implemented in PR-803. The API for querying a service is similar to psutil.Process. You can get a reference to a service object by using its name (which is unique for every service) and then use methods like WindowsService.name() and WindowsService.status():

    >>> s = psutil.win_service_get('alg')
>>> s.name()
'alg'
>>> s.status()
'stopped'

    Initially I thought about providing a full set of APIs to handle all aspects of service management, including start(), stop(), restart(), install(), uninstall() and modify(). However, I soon realized I would have ended up reimplementing what pywin32 already provides, at the cost of overcrowding the psutil API (see my reasoning here). I think psutil really focuses on monitoring, not on installing and modifying system components, especially something as critical as a Windows service.

    Considerations about Windows services

    Typically, a Windows service is an executable (.exe) that runs at system startup and continues running in the background. It is roughly the equivalent of a UNIX init script. All services are controlled by a "manager", which keeps track of their status and metadata (e.g. description, startup type). It is interesting to note that since (most) services are bound to an executable (and hence a process) you can reference them via their process PID:

    >>> s = psutil.win_service_get('sshd')
>>> s
<WindowsService(name='sshd', display_name='Open SSH server') at 38853046>
>>> s.pid()
1865
>>> p = psutil.Process(1865)
>>> p
<psutil.Process(pid=19547, name='sshd.exe') at 140461487781328>
>>> p.exe()
'C:\CygWin\bin\sshd'

    Other improvements

    psutil 4.2.0 comes with 2 other enhancements for Linux:

    • psutil.virtual_memory() returns a new shared memory field. This is the same value reported by free cmdline utility.
    • I changed how /proc was parsed. Instead of reading /proc/{pid}/status line by line I used a regular expression. Here's the speedups:
      • Process.ppid() ~20% faster.
      • Process.status() ~28% faster.
      • Process.name() ~25% faster.
      • Process.num_threads() ~20% faster (on Python 3 only; on Python 2 it's a bit slower; I suppose the re module received some improvements).

    Discussion

  3. NetBSD support

    Roughly two months have passed since I last announced that psutil added support for OpenBSD. Today I'm happy to announce that it's the turn of NetBSD! This was contributed by Thomas Klausner, Ryo Onodera and myself in PR-557.

    Differences with FreeBSD (and OpenBSD)

    The NetBSD implementation has limitations similar to the ones I encountered with OpenBSD. Again, FreeBSD remains the BSD variant with the best support in terms of kernel functionality.

    • Process.memory_maps() is not implemented. The kernel provides the necessary pieces, but I haven't done this yet (hopefully later).
    • Process.num_ctx_switches()'s involuntary field is always 0. The kinfo_proc() syscall provides this info, but it's always set to 0.
    • Process.cpu_affinity() (get and set) is not supported.
    • psutil.cpu_count(logical=False) always returns None.

    As for the rest: it is all there. All memory, disk, network and process APIs are fully supported and functioning.

    Other enhancements available in this psutil release

    Besides NetBSD support, this release has a couple of interesting enhancements:

    • #708: [Linux] psutil.net_connections() and Process.connections() can be up to 3x faster when there are many connections.
    • #718: psutil.process_iter() is now thread safe.

    You can read the rest in the changelog, as usual.

    Move to Prague

    As a personal note I'd like to add that I'm currently in Prague (Czech Republic) and I'm thinking about moving down here for a while.

    Discussion

  4. Real process memory and environ in Python

    psutil 4.0.0 is out, with some interesting news about process memory metrics. I'll get straight to the point and describe what's new.

    "Real" process memory info

    Determining how much memory a process really uses is not an easy matter (see this and this). RSS (Resident Set Size), which most people rely on, is misleading because it includes both memory unique to the process and memory shared with others. What's more interesting for profiling is the memory that would be freed if the process were terminated right now. In the Linux world this is called USS (Unique Set Size), the major feature introduced in psutil 4.0.0 (not only for Linux but also for Windows and macOS).

    USS memory

    The USS (Unique Set Size) is the memory unique to a process, that would be freed if the process were terminated right now. On Linux it can be determined by parsing the "private" blocks in /proc/PID/smaps. The Firefox team pushed this further and got it working on macOS and Windows too.

    >>> psutil.Process().memory_full_info()
pfullmem(rss=101990, vms=521888, shared=38804, text=28200, lib=0, data=59672, dirty=0, uss=81623, pss=91788, swap=0)

    PSS and swap

    On Linux there are two additional metrics that can also be determined via /proc/PID/smaps: PSS and swap.

    pss, aka "Proportional Set Size", represents the amount of memory shared with other processes, accounted so that the amount is divided evenly between the processes that share it. I.e. if a process has 10 MBs all to itself (USS) and 10 MBs shared with another process, its PSS will be 15 MBs.

    swap is simply the amount of memory that has been swapped out to disk. With Process.memory_full_info() it is possible to implement a tool like procsmem.py, similar to smem on Linux, which provides a list of processes sorted by uss. It's interesting to see how rss differs from uss:

    ~/svn/psutil$ ./scripts/procsmem.py
PID     User    Cmdline                            USS     PSS    Swap     RSS
==============================================================================
...
3986    giampao /usr/bin/python3 /usr/bin/indi   15.3M   16.6M      0B   25.6M
3906    giampao /usr/lib/ibus/ibus-ui-gtk3       17.6M   18.1M      0B   26.7M
3991    giampao python /usr/bin/hp-systray -x    19.0M   23.3M      0B   40.7M
3830    giampao /usr/bin/ibus-daemon --daemoni   19.0M   19.0M      0B   21.4M
20529   giampao /opt/sublime_text/plugin_host    19.9M   20.1M      0B   22.0M
3990    giampao nautilus -n                      20.6M   29.9M      0B   50.2M
3898    giampao /usr/lib/unity/unity-panel-ser   27.1M   27.9M      0B   37.7M
4176    giampao /usr/lib/evolution/evolution-c   35.7M   36.2M      0B   41.5M
20712   giampao /usr/bin/python -B /home/giamp   45.6M   45.9M      0B   49.4M
3880    giampao /usr/lib/x86_64-linux-gnu/hud/   51.6M   52.7M      0B   61.3M
20513   giampao /opt/sublime_text/sublime_text   65.8M   73.0M      0B   87.9M
3976    giampao compiz                          115.0M  117.0M      0B  130.9M
32486   giampao skype                           145.1M  147.5M      0B  149.6M

    Implementation

    To get these values (uss, pss and swap) we need to walk the whole process address space. This usually requires higher privileges and is considerably slower than Process.memory_info(), which is probably why tools like ps and top show RSS/VMS instead of USS. A big thanks goes to the Mozilla team for figuring this out on Windows and macOS, and to Eric Rahm who put the psutil PRs together (see PR-744, PR-745 and PR-746). If you don't use Python and want to port the code to another language, here are the interesting parts:

    Memory type percent

    After reorganizing the process memory APIs (PR-744), I added a new memtype parameter to Process.memory_percent(). You can now compare a specific memory type (not only RSS) against the total physical memory. E.g.

    >>> psutil.Process().memory_percent(memtype='pss')
0.06877466326787016

    Process environ

    The second biggest improvement in psutil 4.0.0 is the ability to read a process's environment variables. This opens up interesting possibilities for process recognition and monitoring. For instance, you can start a process with a custom environment variable, then iterate over all processes to find the one of interest:

    import psutil
for p in psutil.process_iter():
    try:
        env = p.environ()
    except psutil.Error:
        pass
    else:
        if 'MYAPP' in env:
            ...

    Process environ was a long-standing issue (#52, from 2009) that I gave up on because the Windows implementation only worked for the current process. Frank Benkstein solved that (PR-747), and it now works on Linux, Windows and macOS for all processes (you may still hit AccessDenied for processes owned by another user):

    >>> import psutil
>>> from pprint import pprint as pp
>>> pp(psutil.Process().environ())
{...
 'CLUTTER_IM_MODULE': 'xim',
 'COLORTERM': 'gnome-terminal',
 'COMPIZ_BIN_PATH': '/usr/bin/',
 'HOME': '/home/giampaolo',
 'PWD': '/home/giampaolo/svn/psutil',
  }
>>>

    Note that the resulting dict usually doesn't reflect changes made after the process started (e.g. os.environ['MYAPP'] = '1'). Again, for anyone porting this to other languages, here are the interesting parts:

    Extended disk IO stats

    psutil.disk_io_counters() now reports additional metrics on Linux and FreeBSD:

    • busy_time: the time spent doing actual I/Os (in milliseconds).
    • read_merged_count and write_merged_count (Linux only): the number of merged reads and writes (see the iostats doc).

    These give a better picture of actual disk utilization (#756), similar to the iostat command on Linux.

    OS constants

    Given the growing number of platform-specific metrics, I added a set of constants to tell which platform you're on: psutil.LINUX, psutil.WINDOWS, etc.

    Other fixes

    The complete list of changes is available in the changelog.

    Porting code

    Since 4.0.0 is a major version, I took the chance to (lightly) change / break some APIs.

    • Process.memory_info() no longer returns just an (rss, vms) namedtuple. It returns a variable-length namedtuple that varies by platform (rss and vms are always present, even on Windows). Essentially the same result as the old Process.memory_info_ex(). This shouldn't break your code unless you were doing rss, vms = p.memory_info().
    • Process.memory_info_ex() is deprecated. It still works as an alias for Process.memory_info(), issuing a DeprecationWarning.
    • psutil.disk_io_counters() on NetBSD and OpenBSD no longer returns write_count and read_count because the kernel doesn't provide them (we were returning the busy time instead). Should be a small issue given NetBSD and OpenBSD support is very recent.

    Discussion

  5. OpenBSD support

    Starting from version 3.3.0 (released just now) psutil officially supports OpenBSD. This was contributed by Landry Breuil and myself in PR-615.

    Differences with FreeBSD

    As expected, the OpenBSD implementation is very similar to FreeBSD's, so I merged most of it into a single C file (_psutil_bsd.c), with 2 separate files (freebsd.c, openbsd.c) for the parts that differ. Here are the functional differences with FreeBSD:

    • Process.memory_maps() is not implemented. The kernel provides the necessary pieces but I haven't done this yet (hopefully later).
    • Process.num_ctx_switches()'s involuntary field is always 0. kinfo_proc provides this info but it is always set to 0.
    • Process.cpu_affinity() (get and set) is not supported.
    • Process.exe() is determined by inspecting the command line, so it may not always be available (returns None).
    • psutil.swap_memory() sin and sout (swap in and swap out) values are not available, and are therefore always set to 0.
    • psutil.cpu_count(logical=False) always returns None.

    As with FreeBSD, Process.open_files() can't return file paths (FreeBSD can sometimes). Otherwise everything is there and I'm satisfied with the result.

    Considerations about BSD platforms

    psutil has supported FreeBSD since the beginning (year 2009). At the time, it made sense to prefer it as the preferred BSD variant as it's the most popular.

    Compared to FreeBSD, OpenBSD appears to be more "minimal", both in terms of kernel facilities and the number of CLI tools available. One thing I particularly appreciate about FreeBSD is that the source code for all CLI tools is available under /usr/src, which was a big help when implementing psutil APIs.

    OpenBSD source code is also available, but it uses CVS and I am not sure it includes the source code for all CLI tools.

    There are still two more BSD variants worth supporting: NetBSD and DragonFlyBSD (in this order). About a year ago, someone provided a patch adding basic NetBSD support, so that will likely happen sooner or later.

    Other enhancements available in this release

    The only other enhancement is #558, which allows specifying a different location for the /proc filesystem on Linux.

    Discussion

Social

Feed