Blog posts for tags/personal

  1. System load average on Windows in Python

    psutil 5.6.2 is out. It implements an emulation of os.getloadavg() on Windows, kindly contributed by Ammar Askar, who originally implemented it for CPython's test suite.

    This idea has been floating around for quite a while. The first proposal dates back to 2010, when psutil was still hosted on Google Code, and it popped up multiple times over the years. There's a bunch of info online mentioning the pieces you'd theoretically use (the so-called System Processor Queue Length), but I couldn't find any real implementation. A quick search suggests there's real demand for this, but very few tools provide it natively (the only ones I could find are sFlowTrend and Zabbix). So I'm glad this finally landed in psutil / Python.

    Other improvements and bugfixes in psutil 5.6.2

    The full list is in the changelog. A couple worth mentioning:

    • #1476: ability to set a process's high I/O priority on Windows.
    • #1458: colorized test output. Nobody will use this directly, but it's nice and I'm porting it to other projects I maintain (e.g. pyftpdlib). Good candidate for a small PyPI module that could also include the unittest extensions I've been re-implementing piece by piece:
      • #1478: re-running failed tests.
      • display test timings / durations. This is something I'm also contributing to CPython: BPO-4080 and PR-12271.

    About me

    I'm currently in China (Shenzhen) for a mix of vacation and work, and I will likely take a break from Open Source for a while (about 2.5 months), during which I'll also go to the Philippines and Japan.

    External

  2. AIX support

    After a long wait psutil finally supports a new exotic platform: AIX!

    Honestly I'm not sure how many AIX Python users are out there (probably not many), but here it is.

    For this we have to thank Arnon Yaari, who started working on the port a couple of years ago (#605). I was skeptical at first, because AIX is the only platform I can't virtualize and test on my laptop, so that made me a bit nervous. Arnon did a great job. The final PR-1123 is huge: it required a considerable amount of work on his part, and a review of more than 140 messages exchanged between us over about a month, during which I was travelling through China.

    The end result is very good: almost all original unit tests pass, and code quality is awesome, which (I must say) is fairly unusual for an external contribution like this. Kudos to you, Arnon! ;-)

    Other changes

    Besides AIX support, release 5.4.0 also includes a couple of important bug fixes for psutil.sensors_temperatures() and psutil.sensors_fans() on Linux, and a fix for a bug on macOS that could cause a segmentation fault when using Process.open_files(). The complete list of bug fixes is in the changelog.

    The future

    Looking ahead at other exotic, still-unsupported platforms, two contributions are worth mentioning: a (still incomplete) PR for Cygwin which looks promising (PR-998), and Mingw32 compiler support on Windows (PR-845).

    psutil is gradually reaching a point where adding new features is becoming rarer, so it's a good moment to welcome new platforms while the API is mature and stable.

    Future work along these lines could also include Android and (hopefully) iOS support. Now that would be really awesome to have.

    Stay tuned.

    Discussion

  3. Announcing psutil 5.3.0

    psutil 5.3.0 is finally out. This release is a major one, bigger than any release before it in terms of improvements and bugfixes. It is interesting to notice how huge the diff between 5.2.2 and 5.3.0 is. This is because I've been travelling quite a lot this year, so I kept postponing it. It may sound weird but I consider publishing a new release and writing a blog post about it more stressful than working on the release itself. =). Anyway, here goes.

    Full Unicode support

    String-returning APIs (Process.exe(), Process.cwd(), Process.username(), etc.) are now Unicode-correct on both Python 2 and 3 (#1040), see detailed separate blog post.

    Improved process_iter()

    process_iter() now accepts attrs and ad_value parameters, letting you pre-fetch process attributes in one shot and skip the try/except NoSuchProcess boilerplate, see detailed separate blog post.

    Automatic overflow handling of numbers

    On very busy or long-lived systems, numbers returned by disk_io_counters() and net_io_counters() functions may wrap (restart from zero). Up to version 5.2.x you had to take this into account, while now this is automatically handled by psutil (see: #802). If a "counter" restarts from 0 psutil will add the value from the previous call for you so that numbers will never decrease. This is crucial for applications monitoring disk or network I/O in real time. Old behavior can be resumed by passing the nowrap=True argument.

    SunOS Process environ()

    Process.environ() is now available also on SunOS (see #1091).

    Other improvements and bug fixes

    Amongst others, here are a couple of important bug fixes I'd like to mention:

    • #1044: on OSX different Process methods could incorrectly raise AccessDenied for zombie processes. This was due to the poor proc_pidpath OSX API.
    • #1094: on Windows, pid_exists() may lie due to the poor OpenProcess Windows API which can return a handle even when a process PID no longer exists. This had repercussions for many Process methods such as cmdline(), environ(), cwd(), connections() and others which could have unpredictable behaviors such as returning empty data or erroneously raising NoSuchProcess exceptions. For the same reason (broken OpenProcess API), processes could unexpectedly stick around after using terminate() and wait().

    BSD systems also received some love (NetBSD and OpenBSD in particular). Different memory leaks were fixed and functions returning connected sockets were partially rewritten. The full list of enhancements and bug fixes can be seen here.

    About me

    I would like to spend a couple more words about my current situation. Last year (2016) I relocated to Prague and remote worked from there the whole year (it's been cool - great city!). This year I have mainly been resting in Turin (Italy) due to some health issues and travelling across Asia once I started to recover. I am currently in Shenzhen, China, and unless the current situation with North Korea gets worse I'm planning to continue my trip until November and visit Taiwan, South Korea and Japan. Once I'm finished, the plan is to briefly return to Turin (Italy) and finally return to Prague. By then I will probably be looking for a new (remote) gig again, so if you have anything for me by November feel free to send me a message. ;-)

  4. Reimplementing ifconfig in Python

    Here we are. It's been a long time since my last blog post and my last psutil release. The reason? I've been travelling! I mean... a lot. I've spent 3 months in Berlin, 3 weeks in Japan and 2 months in New York City. While I was there I finally had the chance to meet my friend Jay Loden in person. We originally started working on psutil together 7 years ago.

    Back then I didn't know any C (and I'm still a terrible C developer), so he was crucial in developing the initial psutil skeleton, including macOS and Windows support. Needless to say that this release builds on that work.

    net_if_addrs()

    We're now able to list network interface addresses similarly to the ifconfig command on UNIX:

    >>> import psutil
    >>> from pprint import pprint
    >>> pprint(psutil.net_if_addrs())
    {'ethernet0': [snic(family=<AddressFamily.AF_INET: 2>,
                        address='10.0.0.4',
                        netmask='255.0.0.0',
                        broadcast='10.255.255.255'),
                   snic(family=<AddressFamily.AF_PACKET: 17>,
                        address='9c:eb:e8:0b:05:1f',
                        netmask=None,
                        broadcast='ff:ff:ff:ff:ff:ff')],
     'localhost': [snic(family=<AddressFamily.AF_INET: 2>,
                        address='127.0.0.1',
                        netmask='255.0.0.0',
                        broadcast='127.0.0.1'),
                   snic(family=<AddressFamily.AF_PACKET: 17>,
                        address='00:00:00:00:00:00',
                        netmask=None,
                        broadcast='00:00:00:00:00:00')]}
    

    This is limited to AF_INET (IPv4), AF_INET6 (IPv6) and AF_LINK (Ethernet) address families. If you want something more powerful (e.g. AF_BLUETOOTH) you can take a look at the netifaces extension. If you want to see how this is implemented, here's the code for POSIX and Windows:

    net_if_stats()

    This new function returns information about network interface cards:

    >>> import psutil
    >>> from pprint import pprint
    >>> pprint(psutil.net_if_stats())
    {'ethernet0': snicstats(isup=True,
                            duplex=<NicDuplex.NIC_DUPLEX_FULL: 2>,
                            speed=100,
                            mtu=1500),
     'localhost': snicstats(isup=True,
                            duplex=<NicDuplex.NIC_DUPLEX_UNKNOWN: 0>,
                            speed=0,
                            mtu=65536)}
    

    The implementation on each platform:

    Also in 3.0

    Beyond the network-interface APIs, psutil 3.0 ships a few other notable changes.

    Several integer/string constants (IOPRIO_CLASS_*, NIC_DUPLEX_*, *_PRIORITY_CLASS) now return enum values on Python 3.4+.

    Support for zombie processes on UNIX was broken. Covered in a separate post.

    Removal of deprecated APIs

    All aliases deprecated in the psutil 2.0 porting guide (January 2014) are gone. For the full list see the changelog.

    Final words

    I must say I'm pretty satisfied with how psutil is evolving and with the enjoyment I still get every time I work on it. It now gets almost 800,000 downloads a month, which is quite remarkable for a Python library.

    At this point, I consider psutil almost "complete" feature-wise, meaning I'm starting to run out of ideas for what to add next (see TODO). Going forward, development will likely focus on supporting more exotic platforms (OpenBSD #562, NetBSD PR-557, Android #355).

    There have also been discussions on the python-ideas mailing list about including psutil in the Python stdlib, but even if that happens, it's still a long way off, as it would require a significant time investment that I currently don't have.

  5. Windows wheels available in psutil 2.1.2

    psutil 2.1.2 is out. This release has been cooking for a while now, and that's because I've been travelling for the past 3 months between Spain, Japan and Germany. Hopefully I will be staying in Berlin for a while now, so I will have more time to dedicate to the project. The main new "feature" of this release is that other than the exe files, Windows users can now also benefit from Python wheels (full story is here) which are available on PYPI. Frankly I don't know much about the new wheels packaging system but long story short is that Windows users can now install psutil via pip and therefore also include it as a dependency in requirements.txt. Other than this 2.1.2 can basically be considered a bug-fix release, including some important fixes amongst which:

    • #506: restored Python 2.4 compatibility
    • #340: Process.get_open_files() no longer hangs on Windows (this was a very old and high-priority issue)
    • #501: disk_io_counters() may return negative values on Windows
    • #504: (Linux) couldn't build RPM packages via setup.py

    The list of all fixes can be found here. For the next release I plan to drop support for Python 2.4 and 2.5 and hopefully add network interfaces information similarly to ifconfig.

Social

Feed