Blog posts for tags/psutil

  1. New Pelican website

    Hello there. Here is my new blog / personal website! This is something I've been wanting to do for a very long time, since the old blog hosted at https://grodola.blogspot.com/ was... well, too old. =) This new site is based on Pelican, a static website generator similar to Jekyll. Unlike Jekyll, it uses Python instead of Ruby, and that's why I chose it. It's minimal, straight to the point and I feel I have control of things. This is what Pelican gave me out of the box:

    • blog functionality
    • ability to write content by using reStructuredText
    • RSS & Atom feed
    • easy integration with GitHub pages
    • ability to add comments via Disqus

    To this I added a mailing list (I used feedburner), so that users can subscribe and receive an email every time I make a new blog post. As you can see the website is very simple, but it's exactly what I wanted (minimalism). As for the domain name I opted for gmpy.dev, mostly because I know my name is hard to type and pronounce for non-English speakers. And also because I couldn't come up with a better name. ;)

    GIT-based workflow

    The main reason I blogged so rarely over the years was that blogger.com provided me no way to edit content in reST or markdown, and lacked GIT integration. This made me lazy. With Pelican + GitHub pages the workflow to create and publish new content is very straightforward. I use 2 branches: gh-pages, which is the source code of this web-site, and master, which is where the generated HTML content lives and is being served by GitHub pages. This is what I do when I have to create a new blog post:

    • under gh-pages branch I create a new file, e.g. content/blog/2020/new-blog-post.rst:
    New blog post
    #############
    
    :date: 2020-06-26
    :tags: announce, python
    
    Hello world!
    
    • commit it:
    git add content/blog/2020/new-blog-post.rst
    git ci -am "new blog post"
    git push
    
    • publish it:
    make github
    

    Within a minute or so, GitHub will automatically serve gmpy.dev with the updated content. And this is why I think I will start blogging more often. =) The core of Pelican is pelicanconf.py, which lets you customize a lot of things by remaining independent from the theme. I still ended up modifying the default theme though, writing a customized "archives" view and editing CSS to make the site look better on mobile phones. All in all, I am very satisfied with Pelican, and I'm keen on recommending it to anyone who doesn't need dynamic content.

    About me

    I spent most of last year (2019) in China, dating my girlfriend, working remotely from a shared office space in Shenzhen, and I even taught some Python to a class of Chinese folks with no previous programming experience. The course was about the basics of the language plus basic filesystem operations, and the most difficult thing to explain was indentation. I guess that shows the difference between knowing a language and knowing how to teach it.

    I got back to Italy in December 2019, just before the pandemic occurred. Because of my connections with China, I knew about the incoming pandemic sooner than the rest of my friends, which for a while (until the lockdown) made them think I was crazy. =)

    Since I knew I would be stuck at home for a long time, I bought a quite nice acoustic guitar (Taylor) after many years, and resumed playing (and singing). I learned a bunch of new songs, mostly by Queen, including Bohemian Rhapsody, which is something I've been wanting to do since forever.

    I also spent some time working on a personal project that I'm keeping private for the moment, something to speed up file copies, which follows the experiments I made in BPO-33671. It's still very beta, but I managed to make file copies around 170% faster compared to the cp command on Linux, which is pretty impressive (and I think I can push it even further). I will blog about that once I have something more solid / stable. Most likely it'll become my next OSS project, even though I have mixed feelings about that, since the amount of time I'm devoting to psutil is already a lot.

    Speaking of which, today I'm also releasing psutil 5.7.1, which adds support for Windows Nano.

    I guess this is all. Cheers and subscribe!

  2. System load average on Windows in Python

    psutil 5.6.2 is out. It implements an emulation of os.getloadavg() on Windows, kindly contributed by Ammar Askar, who originally implemented it for CPython's test suite.

    This idea has been floating around for quite a while. The first proposal dates back to 2010, when psutil was still hosted on Google Code, and it popped up multiple times over the years. There's a bunch of info online mentioning the pieces you'd theoretically use (the so-called System Processor Queue Length), but I couldn't find any real implementation. A quick search suggests there's real demand for this, but very few tools provide it natively (the only ones I could find are sFlowTrend and Zabbix). So I'm glad this finally landed in psutil / Python.

    Other improvements and bugfixes in psutil 5.6.2

    The full list is in the changelog. A couple worth mentioning:

    • #1476: ability to set a process's high I/O priority on Windows.
    • #1458: colorized test output. Nobody will use this directly, but it's nice and I'm porting it to other projects I maintain (e.g. pyftpdlib). Good candidate for a small PyPI module that could also include the unittest extensions I've been re-implementing piece by piece:
      • #1478: re-running failed tests.
      • display test timings / durations. This is something I'm also contributing to CPython: BPO-4080 and PR-12271.

    About me

    I'm currently in China (Shenzhen) for a mix of vacation and work, and I will likely take a break from Open Source for a while (about 2.5 months), during which I'll also go to the Philippines and Japan.

    External

  3. Announcing psutil 5.6.0

    psutil 5.6.0 is out. Highlights: a new Process.parents() method, several important Windows improvements, and the removal of Process.memory_maps() on macOS.

    Process parents()

    The new method returns the parents of a process as a list of Process instances. If no parents are known, an empty list is returned.

    >>> import psutil
    >>> p = psutil.Process(5312)
    >>> p.parents()
    [psutil.Process(pid=4699, name='bash', started='09:06:44'),
     psutil.Process(pid=4689, name='gnome-terminal-server', started='09:06:44'),
     psutil.Process(pid=1, name='systemd', started='05:56:55')]
    

    Nothing fundamentally new here, since this is a convenience wrapper around Process.parent(), but it's still nice to have it built in. It pairs well with Process.children() when working with process trees. The idea was proposed by Ghislain Le Meur.

    Windows

    Certain Windows APIs that need to be dynamically loaded from DLLs are now loaded only once at startup, instead of on every function call. This makes some operations 50% to 100% faster; see benchmarks in PR-1422.

    Process.suspend() and Process.resume() previously iterated over all process threads via CreateToolhelp32Snapshot(), which was unorthodox and broke when the process had been suspended by Process Hacker. They now call the undocumented NtSuspendProcess() / NtResumeProcess() NT APIs, same as Process Hacker and Sysinternals tools. Discussed in #1379, implemented in PR-1435.

    SE DEBUG is a privilege bit set on the Python process at startup so psutil can query processes owned by other users (Administrator, Local System), meaning fewer AccessDenied exceptions for low-PID processes. The code setting it had presumably been broken for years and is now finally fixed in PR-1429.

    Removal of Process.memory_maps() on macOS

    Process.memory_maps() is gone on macOS (#1291). The underlying Apple API would randomly raise EINVAL or segfault the host process, and no amount of reverse-engineering produced a safe fix. So I removed it. This is covered in a separate post.

    Improved exceptions

    One problem that affected psutil maintenance over the years was receiving bug reports whose tracebacks did not indicate which syscall had actually failed. This was especially painful on Windows, where a single routine may invoke multiple Windows APIs. Now the OSError (or WindowsError) exception includes the syscall from which the error originated. See PR-1428.

    Other changes

    See the changelog.

  4. Removing Process.memory_maps() on macOS

    This is part of the psutil 5.6.0 release (see the full release notes).

    As of 5.6.0, Process.memory_maps() is no longer defined on macOS.

    The bug

    #1291: on macOS, Process.memory_maps() would either raise OSError: [Errno 22] Invalid argument or segfault the whole Python process! Both triggered from code as simple as psutil.Process().as_dict(), since Process.as_dict() iterates every attribute, and Process.memory_maps() is one of them.

    The root cause was inside Apple's undocumented proc_regionfilename() syscall. On some memory regions it returns EINVAL. On others it takes the process down. Which regions? Nobody figured out. Arnon Yaari (@wiggin15) did most of the investigation: he wrote a standalone C reproducer and walked me through what he'd tried.

    In PR-1436 I attempted a fix by reverse-engineering vmmap(1) but it didn't work. The fundamental problem is that vmmap is closed source and proc_regionfilename is undocumented. Neither my virtualized macOS (10.11.6) nor Travis CI (10.12.1) could reproduce the bug, which reproduced reliably only on 10.14.3.

    Why remove outright

    While removing the C code I noticed that the macOS unit test had been disabled long ago, presumably by me after recurring flaky Travis runs. Meaning that the method had been broken on some macOS versions far longer than the 2018 bug report suggested.

    Deprecating for a cycle didn't help either: raising AccessDenied breaks code that relied on a successful return, returning an empty list does the same silently, and leaving the method in place doesn't stop the segfault. Basically there was no sane solution, so since 5.6 is a major version I decided to just remove Process.memory_maps() for good.

    On macOS it never supported other processes anyway. Calling it on any PID other than the current one (or its children) raised AccessDenied, even as root.

    If someone finds a Mach API path that works, the method can return. Nobody has found one so far.

  5. AIX support

    After a long wait psutil finally supports a new exotic platform: AIX!

    Honestly I'm not sure how many AIX Python users are out there (probably not many), but here it is.

    For this we have to thank Arnon Yaari, who started working on the port a couple of years ago (#605). I was skeptical at first, because AIX is the only platform I can't virtualize and test on my laptop, so that made me a bit nervous. Arnon did a great job. The final PR-1123 is huge: it required a considerable amount of work on his part, and a review of more than 140 messages exchanged between us over about a month, during which I was travelling through China.

    The end result is very good: almost all original unit tests pass, and code quality is awesome, which (I must say) is fairly unusual for an external contribution like this. Kudos to you, Arnon! ;-)

    Other changes

    Besides AIX support, release 5.4.0 also includes a couple of important bug fixes for psutil.sensors_temperatures() and psutil.sensors_fans() on Linux, and a fix for a bug on macOS that could cause a segmentation fault when using Process.open_files(). The complete list of bug fixes is in the changelog.

    The future

    Looking ahead at other exotic, still-unsupported platforms, two contributions are worth mentioning: a (still incomplete) PR for Cygwin which looks promising (PR-998), and Mingw32 compiler support on Windows (PR-845).

    psutil is gradually reaching a point where adding new features is becoming rarer, so it's a good moment to welcome new platforms while the API is mature and stable.

    Future work along these lines could also include Android and (hopefully) iOS support. Now that would be really awesome to have.

    Stay tuned.

    Discussion

Social

Feed