About dropping Python 2.7 support in psutil, 3 years ago I stated:

Not a chance, for many years to come. [Python 2.7] currently represents 7-10% of total downloads, meaning around 70k / 100k downloads per day.

Only 3 years later, and to my surprise, downloads for Python 2.7 dropped to 0.36%! As such, as of psutil 7.0.0, I finally decided to drop support for Python 2.7!

The numbers

These are downloads per month:

$ pypinfo --percent psutil pyversion
Served from cache: False
Data processed: 4.65 GiB
Data billed: 4.65 GiB
Estimated cost: $0.03

| python_version | percent | download_count |
| -------------- | ------- | -------------- |
| 3.10           |  23.84% |     26,354,506 |
| 3.8            |  18.87% |     20,862,015 |
| 3.7            |  17.38% |     19,217,960 |
| 3.9            |  17.00% |     18,798,843 |
| 3.11           |  13.63% |     15,066,706 |
| 3.12           |   7.01% |      7,754,751 |
| 3.13           |   1.15% |      1,267,008 |
| 3.6            |   0.73% |        803,189 |
| 2.7            |   0.36% |        402,111 |
| 3.5            |   0.03% |         28,656 |
| Total          |         |    110,555,745 |

According to pypistats.org Python 2.7 downloads represents the 0.28% of the total, around 15.000 downloads per day.

The pain

Maintaining 2.7 support in psutil had become increasingly difficult, but still possible. E.g. I could still run tests by using old PYPI backports. GitHub Actions could still be tweaked to run tests and produce 2.7 wheels on Linux and macOS. Not on Windows though, for which I had to use a separate service (Appveyor). Still, the amount of hacks in psutil source code necessary to support Python 2.7 piled up over the years, and became quite big. Some disadvantages that come to mind:

• Having to maintain a Python compatibility layers like psutil/_compat.py. This translated in extra extra code and extra imports.
• The C compatibility layer to differentiate between Python 2 and 3 (#if PY_MAJOR_VERSION <= 3, etc.).
• Dealing with the string vs. unicode differences, both in Python and in C.
• Inability to use modern language features, especially f-strings.
• Inability to freely use enums, which created a difference on how CONSTANTS were exposed in terms of API.
• Having to install a specific version of pip and other (outdated) deps.
• Relying on the third-party Appveyor CI service to run tests and produce 2.7 wheels.
• Running 4 extra CI jobs on every commit (Linux, macOS, Windows 32-bit, Windows 64-bit) making the CI slower and more subject to failures (we have quite a bit of flaky tests).
• The distribution of 7 wheels specific for Python 2.7. E.g. in the previous release I had to upload:

psutil-6.1.1-cp27-cp27m-macosx_10_9_x86_64.whl
psutil-6.1.1-cp27-none-win32.whl
psutil-6.1.1-cp27-none-win_amd64.whl
psutil-6.1.1-cp27-cp27m-manylinux2010_i686.whl
psutil-6.1.1-cp27-cp27m-manylinux2010_x86_64.whl
psutil-6.1.1-cp27-cp27mu-manylinux2010_i686.whl
psutil-6.1.1-cp27-cp27mu-manylinux2010_x86_64.whl

The removal

The removal was done in PR-2841, which removed around 1500 lines of code (nice!). It felt liberating. In doing so, in the doc I still made the promise that the 6.1.* serie will keep supporting Python 2.7 and will receive critical bug-fixes only (no new features). It will be maintained in a specific python2 branch. I explicitly kept the setup.py script compatible with Python 2.7 in terms of syntax, so that, when the tarball is fetched from PYPI, it will emit an informative error message on pip install psutil. The user trying to install psutil on Python 2.7 will see:

$ pip2 install psutil
As of version 7.0.0 psutil no longer supports Python 2.7.
Latest version supporting Python 2.7 is psutil 6.1.X.
Install it with: "pip2 install psutil==6.1.*".

As the informative message states, users that are still on Python 2.7 can still use psutil with:

pip2 install psutil==6.1.*

Related tickets

• 2017-06: #1053
• 2022-04: #2099
• 2023-04: #2246
• 2024-12: PR-635

Lately I've been dealing with an asynchronous TCP client app which sends messages to a remote server. Some of these messages are important, and cannot get lost. Because the connection may drop at any time, I had to implement a mechanism to resend the message once the client reconnects. As such, I needed a way to identify what constitutes a connection error.

Python provides a builtin ConnectionError exception precisely for this purpose, but it turns out it's not enough. After observing logs in production, I found some errors that were not related to the socket connection per se, but rather to the system connectivity, like ENETUNREACH ("network unreachable") or ENETDOWN ("network down"). It's interesting to note how this distinction is reflected in the UNIX errno code prefixes: ECONN* (connection errors) vs. ENET* (network errors). I've noticed ENET* errors usually occur on a DHCP renewal, or more in general when the Wi-Fi signal is weak or absent. Because this code runs on a cleaning robot which constantly moves around the house, connection can become unstable when the robot gets far from the Wi-Fi Access Point, so it's pretty common to bump into errors like these:

File "/usr/lib/python3.7/ssl.py", line 934, in send
    return self._sslobj.write(data)
OSError: [Errno 101] Network is unreachable

File "/usr/lib/python3.7/socket.py", line 222, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

File "/usr/lib/python3.7/ssl.py", line 934, in send
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

File "/usr/lib/python3.7/ssl.py", line 934, in send
    return self._sslobj.write(data)
socket.timeout: The write operation timed out

Production logs also revealed a considerable amount of SSL-related errors. I was uncertain what to do about those. The app is supposed to gracefully handle them, so theoretically they should represent a bug. Still, they are unequivocally related to the connection stream, and represent a failed attempt to send data, so we want to retry it. Example of logs I found:

File "/usr/lib/python3.7/ssl.py", line 934, in send
    return self._sslobj.write(data)
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF)

File "/usr/lib/python3.7/ssl.py", line 934, in send
    return self._sslobj.write(data)
ssl.SSLError: [SSL: BAD_LENGTH] bad length

Looking at production logs revealed what sort of brutal, rough and tumble place the Internet is, and how a network app must be ready to handle all sorts of unexpected error conditions which hardly show up during testing. To handle all of these cases I came up with this solution which I think is worth sharing, as it's generic enough to be reused in similar situations. If needed, this can be easily extended to include specific exceptions of third party libraries, like requests.exceptions.ConnectionError.

import errno, socket, ssl

# Network errors, usually related to DHCP or wpa_supplicant (Wi-Fi).
NETWORK_ERRNOS = frozenset((
    errno.ENETUNREACH,  # "Network is unreachable"
    errno.ENETDOWN,  # "Network is down"
    errno.ENETRESET,  # "Network dropped connection on reset"
    errno.ENONET,  # "Machine is not on the network"
))

def is_connection_err(exc):
    """Return True if an exception is connection-related."""
    if isinstance(exc, ConnectionError):
        # https://docs.python.org/3/library/exceptions.html#ConnectionError
        # ConnectionError includes:
        # * BrokenPipeError (EPIPE, ESHUTDOWN)
        # * ConnectionAbortedError (ECONNABORTED)
        # * ConnectionRefusedError (ECONNREFUSED)
        # * ConnectionResetError (ECONNRESET)
        return True
    if isinstance(exc, socket.gaierror):
        # failed DNS resolution on connect()
        return True
    if isinstance(exc, (socket.timeout, TimeoutError)):
        # timeout on connect(), recv(), send()
        return True
    if isinstance(exc, OSError):
        # ENOTCONN == "Transport endpoint is not connected"
        return (exc.errno in NETWORK_ERRNOS) or (exc.errno == errno.ENOTCONN)
    if isinstance(exc, ssl.SSLError):
        # Let's consider any SSL error a connection error. Usually this is:
        # * ssl.SSLZeroReturnError: "TLS/SSL connection has been closed"
        # * ssl.SSLError: [SSL: BAD_LENGTH]
        return True
    return False

To use it:

try:
    sock.sendall(b"hello there")
except Exception as err:
    if is_connection_err(err):
        schedule_on_reconnect(lambda: sock.sendall(b"hello there"))
    raise

My editor of choice for Python development is Sublime Text. It has been for a very long time (10 years). It's fast, minimalist and straight to the point, which is why I always resisted the temptation to use more advanced and modern IDEs such as PyCharm or VS code, which admittedly have superior auto-completion and refactoring tools.

There is a very simple feature I've always missed in ST: the possibility to "remember" / save the cursor position when a file is closed. The only plugin promising to do such a thing is called BufferScroll, but for some reason it ceased working for me at some point. I spent a considerable amount of time Googling for an alternative but, to my surprise, I couldn't find any plugin which implements such a simple feature. Therefore today I decided to bite the bullet and try to implement this myself, by writing my first ST plugin, which I paste below.

What it does is this:

• every time a file is closed, save the cursor position (x and y axis) to a JSON file
• if that same file is re-opened, restore the cursor at that position

What's neat about ST plugins is that they are just Python files which you can install by copying them in ST's config directory. On Linux you can copy the script below in:

~/.config/sublime-text-3/Packages/User/cursor_positions.py

...and will work out of the box. This is exactly the kind of minimalism which I love about ST, and which I've always missed in other IDEs.

# cursor_positions.py

"""
A plugin for SublimeText which saves (remembers) cursor position when
a file is closed.
Install it by copying this file in ~/.config/sublime-text-3/Packages/User/
directory (Linux).

Author: Giampaolo Rodola'
License: MIT
"""

import datetime
import json
import os
import tempfile
import threading

import sublime
import sublime_plugin


SUBLIME_ROOT = os.path.realpath(os.path.join(sublime.packages_path(), '..'))
SESSION_FILE = os.path.join(
    SUBLIME_ROOT, "Local", "cursor_positions.session.json")
# when reading the session file on startup, we'll remove entries
# older than X days
RM_FILE_OLDER_THAN_DAYS = 180


def log(*args):
    print("    %s: " % os.path.basename(__file__), end="")
    print(*args)


class Session:

    def __init__(self):
        self._lock = threading.Lock()
        os.makedirs(os.path.dirname(SESSION_FILE), exist_ok=True)
        self.prune_old_entries()

    # --- file

    def read_session_file(self):
        try:
            with self._lock:
                with open(SESSION_FILE, "r") as f:
                    return json.load(f)
        except (FileNotFoundError, json.decoder.JSONDecodeError):
            return {}

    def write_session_file(self, d):
        # Use the same FS so that the move operation is atomic:
        # https://stackoverflow.com/a/18706666
        with tempfile.NamedTemporaryFile(
                "wt", delete=False, dir=os.path.dirname(SESSION_FILE)) as f:
            f.write(json.dumps(d, indent=4, sort_keys=True))
        with self._lock:
            os.rename(f.name, SESSION_FILE)

    def prune_old_entries(self):
        old = self.read_session_file()
        new = old.copy()
        now = datetime.datetime.now()
        for file, entry in old.items():
            tstamp = entry["last_update"]
            last_update = datetime.datetime.strptime(
                tstamp, '%Y-%m-%d %H:%M:%S.%f')
            delta_days = (now - last_update).days
            if delta_days > RM_FILE_OLDER_THAN_DAYS:
                log("removing old saved file %r" % file)
                del new[file]
        if new != old:
            self.write_session_file(new)

    # --- operations

    def add_entry(self, file, x, y):
        d = self.read_session_file()
        d[file] = dict(
            x=x,
            y=y,
            last_update=str(datetime.datetime.now()),
        )
        self.write_session_file(d)

    def load_entry(self, file):
        d = self.read_session_file()
        try:
            return d[file]
        except KeyError:
            return None


session = Session()


class Events(sublime_plugin.EventListener):

    # --- utils

    @staticmethod
    def get_cursor_pos(view):
        x, y = view.rowcol(view.sel()[0].begin())
        return x, y

    @staticmethod
    def set_cursor_pos(view, x, y):
        pt = view.text_point(x, y)
        view.sel().clear()
        view.sel().add(sublime.Region(pt))
        view.show(pt)

    def save_cursor_position(self, view):
        file_name = view.file_name()
        if file_name is None:
            return  # non-existent file
        log("saving cursor position for %s" % file_name)
        x, y = self.get_cursor_pos(view)
        session.add_entry(file_name, x, y)

    def load_cursor_position(self, view):
        entry = session.load_entry(view.file_name())
        if entry:
            self.set_cursor_pos(view, entry["x"], entry["y"])

    # --- callbacks

    def on_close(self, view):
        # called when a file is closed
        self.save_cursor_position(view)

    def on_load(self, view):
        # called when a file is opened
        self.load_cursor_position(view)

Hello there. I present you my new blog / personal website! This is something I've been wanting to do for a very long time, since the old blog hosted at https://grodola.blogspot.com/ was... well, too old. =) This new site is based on Pelican, a static website generator similar to Jekyll. Differently from Jekyll, it uses Python instead of Ruby, and that's why I chose it. It's minimal, straight to the point and I feel I have control of things. This is what Pelican gave me out of the box:

• blog functionality
• ability to write content by using reStructuredText
• RSS & Atom feed
• easy integration with GitHub pages
• ability to add comments via Disqus

To this I added a mailing list (I used feedburner), so that users can subscribe and receive an email every time I make a new blog post. As you can see the website is very simple, but it's exactly what I wanted (minimalism). As for the domain name I opted for gmpy.dev, mostly because I know my name is hard to type and pronounce for non-english speakers. And also because I couldn't come up with a better name. ;)

New blog post
#############

:date: 2020-06-26
:tags: announce, python

Hello world!

git add content/blog/2020/new-blog-post.rst
git ci -am "new blog post"
git push

make github

New psutil 5.6.2 release implements an emulation of os.getloadavg() on Windows which was kindly contributed by Ammar Askar who originally implemented it for cPython's test suite. This idea has been floating around for quite a while. The first proposal dates back to 2010, when psutil was still hosted on Google Code, and it popped up multiple times throughout the years. There was/is a bunch of info on internet mentioning the bits with which it's theoretically possible to do this (the so called System Processor Queue Length), but I couldn't find any real implementation. A Google search tells there is quite some demand for this, but very few tools out there providing this natively (the only one I could find is this sFlowTrend tool and Zabbix), so I'm very happy this finally landed into psutil / Python.