Chapter 10

Cleanup

Python frees memory on its own, and a file, a socket, or a lock is not memory.

Why `del()` Is Not Cleanup

Python calls an object’s __del__() method when it destroys that object. That looks like the place to release resources:

# cleanup.py
from typing import ClassVar

class Counter:
    # Number of objects of this class
    count: ClassVar[int] = 0

    def __init__(self, name: str) -> None:
        self.name = name
        print(name, "created")
        Counter.count += 1

    def __del__(self) -> None:
        print(self.name, "deleted")
        Counter.count -= 1
        if Counter.count == 0:
            print("Last Counter object deleted")
        else:
            print(Counter.count,
                  "Counter objects remaining")

    def __repr__(self) -> str:
        return f"Counter({self.name!r} {self.count})"

counters = []
for name in ["First", "Second", "Third"]:
    counters.append(Counter(name))
#: First created
#: Second created
#: Third created

for c in counters:
    print(c)
    del c
#: Counter('First' 3)
#: Counter('Second' 3)
#: Counter('Third' 3)
print("End of delete loop")
#: End of delete loop

del c inside the loop does not delete the object. It unbinds the name c and drops the one reference that name held. The counters list still references each Counter, so no Counter’s reference count reaches zero during the loop. That is why no deleted lines appear while the loop runs, why every __repr__() reports a count of 3, and why End of delete loop prints before any deletion.

Python destroys the objects later, at interpreter shutdown, when it tears down the global counters list. That list holds the only remaining references, so when it goes, the objects it holds go with it. The listing ends at End of delete loop, the program’s last statement, and each __del__() prints afterward. If you run cleanup.py as a standalone program, three more pairs of lines follow End of delete loop:

Third deleted
2 Counter objects remaining
Second deleted
1 Counter objects remaining
First deleted
Last Counter object deleted

That was one run on one machine. This book’s output checker, which executes all of a chapter’s listings in one process, finalizes the same three objects in the opposite order.

The order in which the three __del__() methods run is an implementation detail. It depends on how the interpreter tears down the counters list at shutdown, and it can differ from one CPython build to the next. Another implementation, such as PyPy with a tracing garbage collector, could destroy the objects in a different order, or skip the finalizers at exit.

So __del__() is fragile. The language specifies neither when it runs nor whether it runs. At interpreter shutdown, the globals a __del__() method uses may have vanished. The Python documentation warns:

Warning: Due to the precarious circumstances under which __del__() methods are invoked, exceptions that occur during their execution are ignored, and a warning is printed to sys.stderr instead. In particular:

__del__() can be invoked when arbitrary code is being executed, including from any arbitrary thread. If __del__() needs to take a lock or invoke any other blocking resource, it may deadlock as the resource may already be taken by the code that gets interrupted to execute __del__().

__del__() can be executed during interpreter shutdown. As a consequence, the global variables it needs to access (including other modules) may already have been deleted or set to None. Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the __del__() method is called.

In the direct run, shutdown destroys the objects, and shutdown is the precarious moment the warning describes. Counter and print() were still available, so the output came out cleanly, but nothing guarantees the teardown order that allowed it. Make __del__() do as little as possible, and do not depend on it.

The swallowed exception is the failure that costs most in production:

# del_swallows.py

class Resource:
    def __init__(self, name: str) -> None:
        self.name = name

    def __del__(self) -> None:
        raise RuntimeError(f"{self.name} not released")

resource = Resource("db")
del resource
print("still running")
#: still running

The release fails, and stdout says nothing about it. A traceback goes to sys.stderr labeled Exception ignored, but nothing propagates. No caller can catch the RuntimeError, the exit status is still 0, and a test asserting on stdout passes. In a with block, an exception from close() propagates to the caller instead.

Reference Cycles Delay Destruction

A reference cycle postpones __del__() too. An object that refers to itself, directly or through another object, outlives the moment it becomes unreachable:

# cycle.py
import gc

class Node:
    peer: Node

    def __init__(self, name: str) -> None:
        self.name = name

    def __del__(self) -> None:
        print(self.name, "finalized")

def self_link() -> None:
    node = Node("a")
    node.peer = node
    print(gc.get_referrers(node)[0] is node)

gc.disable()
self_link()
#: True
print("unreachable, but still alive")
#: unreachable, but still alive
gc.collect()
#: a finalized
gc.enable()
print("after collect")
#: after collect

CPython frees most objects by counting references. When the last reference to an object goes away, the object goes with it. A reference cycle defeats that count. self_link() returns and its local node disappears, but the object still refers to itself, so its count stays above zero.

Inside self_link(), gc.get_referrers(node) returns the objects that refer directly to node, and the call destroys nothing. The only referrer is node, which confirms the self-reference. When a real object won’t disappear and you don’t know why, gc.get_referrers() is how you find what still holds it, the same way this listing uses it to show its own cycle.

Freeing the node takes the cyclic garbage collector, a separate mechanism that runs on allocation counts rather than when the object becomes unreachable. gc.disable() above keeps that collector from running on its own, and gc.collect() then forces a run, so the a finalized line marks the moment of destruction. In a real program nothing tells you when the collector runs. Before Python 3.4 the collector refused to finalize a cycle containing a __del__(), leaving the objects in gc.garbage as uncollectable garbage, the term older documentation uses for them. PEP 442 removed that restriction, so a cycle now costs only the delay.

Cycles are one more reason to keep cleanup out of __del__(). One back-reference between two objects is enough to postpone __del__(), and the code that creates the cycle often lives far from the code that owns the resource.

Reliable Alternatives

Two approaches are more reliable:

An Explicit `close()` and a `with` Block

The first is an explicit cleanup method, such as the close() that file objects provide, which a with block calls. Here a class supplies the two methods that with calls, and close() runs whether or not an error interrupts the block:

# closable.py

class Socket:
    def __init__(self, name: str) -> None:
        self.name = name
        self.closed = False
        print(name, "opened")

    def close(self) -> None:
        if self.closed:
            return
        self.closed = True
        print(self.name, "closed")

    def __enter__(self) -> Socket:
        return self

    def __exit__(self, *exc: object) -> None:
        self.close()

with Socket("A") as sock:
    print("using", sock.name)
#: A opened
#: using A
#: A closed
sock.close()
try:
    with Socket("B"):
        raise RuntimeError("boom")
except RuntimeError as e:
    print("caught", e)
#: B opened
#: B closed
#: caught boom

with calls __enter__() at the top of the block and __exit__() as control leaves the block, and __exit__() calls close(). So close() runs at the end of the with block, at a line you can identify, and the Socket("B") block shows close() running when the body raises an exception. Compare cleanup.py, where the cleanup runs at an unknowable moment after the program’s last statement. Context Managers covers the protocol, the @contextmanager shorthand, and what __exit__()’s arguments mean.

close() also guards against a second call. The explicit sock.close() after the with block prints nothing, because self.closed blocks the repeat. The with protocol calls close() for you once. Your own code can still call it again, so a real close() must do nothing on a repeat call, the way a file object’s close() does.

An `init()` That Fails Leaks the Resource

Socket.__init__() prints “opened” before __enter__() runs. That ordering hides a trap. If __init__() raises an exception after acquiring the resource, the with statement receives no context manager, so neither __enter__() nor __exit__() runs, and the resource leaks silently:

# faulty_init.py

class Faulty:
    def __init__(self, name: str) -> None:
        self.name = name
        print(name, "opened")
        raise RuntimeError("boom")

    def __enter__(self) -> Faulty:
        return self

    def __exit__(self, *exc: object) -> None:
        print(self.name, "closed")

try:
    with Faulty("C"):
        pass
except RuntimeError as e:
    print("caught", e)
#: C opened
#: caught boom

C opened has no matching closed. Nothing releases what __init__() acquired before it failed.

One fix keeps __init__() free of resources and acquires them in __enter__(), so a failed construction has nothing to leak. __enter__() has the same exposure, though. __exit__() runs only after __enter__() returns (see exercise 7). Whichever method acquires the resource, a step that can fail after the acquisition needs its own try/except, which releases the resource before re-raising the exception.

`weakref.finalize()` as a Backstop

The second reliable approach is weakref.finalize(), which registers a cleanup callback for an object without giving that callback a reference to the object:

# finalizer.py
from weakref import finalize

class Connection:
    def __init__(self, name: str) -> None:
        self.name = name
        print(name, "opened")
        self.closer = finalize(self, print, name, "closed")

    def close(self) -> None:
        self.closer()

a = Connection("A")
#: A opened
b = Connection("B")
#: B opened
a.close()  # [1]
#: A closed
a.close()  # [2]
print(a.closer.alive, b.closer.alive)
#: False True
del b
#: B closed
print("End of program")
#: End of program

finalize() registers print(name, "closed") to run when Python destroys the Connection. The callback receives name, not the Connection, so registering the cleanup does not keep the object alive. finalize(self, self.close) looks tidier and defeats that separation. The bound method holds a strong reference to the object, so the object survives until the program ends.

The close() at [1] runs the callback immediately. The second close(), at [2], does nothing. A finalizer runs at most once, and alive reports whether it still can. b holds the only reference to its Connection, so del b destroys the object here, where the del c in cleanup.py does not. Nobody calls close(), but the callback still runs, and it runs before interpreter shutdown rather than during it. For an object still alive when the program ends, finalize() runs the callback from the atexit module’s exit handlers, ahead of the teardown that makes __del__() unreliable.

The `self.close` Trap

Passing self.close to finalize() produces no error, but the object stays alive until the program exits:

# finalize_trap.py
import gc
from weakref import finalize, ref

class Leaky:
    def __init__(self, name: str) -> None:
        self.name = name
        finalize(self, self.close).atexit = False

    def close(self) -> None:
        print(self.name, "closed")

class Safe:
    def __init__(self, name: str) -> None:
        self.name = name
        finalize(self, print, name, "closed")

leaky, safe = ref(Leaky("L")), ref(Safe("S"))
#: S closed
gc.collect()
print(leaky() is None, safe() is None)
#: False True

A ref() is a weak reference. It watches its object without keeping it alive, and it reports None once the object disappears. So False True says the interpreter reclaimed Safe and kept Leaky. Safe printed S closed at the ref() line. Reference counting reclaimed it there, before gc.collect() ran. gc.collect() cannot reclaim Leaky, because Leaky is not part of an unreachable cycle. finalize() keeps every callback in a registry, and the callback self.close holds the object, so the registry keeps Leaky reachable. A bound method is a small object that stores its instance in its __self__ attribute, so whatever holds self.close holds Leaky through it. Leaky printed nothing, because its callback did not run and nothing failed. With atexit left on, the callback runs as the program exits, and a late L closed is the one sign of the leak. The listing turns atexit off on Leaky’s finalizer, so its output ends at False True.

A Slotted Class Needs `weakref`

finalize() needs a target that supports weak references, and so does the WeakValueDictionary in the next section. A class that declares __slots__ lists its instance attributes ahead of time, and its instances store those attributes without a per-instance __dict__. A class with __slots__ that omits __weakref__ cannot be weakly referenced:

# slotted_no_weakref.py
from weakref import finalize
from exceptions import expect

class Slotted:
    __slots__ = ("name",)

    def __init__(self, name: str) -> None:
        self.name = name

expect(TypeError, finalize, Slotted("x"), print, "closed")
#: [TypeError] cannot create weak reference to 'Slotted'
#: object

__slots__ removes the instance __dict__ and, by default, the __weakref__ slot along with it, so finalize() has nowhere to attach a reference. The same TypeError meets ref(), weakref.proxy(), and the WeakValueDictionary in the next section, since every weak reference attaches there. Listing __weakref__ among the slots opts back in.

Watching Objects Without Holding Them

The weak-reference machinery behind finalize() also solves a different problem: observing which objects are alive without being the reason they are. Here, a WeakValueDictionary counts live instances, using id(self) as each object’s key:

# weak_value.py
from typing import ClassVar
from weakref import WeakValueDictionary

class Counter:
    _instances: ClassVar[
        WeakValueDictionary[int, Counter]
    ] = WeakValueDictionary()

    def __init__(self, name: str) -> None:
        self.name = name
        self._instances[id(self)] = self

    @classmethod
    def live_count(cls) -> int:
        return len(cls._instances)

counters = []
for name in ["First", "Second", "Third"]:
    counters.append(Counter(name))

print(Counter.live_count())
#: 3
counters.pop()  # Release "Third"
print(Counter.live_count())
#: 2
counters.pop()  # Release "Second"
print(Counter.live_count())
#: 1
counters.clear()  # Release "First"
print(Counter.live_count())
#: 0

A WeakSet would do for counting alone. You need the dictionary as soon as you look instances up rather than count them, as Flyweight does. id(self) is the key here because the registry needs one entry per object, not per name. Keyed by name, two Counter objects that share a name collide, and the second displaces the first. Reused id() values are harmless, since the dictionary holds only live objects and no two live objects share an id.

live_count() returns the size of that registry, so it reports how many Counter objects currently exist. When an instance loses its last ordinary reference, in this case when pop() or clear() removes it from the counters list, the interpreter collects it at once, and the dictionary drops that instance’s entry on its own. The count falls 3, 2, 1, 0 as the list releases the objects, with no __del__() and no explicit cleanup call.

A dict or list as the registry keeps every instance alive forever, so the count never falls. The weak reference lets the registry prune itself.

CPython’s reference counting makes the count fall immediately. On an implementation with a tracing collector, such as PyPy, the entries disappear when its collector runs, so the counts fall late.

cleanup.py keeps its count inside __del__(), so that count is correct only if __del__() runs and succeeds, and Why __del__() Is Not Cleanup lists the ways it can fail. The weak registry runs none of your code when an object goes away.

The Rule

Never release a resource in __del__(). The standard library’s file and socket types bend that rule as a diagnostic backstop. io.IOBase gives every file object a __del__() that closes it, and a file from open() or a socket.socket also reports a ResourceWarning there, catching a forgotten close() rather than replacing it:

# resource_warning.py
import gc
import tempfile
import warnings
from pathlib import Path

path = Path(tempfile.gettempdir()) / "leaky.txt"
path.write_text("data")

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    f = open(path)
    f.read(1)
    del f
    gc.collect()
    print(caught[0].category.__name__)
#: ResourceWarning
path.unlink()

Losing the last reference to an open file finalizes it, and its __del__() closes the file and reports the leak, at the same unpredictable moment as any other __del__(). On CPython del f is that moment. The gc.collect() call covers an implementation that waits for its collector.

The diagnostic backstop exists to catch the mistake, not to be the plan. It still depends on the collector reclaiming the object, and a reference cycle defers that collection until the cyclic collector runs, or until interpreter shutdown if gc.disable() has stopped it.

Give a class that owns a resource a close() method and a with block that calls it, so the cleanup runs at a point in the program you can see. Where a callback must still run if the caller forgets, add weakref.finalize() as the backstop, not as the plan. To track objects without owning them, hold them weakly, so the registry cannot become the leak it exists to catch.

Exercises

This chapter’s solutions give a hint, usually the shape of the code, and a full answer for each exercise.

In weak_value.py, replace the final counters.clear() with counters = [] (rebinding the name) and confirm live_count() still reaches 0. The two do different things to the list object. Say what each one does, then say what a second name bound to the same list sees after each.
In weak_value.py, add a classmethod live_names() to Counter that returns a sorted list of the .name of every live instance, by reading cls._instances.values().
In cleanup.py, change the loop to build counters with a list comprehension instead of append() in a for loop, and confirm the output stays the same: no object goes away before End of delete loop prints.
In weak_value.py, change _instances from a WeakValueDictionary to a dict[int, Counter] and run the file again. Report what live_count() prints after each pop(), and explain the difference in terms of what each container holds.
In finalizer.py, change the finalize() call to finalize(self, self.close), make close() print self.name, "closed" instead of invoking the finalizer, and call a.closer() where the file now calls a.close(). Run it again. Report when B closed now prints relative to End of program, and say what keeps the Connection alive.
In cycle.py, change self_link() to build a two-object cycle (a.peer = b and b.peer = a) instead of a self-reference. Confirm both finalizers run at gc.collect(), then remove the gc.disable()/gc.enable() pair and explain why the language no longer guarantees when the two finalized lines appear, although this small program still prints them in the same place every run.
In faulty_init.py, move the print() and the raise from __init__() into __enter__(), in place of its return self. Run it and report whether C closed prints. Then wrap the raise in a try/except that prints the closed line before re-raising the exception, and confirm C opened now has its matching closed.

Cleanup

Why __del__() Is Not Cleanup

Reference Cycles Delay Destruction

Reliable Alternatives

An Explicit close() and a with Block

An __init__() That Fails Leaks the Resource

weakref.finalize() as a Backstop

The self.close Trap

A Slotted Class Needs __weakref__