bgt¶

Fault-tolerant background threads for Python

POV: you want a framework-agnostic way to reliably run a plain¹ function or method in the background, continuously or with pauses between work units.

bgt comes to the rescue with:

• A service that runs your code in a loop and optionally waits between work units. It wakes up on a fixed time interval, or as soon as something wakes it.

• A supervisor that runs that loop in a background thread. If your code crashes, the supervisor restarts the loop after an exponential backoff. Write crash-only code, bgt takes care of the rest.

• Thorough instrumentation via structlog and Prometheus.

• Framework and platform independence.

bgt is the engine underneath pgbg, which adds PostgreSQL LISTEN / NOTIFY-driven wakeups and leader election with automatic failover on top. If your services should wake up on database events, or only one process at a time should do the work, check it out!

bgt is not a job queue like Celery or RQ. Common use cases include:

• Periodic cleanup duties for expired caches or sessions.
• Flushing buffered metrics or events.
• Processing a continuous stream of data in bounded batches.

Here’s a service that runs a work unit every two seconds and survives its own crashes:

import bgt


def do_work() -> bool:
    ...  # one bounded work unit; if it crashes, it doesn't crash the loop

    return False  # False = wait for next wakeup; True = run again immediately


with bgt.SupervisedService.start(
    bgt.as_work_factory(do_work),
    name="example",
    wakeup=bgt.IntervalOnlyWakeup(),
    interval=2,
):
    ...  # do_work runs in the background until we leave this block

Return True from your work unit to be run again immediately. This keeps work units short, which makes shutdowns prompt.

Credits¶

bgt is written by Hynek Schlawack and distributed under the terms of the MIT license.

The development is kindly supported by my employer Variomedia AG and all my fabulous GitHub Sponsors.