Watching a service over time

A service has been running in staging for a day. Nothing is on fire, but nobody can say with confidence what it actually does: which functions the traffic reaches, how often, how slow each one is, and whether any of that changes through the day. The code has no instrumentation, and adding some means an edit and a redeploy for every question, with the answers scattered through a log file that was never designed to be summed.

Recording every call for a day is the wrong tool, because the answer is a handful of numbers per function, not a million events. What is wanted is a summary that resets on a schedule, so the same table can be compared hour against hour, a way to ask for one right now when something looks off, and, for the occasional deep dive, a raw stream that rotates rather than grows.

wrapture’s windows and collectors are that shape. An Aggregate keeps one row per bound location in bounded memory; a Window opens runs on a schedule or on an operator’s signal and turns each run into a report; a JSONLines sink streams the raw events to rotating files; and a config file makes the whole arrangement zero-code, so the application under observation is never edited.

This example uses: collectors, window() and Window, triggers, reports, JSONLines, output paths and rotation, shutdown() and [[window]] in config.

The service: a shop answering quotes

A stand-in small enough to fit here: a shop that answers price quotes, one class per layer, and a driver that plays the part of a day’s traffic. One item in five is not in the catalog, so a share of the calls raise, and the shop renders each answer as a JSON page listing the whole catalog, which is where most of its own time goes.

>>> import json
>>> import wrapture

>>> def price_list() -> dict[str, int]:
...     base = {"widget": 25, "gadget": 120, "gizmo": 60}
...     packs = {f"{name}-x{n}": price * n for name, price in base.items() for n in range(2, 21)}
...
...     return {**base, **packs}

>>> class Catalog:
...     PRICES = price_list()
...
...     def lookup(self, item: str) -> int:
...         return self.PRICES[item]

>>> class Pricing:
...     def __init__(self) -> None:
...         self.catalog = Catalog()
...
...     def quote(self, item: str, quantity: int = 1) -> dict[str, int | str]:
...         price = self.catalog.lookup(item)
...
...         return {"item": item, "quantity": quantity, "total": price * quantity}

>>> class Shop:
...     def __init__(self) -> None:
...         self.pricing = Pricing()
...
...     def handle(self, item: str) -> str:
...         quote = self.pricing.quote(item, quantity=2)
...         listing = [
...             {"item": name, "price": price}
...             for name, price in sorted(Catalog.PRICES.items())
...         ]
...
...         return json.dumps({"quote": quote, "catalog": listing}, indent=2)

>>> def traffic(shop: Shop, requests: int = 50) -> None:
...     items = ["widget", "gadget", "gizmo", "widget", "missing"]
...     for index in range(requests):
...         try:
...             shop.handle(items[index % len(items)])
...         except KeyError:
...             pass

Nothing here knows it will be observed. The bindings are declared once and applied for the rest of the page; applied but with nothing listening, they record nothing:

>>> shop = Shop()
>>> service = wrapture.bindings(
...     handle=wrapture.binding(Shop, "handle"),
...     quote=wrapture.binding(Pricing, "quote"),
...     lookup=wrapture.binding(Catalog, "lookup"),
... )
>>> service.apply()
<BindingGroup ['handle', 'quote', 'lookup']>
>>> service.handle
<Binding '__main__:Shop.handle' callable active>

One warm-up run before measuring anything, as with any profiling: the first call through a code path pays for imports and caches that later calls do not, and that one-off cost should not land in the numbers.

>>> traffic(shop)

The naive approach: a log line and a stopwatch

The usual answer is a time.perf_counter() pair and a log.info() in each function of interest, then a script over the log file to group and average. Every new question is another edit and redeploy, and the timings mislead: a function that is slow only because of what it calls looks as guilty as one slow in its own right, because a stopwatch around a call cannot see inside it.

One report: the Aggregate collector in a window

Aggregate keeps one row per path and nothing else: how many calls began and completed, how many raised, and total, self, fastest and slowest times. It is a collector, so it lives in a window: the run opens on entry, accumulates, and closes on exit with a report per collector.

>>> with wrapture.window(collect=[wrapture.Aggregate()]) as run:
...     traffic(shop, 2000)

>>> run
<Run 1 of 'window', closed>
>>> report = run.reports[0]
>>> print(report.text)
aggregate "aggregate" run 1, ... to ... (...s), pid ...
3 paths, 6,000 operations begun, 6,000 completed, 1,200 raised

calls  total   self  per-call    min    max  errors  path
2,000    ...    ...       ...    ...    ...     400  __main__:Shop.handle
2,000    ...    ...       ...    ...    ...     400  __main__:Pricing.quote
2,000    ...    ...       ...    ...    ...     400  __main__:Catalog.lookup


The header says which run this was, when it opened and closed in local time, and whose process; the table has one row per bound location, sorted by self time, the figure that answers “where is the time going”: Shop.handle spends more time in its own body than Pricing.quote does, even though every quote runs inside a handle, because self time excludes what the observed children account for. The errors column appears only when something failed, and counts an exception the code caught and noted with note_exception() the same as one that escaped. The same figures are on the report as data, for a dashboard or a test to read without parsing the table:

>>> report.kind, report.window, report.run, report.cut_short
('aggregate', 'window', 1, False)
>>> report.data["begun"], report.data["raised"]
(6000, 1200)
>>> report.data["paths"]["__main__:Shop.handle"]["errors"]
400

window() suits a script or a one-off run of the traffic. For a service that runs all day the schedule has to come from somewhere else.

Reports on a schedule: Window with every=

Window takes the same contents plus the triggers, and start() arms it. every= on its own is the back-to-back shape: each run lasts the whole period and closes as the next opens, so the report at each boundary covers exactly one period, totals reset. Reports are retained on the window (the last ten by default), and on_report= hands each one to a callable the moment its run closes. Durations are the forms rotate= accepts ("30s", "15m", "1h", or a number of seconds); an hour is too long to wait for here, so a tenth of a second stands in, and the page waits for the first report to arrive before sending the second burst of traffic:

>>> import time
>>> seen = []
>>> ticking = wrapture.Window(
...     name="ticking", every=0.1, collect=[wrapture.Aggregate()], on_report=seen.append
... )
>>> ticking.start()
>>> traffic(shop)
>>> deadline = time.monotonic() + 5
>>> while not seen and time.monotonic() < deadline:
...     time.sleep(0.01)
>>> traffic(shop)
>>> ticking.stop()

>>> seen[0].run, seen[0].cut_short, seen[0].data["begun"]
(1, False, 150)
>>> ticking.reports[-1].cut_short
True
>>> ticking.runs >= 2
True

Run 1 closed on schedule and reached seen from the scheduler thread; the run open when stop() was called was closed early and its report says so with cut_short. Adding duration= (config key for) gives the sampled shape, thirty seconds of every hour, say, so the service pays nothing in between; align=True puts the openings on the wall-clock hour.

Reports as files: report= and its path template

An on_report callback needs code to receive it. The file form needs none, and is the one a config file can spell: report= is an output path template written once per run per collector, whole file then rename, so a reader never sees a half-written report. Inside a window three variables join the usual {date} and {pid}: {window} is the window’s name, {first} is when the schedule’s first run opened, the same for every run of one schedule, and {run} is the run number. Driving the runs by hand shows the files appear:

>>> import os
>>> import tempfile

>>> outputs = tempfile.TemporaryDirectory()
>>> daily = wrapture.Window(
...     name="daily",
...     every="1h",
...     collect=[wrapture.Aggregate()],
...     report=os.path.join(outputs.name, "{window}-{first}/run-{run:02}.txt"),
... )
>>> daily.start()
>>> traffic(shop)
>>> daily.close()
<Run 1 of 'daily', closed>
>>> daily.open()
True
>>> traffic(shop, 5)
>>> daily.stop()

>>> from pathlib import Path
>>> for path in sorted(Path(outputs.name).rglob("run-*.txt")):
...     print(path.relative_to(outputs.name).as_posix())
daily-...T...-...-.../run-01.txt
daily-...T...-...-.../run-02.txt

start() opened run 1 at once (no after=), close() ended it and wrote its file, open() began run 2 by hand, and stop() closed that one, cut short. On a real schedule the runs open and close by themselves; the directory is one per schedule, and a restart starts a new one, named by its own first run.

Opening a window on demand: on_signal and on_file

Something looks odd at 15:40 and the hourly report is twenty minutes away. on_signal= and on_file= open a run from outside the process, no restart and no code: kill -USR1 <pid>, or touch a path the window watches. Either can be the only trigger, or sit beside every=, where kicks add runs to the schedule.

peek = wrapture.Window(
    name="peek",
    on_signal="SIGUSR1",
    duration="60s",
    collect=[wrapture.Aggregate()],
    report="reports/{window}-{datetime}.txt",
)
peek.describe()   # 'on SIGUSR1, for 60s'
>>> profile = wrapture.Window(
...     name="profile",
...     on_file=os.path.join(outputs.name, "profile-now"),
...     collect=[wrapture.Aggregate()],
... )
>>> profile.describe()
'on touch of ...profile-now, until the next kick'

peek collects for a minute after each signal, and a signal during that minute is refused rather than overlapping, since runs never do. profile has no duration, so a kick toggles: touch to start collecting, touch again to report and start over, the file being removed as it is consumed. The signal handler is installed by start(), on the main thread, and neither window is started here. SIGUSR1 and SIGUSR2 do not exist on Windows, where naming one is refused at construction; on_file works everywhere, including under servers that own the signal handlers.

The always-on stream: JSONLines with rotate=

Sometimes the individual events are wanted after all, for a jq query over the afternoon’s failures. A JSONLines sink registered at the process tier hears every event from every thread and writes it as one JSON object per line, never blocking the observed call; a time variable in the path plus rotate= keeps one file per day rather than one forever:

>>> stream = wrapture.add_sink(
...     wrapture.JSONLines(
...         os.path.join(outputs.name, "traces/trace-{date}.jsonl"), rotate="1d", align=True
...     )
... )
>>> traffic(shop, 5)

Lines are queued to a writer thread, so the file is complete only once the sink is flushed. At interpreter exit wrapture flushes every process sink and closes any open window run itself; wrapture.shutdown() does exactly that on demand, for hosts that tear the interpreter down without running atexit callbacks. Under mod_wsgi, subscribe it to the process shutdown event, and everything owed is delivered while the interpreter and its threads are still alive. Calling it more than once is safe, and it uninstalls nothing.

>>> wrapture.shutdown()
>>> os.path.basename(stream.path)
'trace-...-...-....jsonl'
>>> with open(stream.path) as lines:
...     events = [json.loads(line) for line in lines]
>>> len(events)
15
>>> sorted({event["path"].rpartition(":")[2] for event in events})
['Catalog.lookup', 'Pricing.quote', 'Shop.handle']
>>> [event["exception"]["type"] for event in events if "exception" in event]
['KeyError', 'KeyError', 'KeyError']

Releasing everything the page applied:

>>> wrapture.remove_sink(stream)
>>> stream.close()
>>> service.remove()
<BindingGroup ['handle', 'quote', 'lookup']>
>>> service.handle
<Binding '__main__:Shop.handle' callable unapplied>
>>> outputs.cleanup()

The same thing from a config file: the Flask shop

The examples/flask-app directory in the repository is a real version of this scenario: a small Flask shop in myapp.py that never imports wrapture, and a wrapture.toml whose [[instrument]] entry instruments Flask itself, so every request records as one tree with its view function beneath it, plus an [[observe]] entry for the app’s own quote helper. Its shipped config ends in a printer sink, the live view; for a day in staging, replace that with a rotating stream and a scheduled report:

pythonpath = "."

[[observe]]
target = "myapp"
name = "quote"

[[instrument]]
name = "wrapture_local.flask_support:FlaskInstrumentation"

# The raw stream, one file per day, rotated at local midnight.
[[sink]]
type = "jsonlines"
path = "traces/trace-{date}.jsonl"
rotate = "1d"
align = true

# A summary of the app's own functions every minute, on the minute,
# one file per run, grouped by the schedule's start.
[[window]]
name = "minutely"
every = "1m"
align = true
report = "reports/{window}-{first}/run-{run:02}.txt"

[[window.collect]]
type = "aggregate"
filter = { kind = "call" }

# On demand: kick with `kill -USR1 <pid>` for a minute's summary.
[[window]]
name = "kick"
on_signal = "SIGUSR1"
for = "60s"
report = "reports/{window}-{datetime}.txt"

[[window.collect]]
type = "aggregate"
filter = { kind = "call" }

filter on a collect entry is the same gating key a sink takes; here it keeps the request events out of the table, so the report is about the view functions and helpers and the request stream is left to the JSONLines file. Relative paths anchor to the config file’s directory. every = "1m" is for watching reports appear; for a day in staging every = "1h" gives one file per hour with the totals reset.

Run the app under the runner, so the config is applied before Flask is imported, and put load on it with any HTTP load tool, here bombardier:

$ cd examples/flask-app
$ uv run --with flask python -m wrapture --config wrapture.toml -m flask --app myapp run --port 8000
$ bombardier -c 4 -d 3m http://localhost:8000/quote/widget
$ bombardier -c 4 -d 3m http://localhost:8000/export
$ ls reports/minutely-*/
run-01.txt  run-02.txt  run-03.txt

Each file is one minute’s table, and the picture emerges from comparing them: which views the traffic reaches, how many quotes fail, and whether per-call moves as the load does. A whole-process run of the same config over a few thousand requests through the Flask test client produced this:

aggregate "aggregate" run 1, 2026-08-18 18:25:42 to 18:25:43 +10:00 (1.1s), pid 16745
4 paths, 2,400 operations begun, 2,400 completed, 600 raised

calls   total    self  per-call   min    max  errors  path
  900  74.7ms  71.7ms      83us  57us  415us     300  myapp:quoted
  300   9.1ms   9.1ms      30us  28us   97us          myapp:index
  300   4.0ms   4.0ms      13us  12us   48us          myapp:export
  900   2.9ms   2.9ms       3us   1us   13us     300  myapp:quote

When something looks off between reports, kill -USR1 <pid> opens the kick window for a minute and drops a timestamped report beside the scheduled ones, no restart involved. Under a pre-fork server each worker has its own windows, so put {pid} in the report path and add jitter to spread the aligned openings. The raw stream is there for when a summary raises a question only the events can answer:

$ jq -c 'select(.exception)' traces/trace-2026-08-18.jsonl | head

Where next

The scheduled tracing page is the full reference for windows: every trigger, the report shape, clocks and restarts, and how to write a collector of your own. The ad-hoc tracing page covers sinks, output paths and rotation, the config file grammar and the runner, and the WSGI request tracing page explains what the request events in the stream contain and how the Flask instrumentation records them.