Finding where a request spends its time

One endpoint is slow. The handler calls a service, the service calls a repository, and a template renders the result, and the question is which of those layers the time is going to. A profiler answers with every frame in the process, most of them framework internals you do not care about, and it cannot tell one request from the next. Logging answers only where someone already thought to add a timer.

wrapture lets you name the four layers you suspect, record one request as one tree, and read the time off each level, in a live interpreter, in a test, or against a running application from a config file, with none of the application’s code changed.

This example uses: WSGI request tracing, observing a bare callable, the Printer sink, sink combinators, the call tree, annotation, exporters and configuring from a file.

The application

A stand-in for the web application: a repository (deliberately slow), a service over it, a template, a view function, and a WSGI application object that dispatches to views by name and exposes its callable as wsgi_app, the way Flask does. Nothing here imports wrapture.

>>> import sys
>>> import time
>>> import wrapture

>>> CATALOG = {"widget": 25, "gadget": 120}

>>> class Repository:
...     def fetch(self, item: str) -> int:
...         time.sleep(0.005)
...         return CATALOG[item]

>>> class QuoteService:
...     def __init__(self, repository: Repository) -> None:
...         self.repository = repository
...     def quote(self, item: str) -> dict:
...         price = self.repository.fetch(item)
...         return {"item": item, "price": price}

>>> class Template:
...     def render(self, quote: dict) -> bytes:
...         return f"{quote['item']}: {quote['price']}\n".encode()

>>> service = QuoteService(Repository())
>>> template = Template()

>>> def quote_view(item: str) -> bytes:
...     return template.render(service.quote(item))

>>> class Shop:
...     def __init__(self) -> None:
...         self.routes = {"quote": quote_view}
...     def wsgi_app(self, environ, start_response):
...         _, name, item = environ["PATH_INFO"].split("/", 2)
...         body = self.routes[name](item)
...         start_response("200 OK", [("Content-Type", "text/plain")])
...         return [body]
...     def __call__(self, environ, start_response):
...         return self.wsgi_app(environ, start_response)

>>> shop = Shop()

To drive it without a server, a helper plays the server’s part: build an environ, call the application, consume the body, then close it. The close matters, because a request event ends when its body closes.

>>> from wsgiref.util import setup_testing_defaults

>>> def get(app, path: str, **headers: str) -> tuple[str, bytes]:
...     environ = {"PATH_INFO": path, **headers}
...     setup_testing_defaults(environ)
...     status = []
...     body = app(environ, lambda s, h: status.append(s))
...     chunks = b"".join(body)
...     close = getattr(body, "close", None)
...     if close is not None:
...         close()
...     return status[0], chunks

>>> get(shop, "/quote/widget")
('200 OK', b'widget: 25\n')

The naive approach

The usual move is a stopwatch around each suspect: time.perf_counter() before and after the service call, a log line with the difference, another pair around the repository, and so on down. Each timer is a code change in a layer that should not know it is being measured, the numbers arrive as separate log lines that you correlate by eye, and none of them are tied to the request they belong to, so a slow request among fast ones is invisible in the average.

One request as one event

WSGIMiddleware wraps a WSGI application and records each request as a single event: the method and path on the opening line, the status line as the result, and the wall time from the call to the close of the body as the duration. Inside a timeline() the request lands on the tape:

>>> application = wrapture.WSGIMiddleware(shop, label="shop")

>>> with wrapture.timeline() as tape:
...     _ = get(application, "/quote/widget")
...     print(tape.tree())
GET /quote/widget (shop)  -> '200 OK'

>>> request = tape.all[0]
>>> request.kind, request.result, request.items
('request', '200 OK', 1)
>>> sorted(request.data)
['app_duration', 'bytes', 'content_type', 'interface', 'method', 'path', 'protocol', 'scheme']

request.duration is time to last byte; data["app_duration"] is the synchronous phase alone, the call that returned the iterable, and body_duration is the time spent producing chunks. For this application the body is one chunk built before the call returned, so almost all of the time is in the synchronous phase, which is where the layers beneath run.

The layers beneath the request

The service, repository and template are methods on classes, so bindings reach them, held here in one group and applied together. The view function lives in a dispatch table rather than at an attribute, so observed() wraps the value instead, and the wrapped value goes where the original was. Once applied, every call through them nests beneath whichever request is in flight:

>>> layers = wrapture.bindings(
...     quote=wrapture.binding(QuoteService, "quote"),
...     fetch=wrapture.binding(Repository, "fetch"),
...     render=wrapture.binding(Template, "render"),
... )
>>> _ = layers.apply()
>>> shop.routes["quote"] = wrapture.observed(quote_view)

>>> with wrapture.timeline() as tape:
...     _ = get(application, "/quote/widget")
...     print(tape.tree())
GET /quote/widget (shop)  -> '200 OK'
  __main__:quote_view(item='widget')  -> b'widget: 25\n'
    __main__:QuoteService.quote(item='widget')  -> {'item': 'widget', 'price': 25}
      __main__:Repository.fetch(item='widget')  -> 25
    __main__:Template.render(quote={'item': 'widget', 'price': 25})  -> b'widget: 25\n'

That is the request as it ran, with real arguments and real results, and no timer in any of the layers.

Watching it live, with timings

Printer is a process sink: register it and every recorded event prints as it happens, an opening line as each operation begins and a closing line with the outcome and the elapsed time. No timeline is needed; the applied bindings and the sink are enough. Timing is on by default:

>>> printer = wrapture.add_sink(wrapture.Printer(sys.stdout))

>>> _ = get(application, "/quote/gadget")
GET /quote/gadget (shop)
  __main__:quote_view(item='gadget')
    __main__:QuoteService.quote(item='gadget')
      __main__:Repository.fetch(item='gadget')
      __main__:Repository.fetch -> 120 [...]
    __main__:QuoteService.quote -> {'item': 'gadget', 'price': 120} [...]
    __main__:Template.render(quote={'item': 'gadget', 'price': 120})
    __main__:Template.render -> b'gadget: 120\n' [...]
  __main__:quote_view -> b'gadget: 120\n' [...]
shop -> '200 OK' [..., body ... over 1 chunk]

>>> wrapture.remove_sink(printer)

The timings are elided above because they differ from run to run; the closing lines from one run read:

      Repository.fetch -> 120 [5.9ms]
    QuoteService.quote -> {'item': 'gadget', 'price': 120} [6.0ms]
    Template.render -> b'gadget: 120\n' [11us]
  __main__.quote_view -> b'gadget: 120\n' [6.3ms]
shop -> '200 OK' [6.4ms, body 2us over 1 chunk]

The answer is already readable: fetch accounts for essentially all of quote, which accounts for essentially all of the view, and render is microseconds. The request’s closing line shows time to last byte and the body’s own share.

Printing less: depth, filters and samples

On a busy application the full tree per request is too much. The combinators wrap a sink and gate what reaches it. Depth(1, ...) forwards only the roots, which for a web application means one opening and one closing line per request, an access log with timings:

>>> printer = wrapture.add_sink(wrapture.Depth(1, wrapture.Printer(sys.stdout)))

>>> _ = get(application, "/quote/gadget")
GET /quote/gadget (shop)
shop -> '200 OK' [..., body ... over 1 chunk]

>>> wrapture.remove_sink(printer)

Filter(predicate, ...) forwards only events the predicate accepts, decided once as each event enters, so it selects by what the event is, its kind, path or label, not by how long it turns out to take. Here it keeps the request lines and the repository, the layer under suspicion, and drops the rest:

>>> suspects = wrapture.Filter(
...     lambda event: event.kind == "request" or event.path.endswith("fetch"),
...     wrapture.Printer(sys.stdout, timing=False),
... )
>>> printer = wrapture.add_sink(suspects)

>>> _ = get(application, "/quote/gadget")
GET /quote/gadget (shop)
      __main__:Repository.fetch(item='gadget')
      __main__:Repository.fetch -> 120
shop -> '200 OK'

>>> wrapture.remove_sink(printer)

Sample(rate, ...) keeps a random fraction of whole request trees, the usual choice when the printer must stay on under real traffic. Because a duration is known only when an event closes, “only the slow requests” is a few lines of your own sink rather than a filter: hear on_exit, and print when the request took longer than a threshold.

>>> class SlowRequests(wrapture.Sink):
...     def __init__(self, threshold: float, stream) -> None:
...         self.threshold = threshold
...         self.stream = stream
...     def on_exit(self, event) -> None:
...         if event.kind == "request" and event.duration > self.threshold:
...             where = f"{event.data.get('method')} {event.data.get('path')}"
...             print(f"slow: {where} {event.duration * 1000:.1f}ms", file=self.stream)

>>> slow = wrapture.add_sink(SlowRequests(0.003, sys.stdout))
>>> _ = get(application, "/quote/gadget")
slow: GET /quote/gadget ...ms
>>> wrapture.remove_sink(slow)

Reading the time off the tree

The tape gives the same picture after the fact, and adds the figure that settles the question: self time, an event’s execution time minus that of its observed children. tape.tree(times=True) shows both, and tape.self_time(event) gives it for one event:

>>> with wrapture.timeline() as tape:
...     _ = get(application, "/quote/widget")
...     print(tape.tree(times=True))
GET /quote/widget (shop)  -> '200 OK'  [..., self ...]
  __main__:quote_view(item='widget')  -> b'widget: 25\n'  [..., self ...]
    __main__:QuoteService.quote(item='widget')  -> {'item': 'widget', 'price': 25}  [..., self ...]
      __main__:Repository.fetch(item='widget')  -> 25  [...]
    __main__:Template.render(quote={'item': 'widget', 'price': 25})  -> b'widget: 25\n'  [...]

>>> request, view, quote, fetch, render = tape.all
>>> tape.self_time(fetch) > 0.5 * view.duration
True
>>> tape.self_time(quote) < 0.5 * quote.duration
True

With the numbers in, one run of that tree looked like this:

GET /quote/widget (shop)  -> '200 OK'  [5.3ms, self 40us]
  __main__.quote_view(item='widget')  -> b'widget: 25\n'  [5.3ms, self 154us]
    QuoteService.quote(item='widget')  -> {'item': 'widget', 'price': 25}  [5.1ms, self 59us]
      Repository.fetch(item='widget')  -> 25  [5.1ms]
    Template.render(quote={'item': 'widget', 'price': 25})  -> b'widget: 25\n'  [6us]

quote and the view are slow only because of what they call; fetch is slow in its own right. That distinction, “slow itself” versus “slow because of a child”, is what the self time column carries and what a wall-clock timer around the service call cannot express.

One thing to know about the request row: a request streams, so its figure in the tree is the application’s own time, the synchronous phase plus the time spent producing body chunks, leaving out any time the server took between chunks. The whole-request wall time and the two phases are on the event itself, and the view’s own duration sits inside the synchronous phase:

>>> request.body_duration < request.data["app_duration"] <= request.duration
True
>>> view.duration <= request.data["app_duration"]
True

Tagging a request with who it was for

A slow endpoint is often slow for one tenant, one account or one request id, and the middleware cannot know which header carries that. annotate(**data) merges values into the in-flight event’s data, and a decorates() handler on the request is a place to call it from where the environ is in hand. That handler lives on the on_request namespace of a mode="wsgi" binding, which installs the same middleware at the attribute the application exposes, so the direct WSGIMiddleware used above is set aside here in favour of the binding form:

>>> app = wrapture.binding(shop, "wsgi_app", mode="wsgi", label="shop")

>>> def tag(application, environ, start_response):
...     wrapture.annotate(tenant=environ.get("HTTP_X_TENANT"),
...                       request_id=environ.get("HTTP_X_REQUEST_ID"))
...     return application(environ, start_response)

>>> _ = app.on_request.decorates(tag)

>>> with wrapture.timeline(app) as tape:
...     _ = get(shop, "/quote/widget", HTTP_X_TENANT="acme", HTTP_X_REQUEST_ID="r-1")
...     _ = get(shop, "/quote/gadget", HTTP_X_TENANT="globex", HTTP_X_REQUEST_ID="r-2")
...     globex = app.events.matching(lambda e: e.data.get("tenant") == "globex")
...     print(globex)
...     [child.path for child in tape.children_of(globex[0])]
<EventLog shop[matching=<lambda>]: 1 event(s)>
    GET /quote/gadget (shop)
['__main__:quote_view']

The tags ride on the request event, so a Filter predicate reading event.data.get("tenant") narrows a printer to one tenant’s requests, and in a test the same expression selects the request to assert on; tape.children_of() walks from there to the layers beneath it. shop(...) is called directly here rather than through application, because the binding has patched shop.wsgi_app and wrapping the app twice would record every request twice.

Handing the trace to other tools

For a request with a dozen layers the printed tree stops being the easiest reading. chrome_trace() renders a tape as Chrome trace JSON, which the Perfetto UI opens as a timeline with nested slices whose widths are the durations, and mermaid() renders it as a sequence diagram that pastes into a pull request:

>>> import json
>>> trace = json.loads(wrapture.chrome_trace(tape))
>>> len([e for e in trace["traceEvents"] if e["ph"] == "X"])
10

>>> print(wrapture.mermaid(tape))
sequenceDiagram
    participant caller
    participant P1 as Shop
    participant P2 as __main__
    participant P3 as QuoteService
    participant P4 as Repository
    participant P5 as Template
    caller->>+P1: GET /quote/widget
    P1->>+P2: quote_view
    P2->>+P3: quote
    P3->>+P4: fetch
    P4-->>-P3: return
    P3-->>-P2: return
    P2->>+P5: render
    P5-->>-P2: return
    P2-->>-P1: return
    P1-->>-caller: return
    caller->>+P1: GET /quote/gadget
    ...

Both exporters also accept the records a JSONLines sink wrote, so a trace file from a running server renders the same way, from a shell as python -m wrapture.tools convert --format chrome -o trace.json trace.jsonl.

That is the end of the in-process walk, so take the patches down:

>>> _ = layers.remove()
>>> shop.routes["quote"] = quote_view

The same thing from a config file

Against a real application, none of the above needs to be code. The flask-app directory in the repository’s examples is a small Flask shop whose wrapture.toml installs the WSGI middleware on every Flask instance and observes every view function from an [[instrument]] entry, and names the application’s own quote helper in an [[observe]] entry. To get the timing view of this page, the [[sink]] entry is a printer with timing on (the default, shown here for emphasis), and depth gates it to the request, the view and one layer beneath:

pythonpath = "."

[[observe]]
target = "myapp"
name = "quote"

[[instrument]]
name = "wrapture_local.flask_support:FlaskInstrumentation"

[[sink]]
type = "printer"
timing = true
timestamps = true
depth = 3

Every key after type goes to the Printer constructor, except the gating keys depth, sample and filter, which are the file’s spelling of the combinators shown earlier; filter = { kind = "request" } would give the access-log view. Save it as timing.toml next to the example and run the example’s script under it; the runner applies the config, then runs main.py as __main__ (Flask is not a dependency of the repository, so uv supplies it):

$ cd examples/flask-app
$ uv run --with flask python -m wrapture --config timing.toml main.py
18:26:03.144 GET / (myapp.wsgi_app)
18:26:03.145   myapp.index()
               myapp.index -> <Response 20 bytes [200 OK]> [77us]
             myapp.wsgi_app -> '200 OK' [553us, body 5us over 1 chunk]
18:26:03.145 GET /quote/widget (myapp.wsgi_app)
18:26:03.145   myapp.quoted(item='widget')
18:26:03.146     myapp.quote(item='widget')
                 myapp.quote -> {'item': 'widget', 'price': 25} [13us]
               myapp.quoted -> <Response 29 bytes [200 OK]> [154us]
             myapp.wsgi_app -> '200 OK' [525us, body 3us over 1 chunk]
...

The same file traces the development server, with the trees arriving in its log as requests come in (uv run --with flask python -m wrapture --config timing.toml -m flask --app myapp run), and adding a second [[sink]] of type jsonlines streams every event to a file for the exporters afterwards.

Where next