A Sentry webhook goes in. A Slack message comes out naming the pull request that last changed the crashing line, who wrote it, and what the reviewers said.
When a production error fires, the first question is "what changed here, and who knows about it?" Answering it by hand is a fixed chain: open the Sentry event, note the file and line, work out which commit was deployed, git blame at that commit (not main, which has moved on and shifted every line number), map the commit to its PR, read the PR. This service runs that chain the moment the alert arrives and posts the result next to it.
- Verifies Sentry's HMAC signature (fails closed), then takes the last in-app stack frame and the release SHA.
- Blames the crash line in GitHub at the release SHA. That is the default because it measured best on real crashes (see Accuracy). With
BLAME_STRATEGY=functionit instead reads the file at the release SHA, finds the enclosing Python function withast, and reports the newest commit in that function. - Maps the winning commit to its PR (title, author, merge time, description, review comments) and posts a Slack Block Kit message.
- No local git clones: everything goes through the GitHub GraphQL API. Redis is only a cache.
Real output from python scripts/demo.py (trimmed, Slack blocks flattened to plain text), which runs the actual service against a fake GitHub and a fake Slack (no accounts needed, see Quick start). The crash is on line 20 of invoice.py. PR #214 introduced the bug: it swapped the empty-items guard above for a tier check and rewrote line 20 from rebate / max(units, 1) to rebate / units. Two weeks later PR #226 edited only the function's docstring. Line blame names #214. The demo's last step restarts the service with BLAME_STRATEGY=function, which names #226, the newest commit in the function; the next section measures how often that happens.
POST /api/v1/ingest -> 200 in 48 ms {'status': 'processed', 'pr_found': True, 'pr_number': 214, 'slack_notified': True}
GitHub GraphQL calls the service made:
blame {'ref': '9f3c2a1b7d4e5f60718293a4b5c6d7e8f9012345', 'path': 'src/billing/invoice.py'}
pull_request {'sha': '22222222'}
Slack message:
:rotating_light: ZeroDivisionError: division by zero
File: /app/src/billing/invoice.py Function: average_discount Line: 20 Blame type: line-level
---
Introduced in PR: #214 Skip rebate for accounts without a tier
Author: @bokafor Merged: 2025-09-14T18:45:00Z
Commit: 22222222 — Skip rebate for accounts without a tier
> Legacy accounts have no tier and should never get a rebate.
Key Review Comments:
• @alicechen: Should we still guard against zero units?
Sentry Event b6d1c0f2a8e34d5f9a7c1e2d3f4a5b6c | Ref: SHA-pinned
Same alert, service started with BLAME_STRATEGY=function:
GitHub calls: ['file', 'blame', 'pull_request'] -> pr_number 226
Blame type: function-level (lines 15-20)
The GitHub half of function mode against a real public repo (python scripts/live_check.py, read-only, psf/requests at the v2.32.3 SHA, function prepare_url): file fetch, AST lookup, blame and PR lookup found PR #6188 in 1.4 to 1.7 s across five runs (three sequential GraphQL calls, measured on my laptop).
How often is the PR in the message the one that introduced the bug? eval/ measures it on 223 real crashes from 12 public Python projects (pylint, pip, pytest, astroid, aiohttp, werkzeug, click, flask, jinja, urllib3, requests, httpx). Each case is a GitHub issue with a traceback that a merged PR fixed. The deepest traceback frame inside the project becomes a signed Sentry webhook pinned to the commit just before the fix, and it goes through the real service against the live GitHub API. The answer counts as correct if the PR it names last wrote a line that the fix had to change inside the crashing function. That labelling method is SZZ, the standard way to find bug-introducing changes from their fixes.
| Blame strategy | Named the bug-introducing PR | 95% CI |
|---|---|---|
| Crash line only (what the service does by default) | 178/223 (80%) | 74% to 85% |
Whole function, newest commit (BLAME_STRATEGY=function) |
93/223 (42%) | 35% to 48% |
| Whole function, skipping formatting/comment/docstring-only commits | 106/223 (48%) | 41% to 54% |
| Whole function, also skipping type-annotation-only commits | 117/223 (52%) | 46% to 59% |
- Why the crash line is the default. The service first shipped function-level blame, on the reasoning that the line that crashes is often not the line that was changed. The data disagreed. In 136 of the 223 crashes the crashing line was itself one the fix rewrote. In the other 87, blaming the crash line still found the culprit more often (53 against 29), because the crashing line and the bug usually arrive in the same PR. Function-level blame lands on whatever touched the function last instead: a black reformat, a typing PR, an unrelated fix further down. Skipping cosmetic commits closes only part of that gap, so it is not offered as a mode.
- Crash-line blame was ahead in 9 of the 12 projects, tied in 2, and behind in 1 (requests, 4 cases).
- Across all 502 traceback-linked fixes, including bugs fixed in a different function from the one that crashed, the rates fall to 42% (line) and 23% (function). Blaming one frame cannot find a bug that lives in a caller.
- The webhook's answer matched the crash-line strategy on all 502 cases. Each alert took a median of 1,116 ms and a p90 of 1,689 ms end to end against the live API, with no cache.
Method, the funnel from 36,130 scanned PRs down to 502 cases, the per-project table and the limits are in eval/README.md. To reproduce (read-only, 24 minutes on my laptop): GITHUB_TOKEN=$(gh auth token) python eval/run_eval.py.
git clone https://github.com/rakshithmuda22/stacktrace-time-machine.git
cd stacktrace-time-machine
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
docker run -d -p 6379:6379 redis:7-alpine # optional: enables the cache check
python scripts/demo.pyThe demo starts a fake GitHub GraphQL server, a fake Slack receiver and the real service, sends signed Sentry-shaped webhooks, and exits non-zero if any of these checks fail: unsigned and wrongly signed requests get 401, every GitHub query uses the release SHA, the right PR is found, a second identical alert is served from Redis with no GitHub calls, a non-SHA release falls back to DEFAULT_BRANCH, and a second service started with BLAME_STRATEGY=function names the docstring-only PR instead. Without Redis it still runs and reports the cache check as skipped. Add --json to print the raw Slack payload.
You need a GitHub token, a Slack incoming webhook, and a Sentry integration (Settings, Developer Settings) whose webhook URL is this service and which is available as an alert action in an issue alert rule.
cp .env.example .env # fill in GITHUB_TOKEN, SENTRY_CLIENT_SECRET, SLACK_WEBHOOK_URL, REPO_MAP
docker compose up -d # Compose v1: docker-compose up -d
curl http://localhost:8000/healthPoint the Sentry webhook at https://your-host/api/v1/ingest and give the service the integration's Client Secret as SENTRY_CLIENT_SECRET. Set Sentry's release to the deployed git commit SHA (for example sentry-cli releases new "$(git rev-parse HEAD)"); otherwise the service falls back to DEFAULT_BRANCH and line numbers can drift. To check a deployment by hand, sign a body the way Sentry does:
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SENTRY_CLIENT_SECRET" | awk '{print $NF}')
curl -X POST localhost:8000/api/v1/ingest -H "sentry-hook-signature: $SIG" --data-binary "$BODY"flowchart LR
S[Sentry issue alert] -->|POST /api/v1/ingest| V{HMAC valid?}
V -->|no| X[401, nothing else runs]
V -->|yes| P[Last in-app frame + release SHA]
P --> N[Strip path prefix, map project to repo]
N --> L{BLAME_STRATEGY}
L -->|line, the default| E[Crash line]
L -->|function, Python file| F[GitHub: file at release SHA] --> A[ast: enclosing function range]
A --> B[GitHub: blame at release SHA]
E --> B
B --> Q[GitHub: commit to PR]
Q --> M[Slack Block Kit message]
R[(Redis)] -.->|blame + PR cache| B
R -.-> Q
Design decisions worth knowing:
- Pinned to the release SHA. The blame (and in function mode the file) is read at the commit Sentry says was deployed, so the line numbers in the stack trace match the code being blamed. The blame query uses
object(expression: $ref): GitHub'sref(qualifiedName:)only resolves branches and tags and returnsnullfor a bare SHA (checked against the live API; a unit test locks the query shape). If the release is missing or is not a commit SHA (for exampleapp@1.4.2), the service usesDEFAULT_BRANCHand labels the message "branch-based". - Crash line by default, function on request. By default the commit that last wrote the crash line is the answer. With
BLAME_STRATEGY=function, the commit with the latestcommittedDateamong the blame ranges that overlap the function wins; timestamps are parsed, not string-compared, because GitHub returns each author's own UTC offset. On real crashes the line named the bug-introducing PR 80% of the time and the function 42% (Accuracy), so line blame ships as the default. - Cache. Blame results are cached in Redis for 30 days when the ref is a commit SHA (immutable) and 1 hour when it is a branch. Commit-to-PR lookups are keyed by commit SHA and cached 30 days. A real-Redis test checks the TTLs. If Redis is down, reads count as misses and the webhook still succeeds.
- Fail closed on auth, fail open on everything else. No
SENTRY_CLIENT_SECRETmeans every request gets 503; a missing or wrong signature gets 401 and no GitHub or Slack call is made. A GitHub failure still produces a Slack message with whatever was found. - Slack text is escaped. Commit messages and PR titles are untrusted;
<!channel>in a commit message is rendered as text, not a ping.
POST /api/v1/ingest takes a Sentry issue-alert webhook with a sentry-hook-signature header (HMAC-SHA256 hex digest of the raw body, keyed with the Client Secret).
| Status | Meaning |
|---|---|
| 200 | Processed. Body: {"status","event_id","pr_found","pr_number","slack_notified"}. Also returned when GitHub or Slack failed partway. |
| 401 | Signature missing or invalid |
| 422 | Malformed payload, no in-app frame, or no REPO_MAP entry for the project |
| 503 | SENTRY_CLIENT_SECRET is not configured |
GET /health returns {"status": "healthy" | "degraded", "version", "redis_connected", "uptime_seconds"}. degraded means Redis is unreachable; the service still works.
| Variable | Required | Default | Description |
|---|---|---|---|
GITHUB_TOKEN |
Yes | GitHub token that can read the repos (repo scope for private ones) |
|
SENTRY_CLIENT_SECRET |
Yes | Sentry integration Client Secret, for signature checks | |
SLACK_WEBHOOK_URL |
Yes | Slack incoming webhook URL | |
REPO_MAP |
Yes | projectId:owner/repo, comma-separated. Sentry sends the numeric project id in data.event.project. Malformed entries stop the service at startup |
|
REDIS_URL |
No | redis://localhost:6379 |
Cache (docker compose overrides it to the bundled Redis) |
PATH_STRIP_PREFIX |
No | empty | Container path prefix to strip, e.g. /usr/src/app/ |
DEFAULT_BRANCH |
No | main |
Ref used when the release is missing or not a SHA |
BLAME_STRATEGY |
No | line |
line blames the crash line. function blames the enclosing Python function and reports its newest commit (one more GitHub call per alert, and less accurate; see Accuracy). Any other value stops the service at startup |
GITHUB_GRAPHQL_URL |
No | https://api.github.com/graphql |
Override for GitHub Enterprise Server |
PORT |
No | 8000 |
Host port published by docker compose (the container always listens on 8000) |
LOG_LEVEL |
No | INFO |
JSON logs on stdout |
pip install -r requirements-dev.txt
pytest tests/ -v # 115 tests; with no Redis on localhost:6379 one is skipped
flake8 src/ tests/ scripts/ eval/ --max-line-length=120 --ignore=E501,W503,E203
python scripts/demo.py # the end-to-end run described above
GITHUB_TOKEN=$(gh auth token) python eval/run_eval.py # accuracy eval, live GitHub, read-only115 tests pass with Redis available; without Redis the real-Redis TTL test is skipped (114 passed, 1 skipped). GitHub and Slack are mocked in the unit tests. GitHub Actions (.github/workflows/ci.yml) runs the same pytest command against a Redis service container, then the demo, plus flake8. The accuracy eval is not in CI: it needs a token and takes about half an hour.
Layout: src/server.py (endpoint and pipeline), analyzer.py (payload parsing, blame range, AST locator), github_client.py (GraphQL queries, retry on 429), cache.py, slack_notifier.py, webhook_auth.py, config.py, schemas.py, logging_config.py. scripts/ holds the demo and the live check; eval/ holds the accuracy evaluation (dataset, scripts, per-case results).
- "The last PR to touch the crash line" is a heuristic, not proof of who introduced the bug. On 223 real crashes it was the bug-introducing PR 80% of the time. It misses when a later change rewrote the crashing line without causing the bug, or when the bug is in other lines. Function mode does worse (42%): a later reformat, typing change or unrelated fix to the same function wins over the real culprit (the live check above landed on an error-message fix). Treat the PR as a lead.
- Only the last in-app frame is analysed, so a bug in a caller is missed. Function mode covers Python only; other languages always use line blame.
- If a commit belongs to several PRs, the most recently created one is shown. Commits pushed straight to a branch have no PR, and the message falls back to commit details.
- In function mode the file fetch is not cached, so a repeated alert still makes one GitHub call; in line mode a repeated alert makes none. GitHub calls are sequential (0.3 to 0.9 s each in my runs). Cached PR details can be up to 30 days stale.
- Review "comments" are the bodies of the PR's reviews (top 3 non-empty), not inline line comments.
- Rate limiting is handled for HTTP 429 with backoff; GitHub's other rate-limit forms surface as a failed lookup and a partial Slack message.
- Tested against payloads shaped like Sentry's documented issue-alert webhook and against the real GitHub API; not run against a live Sentry organisation.
MIT