[theCatWisel]

Hey @arjaythedev, thanks a lot for running and merging #143! Seeing it hold 190k users at 79.6 ms p95 / 96.2 ms p99 (and only missing the 200k step by 5% server CPU) was super motivating.

Looking closely at the 200k run from #143, we noticed a few remaining database and pipeline bottlenecks that started adding up right around the 80% CPU mark, so here is another shot aimed at holding 200k+ users cleanly:

1. Sub-pipeline A1/A2 split for early login, user, and tag dispatch: Previously, GET /api/tags, GET /api/user, and POST /api/users/login waited behind the entire Phase A batch (including the heavier list and feed queries) before draining. Phase A now sends two Sync boundaries in a single socket flush() so tags, uid, and uemail drain first and dispatch logins to the bcrypt workers immediately while PostgreSQL is still executing the list queries in A2.
2. Single-lookup article metadata and body-free list queries: Queries that already visit the articles heap in Phase A (global, tag, favlist, aslug, and aid) now project (slug, title, description, created_at, updated_at) right away instead of re-fetching the same heap rows in Phase B. Meanwhile, author and feed stay pure index-only scans on articles_author_idx (author_id, id DESC), and artmeta in Phase B no longer selects a.body since list endpoints omit body anyway.
3. 1-round-trip POST /api/articles and known_fav flag bypass: Newly created articles (newart) now return their full 201 Created payload directly from the write worker in one round trip (using the request's in-memory title/description/body/tags and favoritesCount = 0), and fav/unfav writes pass their known favorite state to the read queue so Phase B skips redundant favorites_pkey probes.

All 197 conformance checks pass. Looking forward to seeing if this gets us past the 200k mark!

[theCatWisel]

Quick update on this branch (486cd87) before the run: we pushed a few more improvements that gave us a solid boost in local load tests across all concurrency tiers:

• Register-resident x86-64 BMI1/BMI2 bcrypt assembly (build.rs + src/auth.rs): build.rs now emits multi-lane bc_expand0 kernels (ah and bextr byte-extraction styles across widths 1..=6, inspired by @nick-potts's asm_gen in zig-iouring-nick-potts and @WorriedArrow), keeping up to 5 Blowfish lanes in general-purpose registers with single-register displacement addressing over 64-byte cache-line aligned 4,224-byte BfState structs. Startup calibration benchmarks all kernels and lane widths against the host CPU's L1D cache and picks the fastest fit automatically.
• PostgreSQL 18 SAOP (= ANY($1)) scans & simpler batch CTEs (src/queries.rs): Converted the remaining unnest(...) WITH ORDINALITY joins on uid, uemail, prof, aslug, aid, artmeta, artau, and artx to = ANY($1) B-Tree ScalarArrayOp scans so PostgreSQL 18's _bt_preprocess_array_keys walks the index in ascending key order in a single scan and the Rust workers map rows back to KeySet ordinals in $O(1)$. Also simplified newcmt_batch to assign nextval inside the input CTE and drop the WindowAgg + Sort steps.
• Per-worker scratch reuse (src/engine.rs): Moved the per-batch row lookup vectors, page unpack buffers, and write array encoders into persistent worker scratch structs (ReadRenderScratch and WriteWorker ArrayEncoders) to cut allocator churn under heavy batching.

CI is green (197/197). Thanks again @arjaythedev!

[arjaythedev]

Benchmarked at a9ca94b on four load boxes: 190,000 users held 5 minutes, the same as #143 (b194963, merged), so you keep rank 1.

Hold at 190,000 (4 load boxes, each <= 11 % CPU on average, not LG-limited): 47,297 req/s, p50 253 ms, p95 469 ms, p99 757 ms, 0 errors (2 timeouts). Target CPU 99 % (app 34 % / db 62 % of one core). Logins 15,249 at 0 errors. The service log shows one start for conformance, one for the search and no exits.

Probes were clean up to 210k (52,280 req/s, p95 460 / p99 641 ms, 0 errors); 220k failed on p95 (585 ms) and 250k on errors (7.5 %). The holds at 210k (p95 586) and 200k (p95 654 / p99 940) failed on latency only, 190k passed.

The head moved to 486cd87 while this run was going, so I haven't merged this. The leaderboard row now shows this a9ca94b run, with a note saying so. 486cd87 is queued behind the PRs already waiting, and it'll get its own 4-box run when its turn comes.

— benchmark agent

[arjaythedev]

Benchmarked at 486cd87 on four load boxes: 180,000 users held 5 minutes, a bit below the 190,000 that #143 (b194963, merged) and a9ca94b scored, so I'm not merging this one and the leaderboard keeps 190,000 for you (still rank 1).

Hold at 180,000 (4 load boxes, each <= 10 % CPU on average, not LG-limited): 45,077 req/s, p50 228 ms, p95 420 ms, p99 776 ms, 0 errors (1 timeout). Target CPU 99 % (app 30 % / db 67 % of one core). Conformance passed and the service log is clean (no restarts).

Probes passed up to 220,000 (55,138 req/s, p95 456 / p99 685 ms) and failed at 230,000, but the 5-minute holds at 220k, 210k, 200k and 190k all failed on p95 only (761, 605, 574 and 524 ms, essentially 0 errors). So the new bcrypt kernel looks fine on the probes; with Postgres at about two thirds of the box, it's the sustained DB side that's the limit, just like last time.

If you'd rather have this commit merged anyway, or you push a new one, say so here and it goes into the queue.

— benchmark agent

[nick-potts]

This challenge is so annoying. The only way to pass 200k is for higher probes to fail, so the db ages less.

[theCatWisel]

Thanks for running 486cd87! Seeing it clear the 220,000 probe (55,138 req/s, p95 = 456 ms) and get 200,000 hold p95 down to 574 ms (80 ms faster than a9ca94b's 654 ms on 200k, even after an extra 5-minute hold at 220k of database wear) made it much easier to see what was happening during the late holds.

Looking at the 220k -> 210k -> 200k hold progression alongside the aged-database behavior discussed in issue 175, we found four bottlenecks in 486cd87 and pushed 8a89443 to fix them:

1. Write-batch conflict check bug (WriteWorker::commit_group): In 486cd87, has_slug_mutation disabled fav_batch, unfav_batch, follow_batch, unfollow_batch, and newcmt_batch whenever a write batch contained any CreateArticle (POST /api/articles). Because POST /api/articles is ~8% of write calls, any write batch of 30+ items had a >90% chance of containing at least one CreateArticle, causing WriteWorker to fall back to individual statement execution on almost every multi-write batch under high load. CreateArticle generates a new random-suffixed slug and cannot conflict with existing slugs in the batch, so we removed it from conflict blocking and scoped UpdateArticle/DeleteArticle checks to matching slugs only.
2. Comment author deduplication (cmts -> Phase B artau): As comment threads grow toward 100 comments per article over the probes and holds while PUT /api/user dirties users pages, the LEFT JOIN users u ON u.id = c.author_id inside cmts was doing up to ~2,000 repeated users_pkey heap lookups per batch. cmts now returns c.author_id and interns comment authors into the existing Phase B artau (WHERE u.id = ANY($1::int8[])) scan so each unique author in a batch is fetched once.
3. Inline author metadata, 1-row artx, and 32-hub oldest-page feed check: author list queries now project article metadata directly in Phase A2 (has_meta = true) so profile pages skip Phase B artmeta, artx is back to 1 row per article instead of the 5x row-multiplying UNION ALL ... GROUP BY from 486cd87, and feed's hub_floors checks ORDER BY a.id ASC LIMIT 1 OFFSET 999 across 32 shared hub authors from the oldest (frozen seed) end of articles_author_id_id_idx.
4. Two-regime read coalescing window and shrx/rorx bcrypt kernel: When 486cd87 cut bcrypt time in half with BMI2 assembly, READ_QUEUE's max_age (bcrypt_us / lanes) also shrank in half even when all read workers were busy, firing more batches/sec to PostgreSQL. ReadWorker now keeps the short bcrypt_us / lanes window while idle read workers exist (busy < rw) and widens to (tau_us / rw).max(bcrypt_us / (lanes / 2).max(2)) only when all read workers are saturated (busy >= rw). We also added a single-uop shrx/rorx BMI2 kernel variant in build.rs for startup hardware calibration alongside ah, bextr, and rust.

Ready for a re-test whenever the queue is free @arjaythedev!