Update from Vitals World

Receipts under load

When five people open a receipt at the same moment, each now waits about two seconds instead of eight.

Before, on revision 00076 (4 October, about 09:30 Bangkok time), we opened one receipt page fifteen times, five at a time. All fifteen returned: median 8.26 seconds, p90 8.44, slowest 8.76, the whole batch 25.2 seconds. The same page opened alone took 1.96 seconds. After, on revision 00077 (4 October, about 09:34): median 2.19 seconds, p90 2.96, slowest 4.06, batch 8.2 seconds. Again today on revision 00082 (5 October, 18:12): median 2.22, p90 2.56, slowest 3.49, batch 8.2 seconds, and 1.68 alone. About 3.8 times faster at the median.

Why. The ward's server has one loop that writes and four threads that only read. Receipt pages were being answered by the writing loop, one at a time and queued behind every write, while the four readers sat idle. A receipt only reads: its one write is a cache of rows derived from the chain, so two readers write the same rows. Receipts now go to the readers. The change is commit bb54d06 on cwf/ops, in the public repo.

How we measured. Our first load test, on 2 October, ran on staging and showed a p90 of about 21 seconds at five at once. Most of that was the public devnet endpoint's rate limit, which staging uses and production does not. So we measured lightly on production instead: fifteen read-only requests each time.

Receipts still prove: 346 of 346 shifts on the list rebuild to the leaf the chain holds (full walk, 4 October, 09:34).

Open any receipt on world.vitals.academy and follow its "check it yourself" links.