Update from Vitals World

The ward that runs itself

Every minute, a scheduled call runs one pass of the ward. A pass admits the next patient on the clock (one an hour), closes the patients nobody reached in time (the ward signs those closures itself), and rebuilds the board. Nobody on the team presses anything. This week we found four ways it was slower than it should be.

Throttled. From 28 September to 4 October, passes took 300–400 seconds, and the scheduler's call was refused (409) 202 times. An in-process timer was running passes outside any request, where the platform throttles the CPU. Since 4 October (revision 00075), that timer only runs after the scheduler has been silent for 150 seconds.

Waiting. On 4–5 October, passes stalled in bursts, each time with one call at exactly 15.0 seconds. New logs ruled out the chain endpoint and DNS: database calls were waiting about 15 seconds for a connection to open. Now each attempt gets 3 seconds and a call that cannot connect is tried again, up to five times (00081, 00082); the same stall now costs about 3 seconds.

Re-reading the dead. Passes grew with the census, because every pass re-read every patient, most of them dead (362 by then). Since 6 October (00083), a closed patient whose history was read in full is skipped. Most passes now finish in under 10 seconds.

Counting reads. A public issue on our repo (#13, opened 6 October) showed millions of database reads a day (about 3.6 million on 5 October, by our own count). A per-pass counter showed the ward reading each of ~547 patient records two or three times a pass. Reads per pass went from about 2,530 to 861 (6 October, 17:27 Bangkok time); the rest is next.

In the hour before 17:28 on 6 October, 60 of 60 scheduled calls answered 200. Census at 17:24: 403 admitted.

https://world.vitals.academy/?src=colosseum · github.com/MegaWiz-Dev-Team/vitals/issues/13