How we tested Mira's image analysis

Summary

Two things to keep in mind. The AI accuracy numbers were measured with the original Python programs. Since the browser versions give the same results (see "AI parity"), the numbers also apply to Mira. The wound and stained-area fixes were found and tuned on these same data sets, so those results show how well the tuning worked. They are not an independent test.

Data we used

Eight of these pictures come with Mira as examples, unchanged. The credits are in public/examples/mira/NOTICE.txt.

AI parity: browser compared with the original program

To run the models in a browser, we converted each network to ONNX and run it with ONNX Runtime Web (on WebGPU, or WebAssembly if WebGPU is not available). The steps before and after the network are written in Mira's own code. These include the StarDist overlap removal, the Cellpose flow tracking and size model, and the InstanSeg clustering. To compare, we matched objects that overlap with IoU above 0.5 and averaged the IoU in both directions. We only offer a model in Mira if every picture scores at least 0.98 and the average is at least 0.99.

For StarDist and Cellpose we used 20 BBBC039 test pictures. For InstanSeg we used its own example tile in five versions (as it is, flipped two ways, rotated, and as a 2 × 2 mosaic) plus 15 BBBC039 and Kromp pictures recoloured to look like bright-field. This only tests parity, not real tissue. The automatic cell diameters in the browser were the same as in Python to two decimals on every picture. We also checked on the Python side that the ONNX networks match the original networks on 25 pictures each (Cellpose flows within 2.1 × 10⁻⁴ and identical masks; StarDist and InstanSeg identical labels).

AI accuracy on labelled test sets

AP@0.5 = TP / (TP + FP + FN) at IoU 0.5. Count error is the average difference in the number of objects. All runs used automatic settings. * StarDist versatile_fluo was trained on nuclei from the Data Science Bowl 2018, which includes U2OS pictures similar to BBBC039, so its BBBC039 score is not fully independent. Kromp 2020 was not part of its training. We did not check whether Cellpose was trained on these pictures. Kromp 2020 AP@0.5 by magnification (10× / 20× / 40× / 63×): StarDist 0.90 / 0.70 / 0.82 / 0.65, Cellpose nuclei 0.93 / 0.78 / 0.85 / 0.82, Cellpose cyto3 0.94 / 0.74 / 0.83 / 0.79.

Speed

These are median times over the 20 parity pictures (mostly 520 × 696 pixels), using ONNX Runtime Web with WebAssembly and 4 threads in Node on an Apple M2 laptop. The total includes the steps before and after the network. During the cyto3 run another job was using the processor, so its times are probably a bit too high. If you type in the typical cell diameter, Cellpose needs only one network pass instead of three. We have not finished timing inside the browser (WebGPU) yet.

Counting cells on real nuclei pictures

Three BBBC039 test pictures come with Mira as examples. We ran "Count cells" on them with its default threshold method (Otsu threshold and watershed splitting). Cells touching the edge were kept so the count can be compared with the expert masks.

The threshold method joins some nuclei that touch each other, so it counts a little too low. For crowded pictures, "Find cells with AI" works better (StarDist reaches F1 0.95 on this test set). These counts are checked every time the tests run, in mira/src/tasks/realExamples.test.ts.

Scratch / wound area compared with expert masks (BBBC019)

We compared the gap that Mira finds (with default settings) with the expert gap, which is 100 % minus the cell area in the hand-drawn mask. We did this for all 66 pictures that have a mask.

The old method did badly on wounds that were almost closed. In one case it reported an 80 % gap where the experts saw 10 %. The reason is that Otsu always splits the histogram into two parts, so when there is hardly any gap, it splits the texture of the cells themselves. We tried 11 variants of the fix on these same pictures and picked the best one.

For the SN15 pair, Mira gives 84 % closure while the experts give 69 %. So for late, almost closed wounds Mira still finds too little gap. Please look at the overlay on late time points and increase "Gap sensitivity" if the gap looks too small.

Stained area (DAB) compared with pathologists' masks

For the four nuclear stains, we compared Mira's DAB "% area" with the area of the positive nuclei that the pathologists marked (25 pictures per stain). Mira first separates the brown DAB colour from the blue haematoxylin (colour deconvolution) and then applies a threshold.

Because Otsu always splits the histogram, pictures with no brown stain at all used to show some stained area. Adding a lower limit of 0.2 OD fixed this. We took this value from QuPath, where it is the usual DAB positivity level, so it was not fitted to this data. For purely nuclear stains such as Ki67, a fixed cutoff of 0.3 OD comes closer to the pathologists. CDK4 also stains the cytoplasm, which the masks do not include, so "% area" stays higher than the positive-nucleus area for this stain.

Example pictures in Mira (the picture with the median positive area for each stain): Ki67 5.5 % stained compared with 15.8 % positive-nucleus area (too low), and P53 6.9 % compared with 8.1 %. Note that "% area" is not the same as a Ki67 index (positive nuclei divided by all nuclei). Mira cannot count positive nuclei yet.

Limits

• The wound and stained-area settings were tuned on the same data sets we scored them on. We have not tested them on an independent set yet.
• For InstanSeg we only measured parity with the original. Its accuracy on real bright-field tissue is still unknown.
• "Click to fix" with micro-SAM is not in Mira, because we could not show that the browser version matches the original.
• Speed was measured in Node with WebAssembly on one laptop. Browsers, WebGPU and other computers will give different times.
• Cellpose's training data has a CC BY-NC licence. Mira shows this before the model is downloaded (fine for academic research, but check before any commercial use).
• Colony counting and western-blot measurements have not been tested on labelled public data yet.

How to repeat our tests

1. In mira/, run npx vitest run. This runs the 109 unit tests, including the example pictures (src/tasks/realExamples.test.ts).
2. Wound area: download BBBC019v2 and run MIRA_BBBC019=<folder> npx vitest run src/tasks/woundValidation.test.ts.
3. Stained area: download the IHC TMA data set and run MIRA_IHC=<folder> npx vitest run src/tasks/ihcValidation.test.ts. Add MIRA_IHC_FLOORS=0,0.2 to compare cutoffs.
4. AI parity: make reference results with StarDist 0.9.2, Cellpose 3.1.1.2 and InstanSeg, then run MIRA_AI_REF=<folder> MIRA_AI_N=20 npx vitest run src/ai/validation/e2e.parity.test.ts.

The full report is in mira/docs/AI-VALIDATION.md, and the model licences are in public/models/THIRD-PARTY-NOTICES.txt.