Benchmark · BioView single-cell RNA-seq
BioView in the browser vs scanpy on real PBMC data
We ran the same single-cell pipeline — quality control, highly variable genes, scaling, PCA, nearest neighbours, clustering, UMAP and marker genes — in BioView (WebGPU and CPU paths, inside a browser tab) and in scanpy 1.10.3 (Python) on the same laptop, with matched parameters, and compared timings and results. Everything below is measured; nothing is extrapolated.
Summary
Results agree with scanpy as closely as two correct implementations can: the PCA spectra match, most cells land in the same clusters, and UMAP layout quality (trustworthiness) is within 0.004. Cluster agreement is not 1.0 for any pair of tools because clustering at resolution 0.8 is sensitive to small differences in the neighbour graph; this is also true between scanpy's own Leiden implementations.
Data used
| Dataset | File and source | Size | Cells × genes | Cells after QC | Licence / notes |
|---|---|---|---|---|---|
| PBMC 3k (real) | pbmc3k_filtered_gene_bc_matrices.tar.gz from 10x Genomics, cf.10xgenomics.com/samples/cell-exp/1.1.0/pbmc3k/ (hg19, Matrix Market) |
7.6 MB | 2,700 × 32,738 | 2,699 | 10x Genomics public dataset (CC BY 4.0) |
| PBMC 68k (real) | fresh_68k_pbmc_donor_a_filtered_gene_bc_matrices.tar.gz from 10x Genomics, cf.10xgenomics.com/samples/cell-exp/1.1.0/fresh_68k_pbmc_donor_a/ (hg19, Matrix Market) |
124 MB | 68,579 × 32,738 | 68,548 | 10x Genomics public dataset (CC BY 4.0); 37.3 million non-zero counts |
| PBMC 100k panel (simulated) | scRNAseq_PBMC_100k_counts.csv + scRNAseq_PBMC_100k_cell_metadata.csv (dense CSV, cells as rows), with ground-truth columns true_cell_type, true_quality, doublet_components, batch and donor |
109 MB + 9 MB | 100,000 × 500 | 84,449 | Simulated targeted panel with known labels; 9 true types including doublets; 15,323 cells removed by the 200-gene minimum because the panel has only 500 genes |
Methods
Hardware and software
| Machine | Apple M2 (8 CPU cores, 8-core GPU), 16 GB unified memory, macOS 26.6 |
|---|---|
| Browser | Chromium 152 with WebGPU (Metal), cross-origin isolated; GPU self-test 334 GFLOP/s (1024² matrix multiply) |
| BioView | Analysis in the app's background worker, exactly as users run it; GPU path = WebGPU compute shaders with automatic per-stage CPU fallback; CPU path = the same pipeline in JavaScript |
| scanpy | scanpy 1.10.3, anndata 0.10.9, numpy 1.26.4, scipy 1.13.1, scikit-learn 1.6.1, umap-learn 0.5.12, pynndescent 0.6.0, numba 0.60.0, leidenalg 0.12.0, python-igraph 1.0.0, scikit-misc 0.3.1 (needed for seurat_v3); Python 3.9, OpenBLAS 8 threads; numba warmed before timing |
Pipeline and parameters (identical in both tools)
| Step | scanpy | BioView |
|---|---|---|
| Quality control | calculate_qc_metrics; keep ≥ 200 genes and ≤ 20 % mitochondrial counts | min genes 200, max mito 20 % |
| Gene filter | filter_genes(min_cells=3) | min cells per gene 3 |
| Variable genes | highly_variable_genes(flavor="seurat_v3", n_top_genes=2000) on raw counts | Seurat v3 variance-stabilising method, 2,000 genes |
| Normalise | normalize_total(target_sum=1e4), log1p | target sum 10,000, log1p |
| Scale | scale(max_value=10) on variable genes | z-score, clip at 10 |
| PCA | 50 components, ARPACK (randomised also timed) | 50 components, exact Gram-matrix + Lanczos solver |
| Neighbours | pp.neighbors(n_neighbors=15, n_pcs=50) (pynndescent above 4,096 cells) | k = 15; exact tiled search on GPU up to 100k cells, IVF above; NN-descent on CPU |
| Clusters | Leiden, resolution 0.8 (igraph and leidenalg) | Leiden with refinement on the UMAP graph, resolution 0.8 (Louvain on SNN also reported) |
| UMAP | tl.umap(min_dist=0.3) | min_dist 0.3, same fuzzy-graph objective; PCA initialisation |
| Markers | rank_genes_groups(method="wilcoxon", corr_method="benjamini-hochberg"), all genes | Wilcoxon rank-sum, Benjamini–Hochberg (stratified subsample above 50,000 cells) |
How results were compared
- Timing: wall clock per stage; "paired total" sums the stages both tools run (load, QC, genes, variable genes, scale, PCA, neighbours, clusters, UMAP, and markers where shown).
- Variable genes: Jaccard overlap of the top-2,000 lists.
- PCA: Pearson correlation and largest absolute difference of the 50 explained-variance ratios.
- Clusters: adjusted Rand index (ARI) and normalised mutual information (NMI) against scanpy's Leiden labels for the same cells.
- Neighbours: recall@15 against exact neighbours for 2,000 sampled cells.
- UMAP: trustworthiness (k = 15) on a 5,000-cell sample.
- Markers: overlap of top-25 genes per cluster after matching clusters (Hungarian matching on the contingency table).
Speed (seconds, lower is better)
PBMC 68k — 10x Genomics, 68,548 cells after QC
| Stage | BioView GPU | BioView CPU | scanpy |
|---|---|---|---|
| Load (Matrix Market, 506 MB text) | 2.7–3.5 | 2.7–3.5 | 6.10 |
| QC + gene filter + variable genes | 0.41 | 0.63 | 1.05 |
| Normalise + scale | 0.15 | 0.35 | 1.20 |
| PCA | 2.35 | 3.00 | 11.34 |
| Neighbours | 1.86 (exact) | 6.89 | 9.15 |
| Clusters (Leiden) | 0.94 | 0.82 | 2.01 (igraph) · 212 (leidenalg) |
| UMAP | 0.70 | 29.4 | 33.05 |
| Marker genes (Wilcoxon, every cluster) | 2.84 | 2.91 | 66.93 |
| Paired total | 12.0 | 48.0 | 130.8 |
PBMC 100k panel — 84,449 cells after QC
| Stage | BioView GPU | BioView CPU | scanpy |
|---|---|---|---|
| Load (dense CSV 109 MB; scanpy read MTX) | 1.9–2.1 | 1.9–2.1 | 3.67 |
| QC + gene filter + variable genes | 0.35 | 0.34 | 0.53 |
| Scale | 0.18 | 0.73 | 0.30 |
| PCA | 0.51 | 9.07 | 2.98 |
| Neighbours | 2.58 (exact) | 10.63 | 8.92 |
| Clusters (Leiden) | 0.90 | 0.96 | 2.10 (igraph) · 39.3 (leidenalg) |
| UMAP | 0.90 | 35.72 | 37.52 |
| Marker genes | 1.12 | 1.15 | 5.81 |
| Paired total | 8.3 | 60.4 | 61.8 |
PBMC 3k — 10x Genomics, 2,699 cells after QC
| Stage | BioView GPU | BioView CPU | scanpy |
|---|---|---|---|
| Load | 0.2 | 0.2 | 0.32 |
| PCA | 1.54 | 1.83 | 1.48 |
| Neighbours | 0.03 | 0.21 | 0.19 |
| UMAP | 0.18 | 2.94 | 2.79 |
| Marker genes | 0.34 | 0.35 | 1.46 |
| Paired total | 2.4 | 5.7 | 6.4 |
On small datasets "Auto" mode uses the CPU path (below 5,000 cells), where the difference is small.
Agreement with scanpy
| Metric | PBMC 3k | PBMC 68k | PBMC 100k |
|---|---|---|---|
| Variable genes, Jaccard (top 2,000) | 0.997 | 0.982 | all 500 genes |
| PCA variance ratios, Pearson r (max abs. difference) | 1.0000 (1.2×10⁻⁴) | 1.0000 (4.9×10⁻⁴) | 1.0000 (1.5×10⁻⁶) |
| Clusters vs scanpy Leiden, ARI / NMI (GPU, Leiden) | 0.717 / 0.878 | 0.825 / 0.840 | 0.964 / 0.973 |
| Clusters vs scanpy Leiden, ARI (default Louvain/SNN) | 0.929 | 0.761 | 0.875 |
| Neighbour recall@15 in BioView's own PC space (GPU) | 1.000 | 1.000 | 1.000 |
| Neighbour recall@15 vs scanpy's exact neighbours (GPU) | 0.889 | 0.649* | 0.991 |
| UMAP trustworthiness: BioView GPU / scanpy | 0.876 / 0.875 | 0.891 / 0.898 | 0.963 / 0.966 |
| Marker genes, top-25 overlap (matched clusters) | 0.61 | 0.43 | 0.54 |
* On PBMC 68k, 2 % of the variable genes differ, so the two 50-dimensional PC spaces differ and neighbour lists cannot match exactly; within its own space BioView's GPU neighbours are exact. Marker overlap is limited by cluster boundaries differing slightly between tools.
Against known cell types (PBMC 100k panel)
| Method | Clusters | ARI | NMI | Majority-vote ARI | Purity | Types recovered |
|---|---|---|---|---|---|---|
| BioView GPU, Leiden | 21 | 0.380 | 0.695 | 0.934 | 0.949 | 9 / 9 |
| BioView CPU, Leiden | 25 | 0.332 | 0.668 | 0.935 | 0.950 | 9 / 9 |
| scanpy Leiden (leidenalg) | 23 | 0.367 | 0.686 | 0.935 | 0.951 | 9 / 9 |
Raw ARI is low for every tool because resolution 0.8 splits 9 types (with treatment, batch and donor structure) into 20–30 clusters. Mapping each cluster to its majority type gives ARI ≈ 0.93 for all methods.
Scaling to one million cells
Resampled data, not new biology. Cells were drawn with replacement from the real datasets and each count replaced by a Poisson draw (fixed seed). These runs test memory and speed only.
| Run | Cells after QC | Matrix storage | Total (s) |
|---|---|---|---|
| PBMC 100k → 500,000, CPU | 304,953 | 0.77 GiB in memory | 170.6 |
| PBMC 100k → 500,000, GPU | 304,953 | 0.77 GiB in memory | 15.1 |
| PBMC 68k → 1,000,000 × 32,738 genes, GPU, Leiden | 977,522 | 1.49 GiB compact (398 M counts) | 70.4 |
| Same, matrix kept on disk (browser storage) | 977,522 | 0.11 GiB resident | 97.7 |
Machine load from other processes was high during these runs (load average 7–34), so they are pessimistic. At one million cells clustering is the slowest stage (17 s Leiden).
Statistics, plots and methods: reference checks
Besides the single-cell pipeline, every statistic and plot BioView draws is checked against a reference in an automated test (npm test in bioView/). All passed on 20 October 2026.
| Test script | Reference | Checks |
|---|---|---|
check-stats.mjs — tests, post hoc comparisons, dose–response (4PL/5PL, EC50 with 95 % CI), flow cytometry maths | SciPy 1.13 / statsmodels 0.14; the ryegrass dose–response fit matches R drc LL.4 | 275 |
check-methods.mjs — every study method in docs/METHODS.md, including normality, outliers, Deming, logistic and Prism nonlinear fits, Cox regression and number reading | SciPy, statsmodels, lifelines 0.30, scikit-learn; R survival on aml; Hanley and McNeil 1982. Edge cases must give the right value or say why it cannot be computed | 832 |
check-plots.mjs — Kaplan–Meier and log-rank, ROC/AUC, bootstrap estimation intervals, hierarchical clustering, Q–Q, correlation matrices | R's printed output (survival, pROC conventions) and hand-derived values | 256 |
check-matrix.mjs — every plot type with every compatible column and option, on the examples and on edge cases (missing values, text in number columns, n = 1 groups, zeros on log axes, 55 groups, 100,000 rows) | Invariants: no NaN coordinates, every row plotted or counted as left out in the caption, points inside their axes, legends match series, brackets join the right groups, SVG and PDF export well formed | 15,102 builds · 85,949 checks |
check-recipe.mjs — analysis recipe record, export, import and replay | Replay must reproduce the recorded map bit for bit | 43 |
check-analysis.mjs, check-annotate.mjs, check-gpu.mjs — single-cell steps, CellTypist annotation, GPU support | Exact maths; celltypist 1.7.1 on small synthetic models; GPU kernel parity runs in the browser | 135 named |
One wrong-number bug was found this way and fixed: values such as "0.125" were read as 125 (a dot followed by three decimals was taken as a thousands separator). BioView now uses the same number reader as Data and Studio, with 13 regression cases.
Limits and honest caveats
- One laptop (Apple M2, 16 GB) and one browser (Chromium with WebGPU). Safari and Firefox were not benchmarked; other GPUs will differ. Without WebGPU, BioView uses the CPU path automatically.
- GPU UMAP layouts are reproducible run to run for a given seed but are not pixel-identical to the CPU layout; quality was compared by trustworthiness.
- Marker genes on more than 50,000 cells use a fixed-seed subsample stratified by cluster (stated in BioView's methods text).
- The PBMC 100k panel is simulated (500 genes), so its biology is simpler than a whole-transcriptome experiment; it is used because its true labels are known.
- The one-million-cell runs use resampled data; they show the pipeline fits and how long it takes, not clustering quality.
- Wall times include some background load from other processes on the test machine.
Reproduce it
- Download the two 10x datasets listed above.
- Run scanpy with the parameters in the Methods table (the full script and package versions are in the project's
bioView/docs/BENCHMARK-REFERENCE.md). - Open BioView, load the same files, keep the default map settings (or choose Leiden in the map options), and run the map. Every setting used is recorded in BioView's Recipe and methods text.
- Compare outputs with
bioView/scripts/compare-bench.mjs, which computes every metric on this page.
Full technical write-ups: bioView/docs/BENCHMARK.md (GPU vs CPU vs scanpy, kernel parity) and bioView/docs/GPU.md (architecture).