Protein annotation leaderboard

Which tool puts the known domains and features of human proteins in the right place, and what does it cost?

Data from 15 Oct 2026, commit 4a4b1171 (source)

Change what is ranked

Prices for the estimated cost r_cpu, USD per CPU-hour r_gpu, USD per GPU-hour Default prices are AWS on-demand rates for US East (N. Virginia), Linux, read 2026-10-15 from the EC2 on-demand price page (its price file). CPU: a c7i.large costs $0.08925 per hour for 2 vCPU, so $0.044625 per CPU-hour. GPU: a g5.xlarge with one NVIDIA A10G costs $1.006 per hour.

Table 1. Tools ranked by Fmax, highest first, best setting of each tool.

aligns lettersfamily profiles3-D shape lettersreduced-alphabet k-mersPfam models not rungraded on its own models

The marks and colours in Table 1, in full

aligns amino-acid letters: phmmer, MMseqs2builds a profile of a protein family: jackhmmer, MMseqs2, three rounds, HH-suite (hhsearch)compares 3-D shape letters (3Di): Foldseek, ProstT5 + Foldseekcounts shared k-mers in a reduced alphabet: kmerseekscans curated Pfam family models: hmmscan with Pfam

not run: no row in the results table for these filters not measured: the search ran, but no task record holds its cost no correct calls: no call was correct, so there is no median IoU correct calls only on single residues: Fmax is above 0, but every correct call is on a one-residue annotation, where an IoU says nothing about where the ends are needs 3D structure prediction needs GPU what the tool needs besides the sequence graded on its own models hmmscan under the Pfam truth set: Pfam's domains are defined by the models hmmscan runs, so read its row as an upper limit Each row shows a tool's best setting for the column it is sorted by; "+N more settings" opens the rest. Click a column name to sort; rows with no value sort last. Cost and core-hours are measured on Stanford's Sherlock cluster, from the run's own task records

In plain words: Fmax asks whether a tool finds the real domains without adding wrong ones. AUPRC asks whether its correct calls stay at the top of its list. Median IoU asks how well the ends of its right calls line up with the real domain. All run 0 to 1, and higher is better. Hover or tap a ? for more, or see How the scores work, which works each through on a made-up protein.

Each tool searched 998 human proteins, most from chromosome 6, against one relative's proteome and copied its annotations (which proteins).

Estimated cost = core-hours × r_cpu + GPU-hours × r_gpu, per 1,000 queries, for the search alone (how cost is counted).

How to choose between Fmax and AUPRC, and raw against dedup

Fmax is the score at the single best cutoff on the tool's own score, protein by protein: use it when you will keep every call above one cutoff. AUPRC sums precision over every cutoff, counting each call: use it when you will rank calls and read down the list. The two can rank tools in opposite orders, because AUPRC pools every call of every protein while Fmax averages protein by protein. A domain reported many times counts against both. Dedup merges those copies; for phmmer against zebrafish (Pfam) it moves Fmax from 0.333 to 0.457 and AUPRC from 0.029 to 0.111. Definitions: Methods.