Methods

The benchmark run

Everything on this site except the coverage page comes from one experiment, qfo_region_midi_plus, a single benchmark run. It searched 998 human proteins against the proteomes of 15 relatives from the Quest for Orthologs reference proteomes, release 2020_04. Most of the human proteins are encoded on chromosome 6; the rest are immune proteins (KIR and LILR receptors, CD1 and MR1, and β2-microglobulin). The copied tables do not record how many are in each group. The run finished on 2026-09-13 on Stanford's Sherlock cluster. It has 205,653 of the 3,975,483 rows of results.parquet, one per score. The leaderboard groups them into 29,379 combinations of tool, setting, relative, truth set, raw or dedup, and subset. Its cost is in compute.parquet. Both tables were built by make leaderboard-tables in seanome/2024-kmerseek-analysis at commit 4a4b1171. These 998 proteins are not a random sample of the human proteome.

No tool wins everywhere

No tool wins everywhere. So this board ranks tools within a subset of the proteins (by relative, identity, region length, disorder or feature type), and gives no overall score. Two earlier community benchmarks found the same. The Quest for Orthologs benchmark ran 15 orthology methods on 20 benchmarks. Among the balanced methods, "no method consistently obtained a better balance than the other methods across all data sets". Its authors chose not to compute a combined score, because the right trade-off depends on the application (Altenhoff et al. 2016). The Massive Text Embedding Benchmark (MTEB), for text embedding models, reported that "no particular text embedding method dominates across all tasks" (Muennighoff et al. 2023).

Altenhoff, Adrian M., Brigitte Boeckmann, Salvador Capella-Gutierrez, et al. "Standardized Benchmarking in the Quest for Orthologs." Nature Methods 13, no. 5 (2016): 425–30. https://doi.org/10.1038/nmeth.3830.
Muennighoff, Niklas, Nouamane Tazi, Loic Magne, and Nils Reimers. "MTEB: Massive Text Embedding Benchmark." In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2014–37. Dubrovnik, Croatia, 2023. https://doi.org/10.18653/v1/2023.eacl-main.148.

From a search hit to a call

Each tool reports aligned regions: a stretch of the human protein matched to a stretch of a protein in the relative. A region becomes a call when it covers at least half of an annotated domain or feature on the relative's protein. The call names that annotation and sits on the human protein where the region is. A call is correct when the human protein carries that annotation and the call overlaps it with an IoU of at least 0.5. IoU (intersection over union) is the number of residues the call and the true region share, divided by the number of residues covered by either:

\[\text{IoU} = \frac{|\text{call} \cap \text{true region}|}{|\text{call} \cup \text{true region}|}\]

For example, a call on residues 30–129 and a true domain on residues 10–109 share 80 residues (30–109). Together they cover 120 (10–129). So IoU = 80 / 120 = 0.67, and the call counts as correct. A call of the right family that misses the region counts as wrong. The IoU is drawn as bars on a protein line, with ends you can move, on How the scores work.

A true annotation is reachable when its family occurs somewhere in the relative's proteome. A family the relative lacks cannot be copied from it by any tool.

The scores

Fmax

Fmax is the best balance of precision and recall a tool reaches at any score cutoff, counted protein by protein as in the CAFA function-prediction challenge. At a cutoff \(t\), the precision of protein \(i\) is its correct calls divided by all its calls scoring at least \(t\). Its recall is the true annotations found divided by its true annotations. Precision is averaged over the \(m(t)\) proteins with at least one call at or above \(t\). Recall is averaged over all \(n\) proteins with an annotation, and counts every true annotation, whether or not the relative could supply it:

\[\text{Pr}(t) = \frac{1}{m(t)}\sum_{i} \frac{\text{correct}_i(t)}{\text{calls}_i(t)}\]

\[\text{Rc}(t) = \frac{1}{n}\sum_{i} \frac{\text{found}_i(t)}{\text{true}_i}\]

\[F_{\max} = \max_t \frac{2\,\text{Pr}(t)\,\text{Rc}(t)}{\text{Pr}(t)+\text{Rc}(t)}\]

The cutoffs tried are 101 evenly spaced quantiles of the tool's call scores. Worked example: phmmer searching zebrafish, Pfam truth set, raw calls, all proteins. At the best cutoff (bit score on the tool's bit score) precision is 0.2471 and recall is 0.5108. So F = 2 × 0.2471 × 0.5108 / (0.2471 + 0.5108) = 0.3331, the Fmax in Table 1.

AUPRC

AUPRC, the area under the precision-recall curve, sums precision over every cutoff, weighted by the recall gained at that cutoff:

\[\text{AUPRC} = \sum_k (R_k - R_{k-1})\,P_k\]

Unlike Fmax, AUPRC pools all proteins. \(P_k\) is correct calls ÷ all judged calls at cutoff \(k\). \(R_k\) is reachable annotations found ÷ all reachable annotations. A tool that makes many calls per protein pays for each wrong one. A call in a part of a protein that no annotation covers is left out of \(P_k\), since nobody has looked there; Fmax counts it. Worked example, same row: AUPRC = 0.0291. The curve itself is in the source run (all_domain_curves.parquet), not in these tables, so the sum cannot be recomputed here.

Median IoU

Median IoU is the middle IoU of all correct calls, with single-residue annotations left out (an IoU against one residue says nothing about where the ends are). Worked example, same row: 0.700, so half of phmmer's correct calls on zebrafish overlap their true domain by at least that much. A correct call on a region longer than one residue has an IoU of at least 0.5. So the run stores 0 only when there is no such call. Either there are no correct calls at all, or the correct calls are on single residues only (the 1 aa length bin and the ACT_SITE, BINDING and SITE features). The pages show the first case as no correct calls and the second as correct calls only on single residues, never as 0. They tell the two apart by Fmax: an Fmax above 0 means at least one call was correct.

True instances

True instances (n) is the number of annotations on the human proteins in the subset, reachable or not. Same row: 2,435.

Core-hours per 1,000 queries

Core-hours are taken from the run's Nextflow task records: the cores a task reserved multiplied by its wall time, so a task that sat partly idle still counts all its cores. The tables record them per tool and setting, summed over every relative searched, with building the databases as separate rows. The leaderboard divides the search's core-hours by the number of relatives searched and by the 998 queries, times 1,000: the mean cost of searching one relative. Worked example: phmmer reserved 8 cores for 2,806 s over its 15 searches. That is 8 × 2,806 / 3,600 = 6.2356 core-hours, and 6.2356 / 15 × 1,000 / 998 = 0.4165 core-hours per 1,000 queries per relative. Peak RAM is the largest resident memory of any of those tasks. The tool cards give the database builds separately.

Numbers marked measured come from these records, on Sherlock (partition hns). No search used a GPU. ProstT5 has to turn sequences into 3Di shape letters before it can search; that step ran on one GPU as part of building its databases, queries included, and is on the ProstT5 card (0.664 GPU-hours over 3 databases), not in the per-query cost. Numbers marked reported would come from a tool's submitter, on hardware they name; none have been submitted yet.

Estimated cost

\[\text{cost\_usd\_est} = \text{CPU-hours} \times r_\text{cpu} + \text{GPU-hours} \times r_\text{gpu}\]

per 1,000 queries. The default \(r_\text{cpu}\) = $0.044625 per CPU-hour is half of AWS's on-demand price for a c7i.large (2 vCPU, $0.08925 per hour). The default \(r_\text{gpu}\) = $1.006 per GPU-hour is a g5.xlarge (one NVIDIA A10G). Both are for US East (N. Virginia), Linux, from the EC2 on-demand price page (price file, published 2026-10-12, read 2026-10-15). A core-hour is priced as one CPU-hour. Memory is not priced. Worked example: 0.4165 × 0.044625 + 0 × 1.006 = $0.01859 for phmmer against zebrafish. Change either price on the leaderboard page.

Truth sets

Pfam

Every Pfam-A domain with coordinates on the human proteins, from the Pfam regions file for UniProt reference proteomes. The same file supplies the domains on each relative's proteins, which are what a call copies. Pfam families are defined by profile models, so tools that build profiles (jackhmmer, HH-suite) are graded against the kind of model they build, and hmmscan against its own models. Release of the regions file: not recorded in the run; the hmmscan models are Pfam-A 38.2. hmmscan was scored on raw calls only, so it shows "not run" under dedup.

Pfam-N

Pfam-N is an extension of Pfam made by a deep-learning model: it labels regions the Pfam profile models missed. It is only published for Pfam 35.0. This is the one truth set the profile models did not produce. Its regions were filtered to the human proteins queried here. Few of them can be copied from any relative: against zebrafish, 44,246 of the 44,315 Pfam-N regions have no domain of the same family in the zebrafish proteome. So every tool scores low on Pfam-N, and the set says little about which tool is better.

Swiss-Prot features

Curated features from Swiss-Prot entries, of 12 types (the feature keys on the leaderboard, such as TRANSMEM for a membrane-spanning segment and ACT_SITE for an active site residue). They come from the literature, not from any sequence or structure model. Single-residue features (ACT_SITE, BINDING, SITE) are scored by whether the call covers the residue, since an IoU against one residue cannot reach 0.5. A Swiss-Prot feature counts as reachable when the relative has a protein that carries the same feature type and shares a Pfam family with the human protein.

Dedup against raw

Raw keeps every call a tool makes. One human region that matches 300 mouse kinases becomes 300 kinase calls on nearly the same residues, and all but one count as wrong. Dedup merges calls of the same family on the same human protein that overlap each other with an IoU of 0.5 or more, keeping the highest-scoring one. Two copies of a domain side by side barely overlap, so both are kept. Comparing the two separates finding a domain from reporting it many times. For phmmer on zebrafish (Pfam, all proteins): Fmax 0.333 raw and 0.457 dedup. The same calls drawn raw and after dedup, on a made-up protein: How the scores work.

Subsets of the proteins

The tables call each subset a stratum. There are four ways to cut the proteins:

• Nearest reviewed relative, % identity: for each human domain, the percent identity to the closest domain of the same family in the relative. Bins: 0–20, 20–30, 30–40, 40–60 and 60–100%. A domain whose family is absent from the relative is "no same-family domain".
• Nearest reviewed relative, Mya: the relative searched, labelled with how many million years ago its line split from ours.
• Region type and length: the length of the annotated region, in residues. Bins up to 60 aa are short features; 61 aa and up is the size of a folded domain. Whole-protein annotations were not scored in this run.
• Disorder: the mean AlphaFold confidence (pLDDT, 0–100) of the human protein. Low pLDDT marks regions without a fixed shape.
• Feature type: the Swiss-Prot feature key, for the Swiss-Prot truth set only.

The run computed each subset on its own, so two subsets cannot be combined; a combination with no row shows × and "not run".

Checks on this site

A headless browser renders every filter combination and confirms that each × on the leaderboard is a missing row in results.parquet, and that every number matches the tables. The source table checksums are in data/SOURCE.json of the site repository.

Experiments in the tables

The tables hold 3,975,483 rows from 12 experiments; the coverage page counts all of them. Their ids: alphabet_ranking_three_cases, dark_set_accuracy, dark_set_gain, dark_set_pfam_fmax, deep_twilight, elm_cover, qfo_region_midi, qfo_region_midi_plus, qfo_region_mini, scope40_v040, scope40_v040_below_kmin, scope40_v040_shuffled_k2_seed1.