How the scores work
The leaderboard ranks tools by three scores and one cost. Each section below works one of them through on a made-up protein that you can change. Every number in Figures 4 to 7 is computed in your browser from that made-up example, not from the benchmark.
Data from 15 Oct 2026, commit 4a4b1171 (source)
The made-up example
Protein A is 400 residues (aa) long and has three true domains: the known annotations a tool should find. A tool searched it and made seven calls. A call is a region the tool says carries a domain of a given family, copied from a protein of the relative it searched. Each call has the tool's score: higher means the tool is more sure. The family names are only labels.
A call is correct when it names the family of a true domain and overlaps that domain with an IoU of at least 0.5 (Figure 6 shows IoU). Each true domain can be found once. If two calls find the same domain, the higher-scoring one is correct and the other counts as wrong, as on the leaderboard.
In this example every domain can be found in the relative, and every call lies on an annotated part of the protein. So the small differences in what Fmax and AUPRC count (see Methods) do not arise here.
Fmax: the best single score cutoff
To use a tool's calls you pick a score cutoff: you keep every call scoring at least that much and drop the rest. At a cutoff, precision is the share of kept calls that are correct. Recall is the share of true domains that a kept, correct call found. A high cutoff keeps few calls, mostly correct; a low cutoff finds more domains but keeps more wrong calls. F combines the two into one number, high only when both are high:
\[F = \frac{2 \times \text{precision} \times \text{recall}}{\text{precision} + \text{recall}}\]
Fmax is the highest F over all cutoffs: the score of the tool at its best single cutoff. Move the slider in Figure 4 to try each cutoff.
How to read Figure 4
Top: the protein as a line with its three true domains as boxes. Each call is a bar below it, at the residues it covers, labelled with its family and score. Bottom: F at each cutoff. Move the cutoff and read off which calls are kept and what F they give; the ring marks the highest F, which is the Fmax.How the leaderboard does this over many proteins
The leaderboard has many proteins, not one. At each cutoff it works out precision and recall for each protein, then averages them over proteins; F comes from the two averages. This is the protein-by-protein Fmax of the CAFA function-prediction challenge. Precision is averaged over the proteins with at least one call kept; recall over every protein with a true annotation. The true instances (n) on the leaderboard are those true annotations, the denominator of recall. The leaderboard tries 101 cutoffs, evenly spaced through the tool's own scores; here every whole-number cutoff from 0 to 100 is tried.
AUPRC: every cutoff at once
Fmax keeps one cutoff. AUPRC, the area under the precision-recall curve, looks at every cutoff. Put every call in one list, best score first. Take the calls one at a time; after each, write down the precision and recall so far. Each call that finds a new domain adds a strip of area: its width is the recall gained and its height is the precision at that point.
\[\text{AUPRC} = \sum_k (R_k - R_{k-1})\,P_k\]
Here \(P_k\) and \(R_k\) are the precision and recall after the \(k\)-th call of the list. A wrong call adds no area itself, but it lowers the precision, and so the height of every strip after it. AUPRC runs from 0 to 1, and higher is better.
How to read Figure 5
Each point is one call, taken in score order; its position is the precision and recall after taking it. The shaded strips add up to the AUPRC. Lower the cutoff to add calls one by one, and switch protein B on to see Fmax and AUPRC move in opposite directions.Area of the calls taken so far: . AUPRC, with every call taken: . Fmax of the same proteins: .
| score | protein A only | A and B |
|---|---|---|
| Fmax | ||
| AUPRC |
The list behind the curve, best score first. Grey rows score below the cutoff.
| rank | protein | call | precision | recall | area added |
|---|
Why adding protein B moves Fmax up and AUPRC down
Protein B has five Ig domains, and its five calls are all correct, but they score 45 to 60. Fmax averages protein by protein, so protein B counts as one protein of two, whatever its number of domains. On its own protein B is perfect, so Fmax goes up. AUPRC pools every call of every protein into one list. Protein B's domains are five of the eight true domains. In the list its calls come after one or two of protein A's wrong calls (scores 70 and 55), so AUPRC goes down. Use Fmax when you will keep every call above one cutoff. Use AUPRC when you will rank calls and read down the list.
Median IoU: are the ends in the right place?
IoU (intersection over union) is the number of residues the call and the true domain share, divided by the number of residues covered by either. It is 1 when the call starts and ends where the domain does, and 0 when they do not overlap. A call needs an IoU of 0.5 or more to be correct.
\[\text{IoU} = \frac{|\text{call} \cap \text{true domain}|}{|\text{call} \cup \text{true domain}|}\]
How to read Figure 6
Top: a 200-residue protein with one true domain, and the call below it; the brackets underneath show the shared residues and the residues covered by either. It starts at the example on the Methods page (call 30–129, domain 10–109). Drag the call, or its handles, to see IoU change. Bottom: the IoU of each correct call of protein A in Figure 4, with your call added when it is correct; the dashed line is their median.Drag the black call to move it, or drag its handles to move one end.
Median IoU on the leaderboard
Median IoU on the leaderboard is the middle IoU of all of a tool's correct calls. Half of the correct calls match their domain at least that well. Since a correct call has an IoU of at least 0.5, the median runs from 0.5 to 1. Single-residue annotations, such as an active site, are left out: an IoU against one residue says nothing about where the ends are.
Raw or dedup: one domain, many calls
A tool often finds the same domain through several proteins of the relative, and reports one call for each. Raw keeps every call: the first copy can be correct, and every other copy counts as wrong. Dedup merges calls of the same family on the same protein that overlap each other with an IoU of 0.5 or more, keeping the highest-scoring one. Two copies of a domain side by side barely overlap, so both are kept.
How to read Figure 7
Switch between the two: dedup merges the two lower-scoring PH calls into the PH call scoring 95, and the table shows how the scores change.| score | raw | dedup |
|---|---|---|
| calls | ||
| precision, every call kept | ||
| Fmax | ||
| AUPRC |
When dedup raises a score, and when it lowers one
Here both merged calls were wrong copies, so recall does not change, and precision, Fmax and AUPRC go up. Dedup can also lower a score: when the higher-scoring call misses the domain's ends and a lower-scoring copy matches them, dedup keeps the one that misses. Comparing raw with dedup separates finding a domain from reporting it many times.
Cost: core-hours, GPU-hours and the price of an hour
Core-hours are the cores a search reserved multiplied by its wall time, so a task that sat partly idle still counts all its cores. They come from the run's own task records on Stanford's Sherlock cluster, so they are marked measured. Numbers a tool's submitter gives, on hardware they name, would be marked reported (CPU-h reported); none have been submitted yet. The leaderboard divides the search's core-hours by the number of relatives searched and scales them to 1,000 human proteins (queries). GPU-hours are the same for GPU time; no search here used a GPU. Peak RAM is the largest memory any of the search's tasks held, in GB; it is not priced.
The estimated cost, cost_usd_est, puts a price on each hour:
\[\begin{aligned}\text{cost\_usd\_est} &= \text{core-hours} \times r_\text{cpu}\\ &+ \text{GPU-hours} \times r_\text{gpu}\end{aligned}\]
per 1,000 queries, where \(r_\text{cpu}\) is the price of one CPU-hour and \(r_\text{gpu}\) of one GPU-hour, in US dollars. A core-hour is priced as one CPU-hour.
How to read Figure 8
It starts at phmmer's measured core-hours per 1,000 queries against zebrafish (from Table 1) and the site's default prices. These are AWS on-demand rates for a c7i.large CPU (half its 2-vCPU hourly price) and a g5.xlarge GPU. Drag any underlined number to see the cost change.Drag any underlined number left or right to change it (hold Shift for small steps), or click it and use the arrow keys.
cost per 1,000 queries = core-hours × $ per CPU-hour + GPU-hours × $ per GPU-hour
Type exact numbers
The made-up example is also in this page's JSON copy. Definitions of every score as the benchmark computes them, with a worked example from the real data, are on Methods.