{
 "page": "claims.html",
 "title": "Claims",
 "source": {
  "repo": "seanome/2024-kmerseek-analysis",
  "commit": "4a4b11714ddb740f8a7e324058b26bee8fb35af4",
  "commit_date": "2026-10-15T13:12:56Z",
  "path": "results/leaderboard",
  "copied_at": "2026-10-15T13:44:05Z",
  "sha256": {
   "results.parquet": "0c0cf94b5fc40587cab2ffbf7aaf84df9b5475037cc0d818ad2f452a9c73a837",
   "compute.parquet": "8a47ba2123c60cbb41b4833c9f96661934369137b72c5d81ef0808c98247e435",
   "coverage.md": "c10b26730d8dc44b16120bd2f6ee87eb8ed0db3efa40398b339ddac0b0cb7781",
   "provenance.json": "22e54f206354e42208f8a78c4f4827f980ac1db2ce13153f2b077274552e4141",
   "inventory.md": "8dc7040a38cf06af126bd08211edf1f6055c69fe9ac2b4088c6ba8f7f90e6c71",
   "pileup/regions.parquet": "e97701d2fcb601babe25c7d5c6c82760ea19e460e9bc1d555fe5f8f69cb4d6d8",
   "pileup/calls.parquet": "41bc1f6db7bf54f00370b11db2ffed61031d90a42895aa1697be26fc449ea958",
   "pileup/provenance.json": "5ed6e70dd9e82b5200f089b9da3afc62d0a10cc6a51da86715f5f26d3aac9b9e"
  },
  "pileup_commit": "39d81ec90815cacbf3744199d1fa8e99ab28c6f4"
 },
 "text": [
  {
   "tag": "h1",
   "text": "Claims"
  },
  {
   "tag": "p",
   "text": "What each tool's authors claim, the result that would disprove each claim, and where it stands. Only kmerseek has filed claims so far."
  },
  {
   "tag": "p",
   "text": "Data from 15 Oct 2026, commit 4a4b1171 (source)"
  },
  {
   "tag": "p",
   "text": "The build recomputes every verdict from the tables or checks it against the recorded notebook output, so nothing on a card is typed in by hand."
  },
  {
   "tag": "h2",
   "text": "kmerseek"
  },
  {
   "tag": "p",
   "text": "each row: the claim, the result that would disprove it, the verdict with its number, the evidence"
  },
  {
   "tag": "h3",
   "text": "● Holds (1)"
  },
  {
   "tag": "summary",
   "text": "Scope and related numbers"
  },
  {
   "tag": "p",
   "text": "Swiss-Prot features, raw calls, setting hp_thomas_dill2_k24_lcFalse, 998 human queries"
  },
  {
   "tag": "li",
   "text": "on Pfam domains, the best kmerseek setting for each mammal: −0.193 to +0.130 across 6 of 6; furthest: clupus −0.193"
  },
  {
   "tag": "li",
   "text": "settings that pass on Swiss-Prot features (raw or dedup): 2 of 30 setting and dedup choices pass (15 settings): hp_thomas_dill2_k24_lcFalse raw, hp_thomas_dill2_k24_lcFalse dedup"
  },
  {
   "tag": "p",
   "text": "Source: seanome/2024-kmerseek-analysis@4a4b1171 results.parquet"
  },
  {
   "tag": "li",
   "text": "●Parity with phmmer inside mammalsDisproved if: Fmax more than 0.05 from phmmer's on any of the 6 mammalsholds · −0.042 to +0.013 across 6 of 6 · leaderboard"
  },
  {
   "tag": "h3",
   "text": "○ Not yet tested (3)"
  },
  {
   "tag": "summary",
   "text": "Scope and related numbers"
  },
  {
   "tag": "p",
   "text": "Pfam-N regions, raw calls, kmerseek hp_thomas_dill2 settings, all 15 relatives"
  },
  {
   "tag": "li",
   "text": "share of Pfam pairs at 20-30% identity sharing one exact hp_thomas_dill2 23-mer: 0.038"
  },
  {
   "tag": "li",
   "text": "recall at a 0.001 false-positive rate for an exact 23-mer seed, Pfam 20-30%: 0.039"
  },
  {
   "tag": "li",
   "text": "○Exact HP seeds find homologs below 40% identityDisproved if: Fmax under 0.05 in the under-40% identity bins on Pfam-N, for every hydrophobic-polar settingnot yet tested · the 0-20%, 20-30%, 30-40% bins hold at most 1 region for any setting and relative; at least 30 are needed to decide · notebook 230 to 232"
  },
  {
   "tag": "summary",
   "text": "Scope and related numbers"
  },
  {
   "tag": "p",
   "text": "the 87 short Swiss-Prot features of notebook 244 where kmerseek drew the boundary and every other tool's call ran past it"
  },
  {
   "tag": "li",
   "text": "short features passing every criterion in notebook 244: 87"
  },
  {
   "tag": "li",
   "text": "across all Swiss-Prot features, not only short ones, kmerseek settings above a Kyte-Doolittle scan: 0 of 406"
  },
  {
   "tag": "li",
   "text": "○Short features under 60 aa placed at 50 to 90% identityDisproved if: a Kyte-Doolittle hydrophobicity window, with no search, lands on as many of the 87 casesnot yet tested · the Kyte-Doolittle window was not run on the 87 cases · notebook 244"
  },
  {
   "tag": "summary",
   "text": "Scope and related numbers"
  },
  {
   "tag": "p",
   "text": "Botryllus histocompatibility factor (BHF)"
  },
  {
   "tag": "li",
   "text": "best E-value of any human protein for BHF, any kmerseek setting (notebook 241): 0.609548"
  },
  {
   "tag": "li",
   "text": "○The BHF membrane tether is a find other tools missDisproved if: a Kyte-Doolittle window on BHF calls the same regionnot yet tested · no Kyte-Doolittle window has been run on BHF · notebook 241"
  },
  {
   "tag": "h3",
   "text": "⊘ Disproved (2)"
  },
  {
   "tag": "summary",
   "text": "Scope and related numbers"
  },
  {
   "tag": "p",
   "text": "1,000 Swiss-Prot queries against Swiss-Prot on one laptop; kmerseek hp k15, scaled 10"
  },
  {
   "tag": "li",
   "text": "kmerseek search peak memory, GB: 82.68"
  },
  {
   "tag": "li",
   "text": "MMseqs2 -s 5.7 search peak memory, GB: 2.55"
  },
  {
   "tag": "p",
   "text": "Source: notebooks/239_laptop_scale_swissprot.ipynb @ ba20e8b, cell 1"
  },
  {
   "tag": "li",
   "text": "⊘Faster per query than MMseqs2Disproved if: kmerseek answers fewer queries per second than MMseqs2 on the same 1,000 Swiss-Prot queriesdisproved 13 Sep 2026 · 2.29 queries per second against MMseqs2's 78.6 · notebook 239"
  },
  {
   "tag": "summary",
   "text": "Scope and related numbers"
  },
  {
   "tag": "p",
   "text": "SCOPe40, hp_thomas_dill at k 26, family level"
  },
  {
   "tag": "li",
   "text": "per-query family sensitivity, kmerseek better than FoldSeek on 228 shared queries (Wilcoxon): p=0.1171"
  },
  {
   "tag": "li",
   "text": "best kmerseek 0.4 AUC over all 5,713 reference queries (family), any setting: 0.295 (mmseqs12.k6.c_max|region_evalue, n = 5,713 queries)"
  },
  {
   "tag": "li",
   "text": "best kmerseek 0.4 AUC over only the queries it fires on (family), any setting: 0.999 (uniprot18.k9.exact|f_weighted_target_in_query, n = 753 queries)"
  },
  {
   "tag": "p",
   "text": "Source: notebooks/076_hp_alphabet_sweep_per_class.ipynb @ 31f53c1, cell 19"
  },
  {
   "tag": "li",
   "text": "⊘Outperforms Foldseek on SCOPeDisproved if: inside FoldSeek's 5,713-query SCOPe40 benchmark, kmerseek detects no query that FoldSeek missesdisproved 30 Aug 2026 · fired on 188 of 5,713 benchmark queries (3.3%); 0 that FoldSeek missed · notebook 076"
  }
 ],
 "data": {
  "claims": [
   {
    "tool": "kmerseek",
    "title": "kmerseek",
    "updated": "2026-10-15",
    "file": "claims/kmerseek.yaml",
    "claims": [
     {
      "id": "phmmer-parity-mammals",
      "claim": "Parity with phmmer inside mammals",
      "kill": "Fmax more than 0.05 from phmmer's on any of the 6 mammals",
      "scope": "Swiss-Prot features, raw calls, setting hp_thomas_dill2_k24_lcFalse, 998 human queries",
      "status": "alive",
      "verdict": "−0.042 to +0.013 across 6 of 6",
      "date": null,
      "source": "seanome/2024-kmerseek-analysis@4a4b1171 results.parquet",
      "context": [
       {
        "label": "on Pfam domains, the best kmerseek setting for each mammal",
        "text": "−0.193 to +0.130 across 6 of 6; furthest: clupus −0.193",
        "source": "seanome/2024-kmerseek-analysis@4a4b1171 results.parquet"
       },
       {
        "label": "settings that pass on Swiss-Prot features (raw or dedup)",
        "text": "2 of 30 setting and dedup choices pass (15 settings): hp_thomas_dill2_k24_lcFalse raw, hp_thomas_dill2_k24_lcFalse dedup",
        "source": null
       }
      ],
      "evidence": {
       "label": "leaderboard",
       "href": "index.html"
      }
     },
     {
      "id": "hp-seeds-below-40",
      "claim": "Exact HP seeds find homologs below 40% identity",
      "kill": "Fmax under 0.05 in the under-40% identity bins on Pfam-N, for every hydrophobic-polar setting",
      "scope": "Pfam-N regions, raw calls, kmerseek hp_thomas_dill2 settings, all 15 relatives",
      "status": "untested",
      "verdict": "the 0-20%, 20-30%, 30-40% bins hold at most 1 region for any setting and relative; at least 30 are needed to decide",
      "date": null,
      "source": null,
      "context": [
       {
        "label": "share of Pfam pairs at 20-30% identity sharing one exact hp_thomas_dill2 23-mer",
        "text": "0.038",
        "source": "notebooks/230_hp_class_conservation_in_aligned_homologs.ipynb @ ba20e8b, cell 11"
       },
       {
        "label": "recall at a 0.001 false-positive rate for an exact 23-mer seed, Pfam 20-30%",
        "text": "0.039",
        "source": "notebooks/232_mismatch_tolerant_seed_extend.ipynb @ ba20e8b, cell 5"
       }
      ],
      "evidence": {
       "label": "nb 230 to 232",
       "href": "https://github.com/seanome/2024-kmerseek-analysis/blob/fe72334b6c6c34ae19728ae292ad0ff1c4ae95d9/data/experiments.json"
      }
     },
     {
      "id": "faster-than-mmseqs2",
      "claim": "Faster per query than MMseqs2",
      "kill": "kmerseek answers fewer queries per second than MMseqs2 on the same 1,000 Swiss-Prot queries",
      "scope": "1,000 Swiss-Prot queries against Swiss-Prot on one laptop; kmerseek hp k15, scaled 10",
      "status": "retired",
      "verdict": "2.29 queries per second against MMseqs2's 78.6",
      "date": "2026-09-13",
      "source": "notebooks/239_laptop_scale_swissprot.ipynb @ ba20e8b, cell 1",
      "context": [
       {
        "label": "kmerseek search peak memory, GB",
        "text": "82.68",
        "source": "notebooks/239_laptop_scale_swissprot.ipynb @ ba20e8b, cell 1"
       },
       {
        "label": "MMseqs2 -s 5.7 search peak memory, GB",
        "text": "2.55",
        "source": "notebooks/239_laptop_scale_swissprot.ipynb @ ba20e8b, cell 1"
       }
      ],
      "evidence": {
       "label": "nb 239",
       "href": "https://github.com/seanome/2024-kmerseek-analysis/blob/ba20e8b/notebooks/239_laptop_scale_swissprot.ipynb"
      }
     },
     {
      "id": "beats-foldseek-scope",
      "claim": "Outperforms Foldseek on SCOPe",
      "kill": "inside FoldSeek's 5,713-query SCOPe40 benchmark, kmerseek detects no query that FoldSeek misses",
      "scope": "SCOPe40, hp_thomas_dill at k 26, family level",
      "status": "retired",
      "verdict": "fired on 188 of 5,713 benchmark queries (3.3%); 0 that FoldSeek missed",
      "date": "2026-08-30",
      "source": "notebooks/076_hp_alphabet_sweep_per_class.ipynb @ 31f53c1, cell 19",
      "context": [
       {
        "label": "per-query family sensitivity, kmerseek better than FoldSeek on 228 shared queries (Wilcoxon)",
        "text": "p=0.1171",
        "source": "notebooks/076_hp_alphabet_sweep_per_class.ipynb @ 31f53c1, cell 13"
       },
       {
        "label": "best kmerseek 0.4 AUC over all 5,713 reference queries (family), any setting",
        "text": "0.295 (mmseqs12.k6.c_max|region_evalue, n = 5,713 queries)",
        "source": null
       },
       {
        "label": "best kmerseek 0.4 AUC over only the queries it fires on (family), any setting",
        "text": "0.999 (uniprot18.k9.exact|f_weighted_target_in_query, n = 753 queries)",
        "source": null
       }
      ],
      "evidence": {
       "label": "nb 076",
       "href": "https://github.com/seanome/2024-kmerseek-analysis/blob/31f53c1/notebooks/076_hp_alphabet_sweep_per_class.ipynb"
      }
     },
     {
      "id": "short-features-placed",
      "claim": "Short features under 60 aa placed at 50 to 90% identity",
      "kill": "a Kyte-Doolittle hydrophobicity window, with no search, lands on as many of the 87 cases",
      "scope": "the 87 short Swiss-Prot features of nb 244 where kmerseek drew the boundary and every other tool's call ran past it",
      "status": "untested",
      "verdict": "the Kyte-Doolittle window was not run on the 87 cases",
      "date": null,
      "source": null,
      "context": [
       {
        "label": "short features passing every criterion in nb 244",
        "text": "87",
        "source": "notebooks/244_hero_example_candidates.ipynb @ 972948e, cell 8"
       },
       {
        "label": "across all Swiss-Prot features, not only short ones, kmerseek settings above a Kyte-Doolittle scan",
        "text": "0 of 406",
        "source": "notebooks/231_swissprot_composition_controls.ipynb @ ba20e8b, cell 5"
       }
      ],
      "evidence": {
       "label": "nb 244",
       "href": "https://github.com/seanome/2024-kmerseek-analysis/blob/fe72334b6c6c34ae19728ae292ad0ff1c4ae95d9/data/experiments.json"
      }
     },
     {
      "id": "bhf-tether",
      "claim": "The BHF membrane tether is a find other tools miss",
      "kill": "a Kyte-Doolittle window on BHF calls the same region",
      "scope": "Botryllus histocompatibility factor (BHF)",
      "status": "untested",
      "verdict": "no Kyte-Doolittle window has been run on BHF",
      "date": null,
      "source": null,
      "context": [
       {
        "label": "best E-value of any human protein for BHF, any kmerseek setting (nb 241)",
        "text": "0.609548",
        "source": "notebooks/241_alphabet_ranking_BCL2-Ced9_P66-CD47_BHF.ipynb @ 979de89, cell 19"
       }
      ],
      "evidence": {
       "label": "nb 241",
       "href": "https://github.com/seanome/2024-kmerseek-analysis/blob/fe72334b6c6c34ae19728ae292ad0ff1c4ae95d9/data/experiments.json"
      }
     }
    ],
    "counts": {
     "alive": 1,
     "retired": 2,
     "untested": 3
    },
    "last_kill": "2026-09-13",
    "source": {
     "tables": {
      "repo": "seanome/2024-kmerseek-analysis",
      "commit": "4a4b11714ddb740f8a7e324058b26bee8fb35af4",
      "commit_date": "2026-10-15T13:12:56Z"
     },
     "inventory": "seanome/2024-kmerseek-analysis data/experiments.json @ fe72334b (PR 97)"
    }
   }
  ]
 }
}