|
|
|
@@ -932,6 +932,17 @@
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
|
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
|
|
|
|
|
<span class="md-ellipsis">
|
|
|
|
|
|
|
|
|
|
Automatic α estimation (--gamma-shape auto)
|
|
|
|
|
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
@@ -943,6 +954,45 @@
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
|
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
|
|
|
|
|
<span class="md-ellipsis">
|
|
|
|
|
|
|
|
|
|
α-estimation literature survey and design discussion (2026-09-11)
|
|
|
|
|
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
|
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
|
|
|
|
|
<span class="md-ellipsis">
|
|
|
|
|
|
|
|
|
|
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
|
|
|
|
|
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
|
|
|
|
|
<ul class="md-nav__list">
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
|
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
|
|
|
|
|
<span class="md-ellipsis">
|
|
|
|
|
|
|
|
|
|
Explicit --session (implemented 2026-09-12)
|
|
|
|
|
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
</ul>
|
|
|
|
|
</nav>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
@@ -1691,7 +1741,7 @@
|
|
|
|
|
<span class="md-ellipsis">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Sibling annex (discussion)
|
|
|
|
|
Sibling annex
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -2268,6 +2318,17 @@
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
|
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
|
|
|
|
|
<span class="md-ellipsis">
|
|
|
|
|
|
|
|
|
|
Automatic α estimation (--gamma-shape auto)
|
|
|
|
|
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
@@ -2279,6 +2340,45 @@
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
|
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
|
|
|
|
|
<span class="md-ellipsis">
|
|
|
|
|
|
|
|
|
|
α-estimation literature survey and design discussion (2026-09-11)
|
|
|
|
|
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
|
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
|
|
|
|
|
<span class="md-ellipsis">
|
|
|
|
|
|
|
|
|
|
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
|
|
|
|
|
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
|
|
|
|
|
<ul class="md-nav__list">
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
|
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
|
|
|
|
|
<span class="md-ellipsis">
|
|
|
|
|
|
|
|
|
|
Explicit --session (implemented 2026-09-12)
|
|
|
|
|
|
|
|
|
|
</span>
|
|
|
|
|
</a>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
</ul>
|
|
|
|
|
</nav>
|
|
|
|
|
|
|
|
|
|
</li>
|
|
|
|
|
|
|
|
|
|
<li class="md-nav__item">
|
|
|
|
@@ -4476,9 +4576,75 @@ F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
|
|
|
|
|
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
|
|
|
|
|
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
|
|
|
|
|
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
|
|
|
|
|
formula as the base correction, weighted by a shape parameter <code>α</code> supplied
|
|
|
|
|
by the user (<code>--gamma-shape <alpha></code>), not estimated by ML. A modifier on
|
|
|
|
|
existing values, not a separate enum arm per distance.</p>
|
|
|
|
|
formula as the base correction, weighted by a shape parameter <code>α</code> either
|
|
|
|
|
supplied by the user (<code>--gamma-shape <alpha></code>) or estimated from the data
|
|
|
|
|
(<code>--gamma-shape auto</code>/<code>estimate</code>, method-of-moments — not ML; see
|
|
|
|
|
"Automatic α estimation" below). A modifier on existing values, not a
|
|
|
|
|
separate enum arm per distance.</p>
|
|
|
|
|
<h3 id="automatic-estimation-gamma-shape-auto">Automatic α estimation (<code>--gamma-shape auto</code>)</h3>
|
|
|
|
|
<p><strong>Correction (verified against the primary source, 2026-09-11):</strong> Jin &
|
|
|
|
|
Nei (1990) itself (<em>"Limitations of the Evolutionary Parsimony Method of
|
|
|
|
|
Phylogenetic Analysis"</em>, Mol. Biol. Evol. 7(2):82–102 — the paper this
|
|
|
|
|
whole <code>+Γ</code> correction is cited from, confirmed algebraically to match this
|
|
|
|
|
codebase's <code>corrected_log</code>/<code>k2p</code> exactly against their eq. A4, general, and
|
|
|
|
|
A8, the <code>a = 1</code> case) contains <strong>no</strong> data-driven α-estimation procedure.
|
|
|
|
|
Their own recommendation (p. 98) is a fixed default: <em>"we suggest that the
|
|
|
|
|
gamma distance with a = 1 [eq. A8] be used. However, one may choose a
|
|
|
|
|
different gamma distance, estimating a from data. Wilson et al. (1989)
|
|
|
|
|
recently used a distance with a = 1/2 for restriction-site data of
|
|
|
|
|
mitochondrial DNA in hominoids."</em> — i.e. Jin & Nei explicitly punt
|
|
|
|
|
data-driven estimation to a <em>different</em> paper (Wilson et al. 1989), not
|
|
|
|
|
read/verified here. The estimator below is therefore <strong>not</strong> "Jin & Nei's
|
|
|
|
|
method" under any framing — that attribution (present in an earlier
|
|
|
|
|
revision of this section) was wrong, not just under-cited.</p>
|
|
|
|
|
<p><strong>Implemented</strong> (<code>PartitionDispersion</code>, <code>obikphylo/src/siblings/algorithms/pairwise.rs</code>)
|
|
|
|
|
as an independent method-of-moments estimator, unrelated to any specific
|
|
|
|
|
published procedure: pools substitution counts by <strong>partition</strong> rather
|
|
|
|
|
than by genome pair, during the same <code>reduce_pairwise</code> pass that builds
|
|
|
|
|
<code>PairwiseTally</code> (no second scan).</p>
|
|
|
|
|
<p>For partition <code>i</code>: <code>n_i</code> = substitutions pooled over every genome pair,
|
|
|
|
|
<code>L_i</code> = eligible loci pooled over every genome pair, <code>R_i = n_i / L_i</code>.
|
|
|
|
|
Modeling among-site rate heterogeneity the same way as the <code>+Γ</code> correction
|
|
|
|
|
itself (a <code>Gamma(α, α)</code>-distributed, mean-1, multiplicative rate shared by
|
|
|
|
|
every locus in a partition — the classical Poisson-Gamma/negative-binomial
|
|
|
|
|
mixture, the general identity behind gamma-rate-heterogeneity corrections,
|
|
|
|
|
also behind Uzzell & Corbin's (1971) original observation that substitution
|
|
|
|
|
counts across sites/regions are over-dispersed relative to Poisson):</p>
|
|
|
|
|
<div class="arithmatex">\[
|
|
|
|
|
\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
|
|
|
|
|
\]</div>
|
|
|
|
|
<p>Weighting each partition's squared deviation by its own <code>L_i</code> cancels the
|
|
|
|
|
Poisson term before attributing what's left to <code>α</code>:</p>
|
|
|
|
|
<div class="arithmatex">\[
|
|
|
|
|
\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
|
|
|
|
|
V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
|
|
|
|
|
\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
|
|
|
|
|
\]</div>
|
|
|
|
|
<p>where <code>\bar L</code> is the mean partition size. Returns "no estimate" (falls
|
|
|
|
|
back to the uncorrected formula, warns) when fewer than 2 partitions have
|
|
|
|
|
data, <code>\hat\mu \le 0</code>, or <code>V</code> doesn't exceed the Poisson floor
|
|
|
|
|
<code>\hat\mu/\bar L</code> — no detectable over-dispersion, <code>α</code> would be unbounded.</p>
|
|
|
|
|
<p><strong>Caveat, stated explicitly rather than left implicit</strong>: unlike every
|
|
|
|
|
closed-form correction in <code>snp_distance.rs</code> (each verified line-by-line
|
|
|
|
|
against <code>ape</code>'s <code>dist_dna.c</code>, and now also against Jin & Nei 1990 directly
|
|
|
|
|
for the base <code>+Γ</code> formula), this estimator is derived from first
|
|
|
|
|
principles (the general Poisson-Gamma/negative-binomial identity) with no
|
|
|
|
|
primary-source procedure behind it at all — not Jin & Nei's (confirmed
|
|
|
|
|
above), and Wilson et al. (1989), the paper they point to instead, hasn't
|
|
|
|
|
been read/verified either. Mathematically self-consistent (re-derived and
|
|
|
|
|
checked, not guessed), but a from-scratch method, not a literature
|
|
|
|
|
implementation. If <code>--gamma-shape</code> needs a value with a literature
|
|
|
|
|
pedigree rather than an estimated one, Jin & Nei's own stated default,
|
|
|
|
|
<code>α = 1</code> (<code>--gamma-shape 1</code>), is the better-supported choice today.</p>
|
|
|
|
|
<p>Deliberately <strong>not</strong> gated by <code>--sankoff-ratio-ceiling</code> the way
|
|
|
|
|
<code>base_pair_tally</code> is (same precedent as <code>cardinality_tally</code> — see its own
|
|
|
|
|
doc comment): that filter excludes individual saturated <em>pairs</em> from a
|
|
|
|
|
composition estimate computed once at the very end, from the complete
|
|
|
|
|
<code>PairwiseTally</code>; the partition axis needed here only exists transiently,
|
|
|
|
|
one partition at a time, while <code>PairwiseTally</code> is still being built — long
|
|
|
|
|
before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
|
|
|
|
|
is known. <code>--exclude-genome</code> isn't applied either, matching
|
|
|
|
|
<code>reduce_pairwise</code>'s own raw per-pair fold.</p>
|
|
|
|
|
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
|
|
|
|
|
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
|
|
|
|
|
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
|
|
|
|
@@ -4591,6 +4757,430 @@ special-casing K80). K81/T92/TN93's gamma branches follow the same
|
|
|
|
|
mechanical substitution but weren't independently checked against an
|
|
|
|
|
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
|
|
|
|
|
silently assumed correct.</p>
|
|
|
|
|
<h3 id="-estimation-literature-survey-and-design-discussion-2026-09-11">α-estimation literature survey and design discussion (2026-09-11)</h3>
|
|
|
|
|
<p>Follow-up discussion after implementing <code>PartitionDispersion::estimate_alpha</code>
|
|
|
|
|
(previous section), triggered by the user pointing out that the literature
|
|
|
|
|
conflates two distinct problems: <em>estimating α from an alignment</em> versus
|
|
|
|
|
<em>using a given α in a Jin-Nei-style corrected distance</em>. Jin & Nei (1990)
|
|
|
|
|
itself is squarely in the second camp — α is a user-supplied parameter
|
|
|
|
|
there, not something their paper estimates (confirmed by reading the paper
|
|
|
|
|
directly, see previous section). This section catalogs the actual
|
|
|
|
|
α-estimation literature and records which approaches were considered and
|
|
|
|
|
why each was or wasn't adopted.</p>
|
|
|
|
|
<p><strong>Bibliography</strong> (methodological papers, not exhaustive users of <code>+Γ</code>):</p>
|
|
|
|
|
<table>
|
|
|
|
|
<thead>
|
|
|
|
|
<tr>
|
|
|
|
|
<th>Method</th>
|
|
|
|
|
<th>Reference</th>
|
|
|
|
|
<th>Needs a tree?</th>
|
|
|
|
|
<th>Estimates "true" α or a task-specific one?</th>
|
|
|
|
|
<th>Cost</th>
|
|
|
|
|
</tr>
|
|
|
|
|
</thead>
|
|
|
|
|
<tbody>
|
|
|
|
|
<tr>
|
|
|
|
|
<td>ML, continuous Γ (foundational)</td>
|
|
|
|
|
<td><strong>Gu, Fu & Li (1995)</strong>, MBE 12:546–557 — read in full</td>
|
|
|
|
|
<td>yes, topology fixed a priori</td>
|
|
|
|
|
<td>true α (+ θ jointly, invariant+Γ)</td>
|
|
|
|
|
<td>very high, ≤5–6 taxa in practice</td>
|
|
|
|
|
</tr>
|
|
|
|
|
<tr>
|
|
|
|
|
<td>ML, discrete Γ</td>
|
|
|
|
|
<td>Yang (1993/94), MBE 10:1396; JME 39:306</td>
|
|
|
|
|
<td>yes</td>
|
|
|
|
|
<td>true α</td>
|
|
|
|
|
<td>high</td>
|
|
|
|
|
</tr>
|
|
|
|
|
<tr>
|
|
|
|
|
<td>Parsimony-based ML on inferred changes</td>
|
|
|
|
|
<td>Yang & Kumar (1996), MBE 13:650</td>
|
|
|
|
|
<td>yes, equal branch lengths assumed</td>
|
|
|
|
|
<td>true α, biased when branches unequal</td>
|
|
|
|
|
<td>low</td>
|
|
|
|
|
</tr>
|
|
|
|
|
<tr>
|
|
|
|
|
<td>Corrected substitutions/site → NB fit</td>
|
|
|
|
|
<td><strong>Gu & Zhang (1997)</strong>, MBE 14:1106 — read in full</td>
|
|
|
|
|
<td>yes, + ancestral reconstruction</td>
|
|
|
|
|
<td>true α, close to ML accuracy</td>
|
|
|
|
|
<td>low (given a tree)</td>
|
|
|
|
|
</tr>
|
|
|
|
|
<tr>
|
|
|
|
|
<td>Topology-optimizing grid search</td>
|
|
|
|
|
<td><strong>Guindon & Gascuel (2002)</strong>, MBE 19:534 — read in full</td>
|
|
|
|
|
<td>yes, but built <em>from</em> the candidate distances themselves (BIONJ)</td>
|
|
|
|
|
<td><strong>not</strong> true α — deliberately biased <em>upward</em> for topological accuracy</td>
|
|
|
|
|
<td>low–moderate (grid × tree build)</td>
|
|
|
|
|
</tr>
|
|
|
|
|
<tr>
|
|
|
|
|
<td>Free (non-parametric) rate distribution</td>
|
|
|
|
|
<td>Susko et al. (2003), Syst. Biol. 52:594</td>
|
|
|
|
|
<td>yes</td>
|
|
|
|
|
<td>tests the Γ assumption itself</td>
|
|
|
|
|
<td>high</td>
|
|
|
|
|
</tr>
|
|
|
|
|
<tr>
|
|
|
|
|
<td>Parsimony counts/site → NB fit</td>
|
|
|
|
|
<td><strong>Wakeley (1993)</strong>, JME 37:613 — read in full</td>
|
|
|
|
|
<td>yes, NJ tree + Fitch parsimony ancestral reconstruction</td>
|
|
|
|
|
<td>true α, biased <em>upward</em> by parsimony (conservative)</td>
|
|
|
|
|
<td>low (given a tree)</td>
|
|
|
|
|
</tr>
|
|
|
|
|
<tr>
|
|
|
|
|
<td>Poisson-Gamma/negative-binomial, foundational</td>
|
|
|
|
|
<td>Uzzell & Corbin (1971), Science 172:1089 — <strong>not read</strong>, only seen via citations</td>
|
|
|
|
|
<td>unknown — likely also count-per-site/tree-based, unverified</td>
|
|
|
|
|
<td>true α (indirect, via NB overdispersion)</td>
|
|
|
|
|
<td>unknown</td>
|
|
|
|
|
</tr>
|
|
|
|
|
<tr>
|
|
|
|
|
<td>Bayesian posterior over α</td>
|
|
|
|
|
<td>BPP/MrBayes/BEAST2/RevBayes</td>
|
|
|
|
|
<td>yes</td>
|
|
|
|
|
<td>posterior, not point estimate</td>
|
|
|
|
|
<td>very high</td>
|
|
|
|
|
</tr>
|
|
|
|
|
</tbody>
|
|
|
|
|
</table>
|
|
|
|
|
<p><strong>Correction (2026-09-11, after reading Wakeley in full):</strong> an earlier
|
|
|
|
|
revision of this table filed Wakeley (1993) under "no tree needed" —
|
|
|
|
|
wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei
|
|
|
|
|
1987, on the real mtDNA data; coalescent-simulated for the simulation
|
|
|
|
|
study), reconstruct ancestral states at internal nodes by Fitch (1971)
|
|
|
|
|
parsimony, count the <em>minimum</em> number of changes per site this implies,
|
|
|
|
|
then fit either a two-rate Poisson mixture or a gamma-distributed-rates
|
|
|
|
|
model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
|
|
|
|
|
Johnson & Kotz 1969) to the resulting per-site count distribution — the
|
|
|
|
|
same general "tree + ancestral reconstruction + per-site counts + NB fit"
|
|
|
|
|
shape as Gu & Zhang (1997), just using raw parsimony counts directly
|
|
|
|
|
rather than Gu & Zhang's multiple-hit-corrected <code>k̂</code>. Confirmed by
|
|
|
|
|
simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
|
|
|
|
|
mean <em>and</em> (more severely) the variance of per-site change counts, which
|
|
|
|
|
biases <code>α</code> <strong>upward</strong> (toward apparent uniformity) — consistent with Gu &
|
|
|
|
|
Zhang's own citation of Wakeley for exactly this bias, and with the
|
|
|
|
|
general "parsimony overestimates α" pattern noted throughout this
|
|
|
|
|
literature. Real hypervariable-region-1 mtDNA data gave <code>α̂ ≈ 0.44–0.60</code>
|
|
|
|
|
per data set, <code>0.47</code> combined — despite the conservative bias, still
|
|
|
|
|
solidly in the "strong heterogeneity" range, which is the paper's own
|
|
|
|
|
point: the method is biased but usefully conservative, not useless.</p>
|
|
|
|
|
<p><strong>Where <code>PartitionDispersion::estimate_alpha</code> actually sits</strong>: nowhere in
|
|
|
|
|
this table's tree-dependent rows. It shares only the <em>abstract
|
|
|
|
|
mathematical identity</em> (Poisson-Gamma mixture ⟹ negative binomial
|
|
|
|
|
over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a
|
|
|
|
|
genuinely different, tree-free and ancestral-reconstruction-free route:
|
|
|
|
|
pairwise genome-to-genome substitution counts pooled by <em>partition</em>
|
|
|
|
|
instead of per-site counts pooled across an alignment's columns after
|
|
|
|
|
ancestral state reconstruction. As far as this survey has established
|
|
|
|
|
(Uzzell & Corbin not read, so not fully ruled out), <strong>no tree and no
|
|
|
|
|
ancestral reconstruction of any kind</strong> appears to be a genuinely
|
|
|
|
|
distinguishing property of <code>PartitionDispersion</code>, not something it
|
|
|
|
|
inherits from prior art — consistent with it being an independent
|
|
|
|
|
derivation rather than a literature implementation (see previous
|
|
|
|
|
section's caveat, now on firmer footing).</p>
|
|
|
|
|
<p><strong>Why a tree-free estimate is even valid — the identifiability question.</strong>
|
|
|
|
|
Raised directly by the user after reading Gu, Fu & Li (1995), which states
|
|
|
|
|
explicitly: <em>"one cannot estimate the rate heterogeneity when the number
|
|
|
|
|
of sequences is <3."</em> This is a real non-identifiability result, not a
|
|
|
|
|
practical inconvenience: with only 2 sequences, a site's history is
|
|
|
|
|
summarized by one binary observation (differ / don't differ), collapsed
|
|
|
|
|
across all sites into two aggregate numbers (transition and transversion
|
|
|
|
|
proportions) — divergence time and among-site rate variance are
|
|
|
|
|
confounded in that pair of numbers, with no way to separate them. Every
|
|
|
|
|
method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3
|
|
|
|
|
sequences <em>sharing one evolutionary history</em> (a tree) specifically to get
|
|
|
|
|
multiple independent looks at the <em>same</em> site's rate across different
|
|
|
|
|
lineages, which is what identifies its variance.</p>
|
|
|
|
|
<p><code>PartitionDispersion</code> doesn't violate this, because it isn't estimating
|
|
|
|
|
the same quantity. The literature above estimates rate variation <strong>across
|
|
|
|
|
sites, within one shared tree</strong>. <code>PartitionDispersion</code> estimates rate
|
|
|
|
|
variation <strong>across partitions, pooled over every genome pair in the
|
|
|
|
|
index</strong> — each genome pair stands in for one independent draw of a
|
|
|
|
|
partition's relative rate, the same identifying role multiple lineages
|
|
|
|
|
play in the tree-based methods, just substituting "genome pair" for
|
|
|
|
|
"lineage" and "partition" for "site." With <code>P</code> genome pairs (potentially
|
|
|
|
|
in the hundreds here) all contributing to every partition's pooled count,
|
|
|
|
|
the ≥3-samples identifiability requirement is met by the pair count, not
|
|
|
|
|
by tree depth — closer in spirit to Jin & Nei's own original motivating
|
|
|
|
|
scenario (comparing several genomic <em>regions</em> across one fixed panel of
|
|
|
|
|
taxa) than to single-tree site-rate estimation.</p>
|
|
|
|
|
<p>This substitution carries its own assumption, which must be named rather
|
|
|
|
|
than left implicit: that a partition's <em>relative</em> rate (fast/slow) is
|
|
|
|
|
reasonably stable <strong>across genome pairs</strong>, not just across sites within
|
|
|
|
|
one pair — the direct analogue of the "no lineage-specific rate variation,
|
|
|
|
|
only site-specific" assumption every tree-based <code>+Γ</code> method already makes,
|
|
|
|
|
just moved from lineage→pair and site→partition. If this breaks — e.g. an
|
|
|
|
|
index mixing very closely related and very divergent genome pairs, where a
|
|
|
|
|
partition's saturation behavior differs qualitatively between the two —
|
|
|
|
|
the estimate could be misled in a way a tree-based method would at least
|
|
|
|
|
have the topology to detect and a tree-free, pool-everything method
|
|
|
|
|
cannot. Not yet tested against real data with strongly heterogeneous
|
|
|
|
|
pairwise divergence; worth keeping in mind as the main open validity
|
|
|
|
|
question for <code>--gamma-shape auto</code>, not the citation question (now settled)
|
|
|
|
|
this whole discussion started from.</p>
|
|
|
|
|
<p><strong>Gu, Fu & Li (1995), read in full — the foundational ML method, not a
|
|
|
|
|
candidate to implement.</strong> Models rate variation as invariant+Γ (<code>θ</code> =
|
|
|
|
|
proportion of invariant sites, <code>α</code> = gamma shape among the variable sites),
|
|
|
|
|
and derives the exact joint likelihood over nucleotide <em>configurations</em>
|
|
|
|
|
across all <code>n</code> sequences on a <strong>fixed, a-priori-known tree topology</strong> —
|
|
|
|
|
for 3 sequences this is closed-form (Jukes-Cantor), for <code>n</code> sequences it's
|
|
|
|
|
a sum over <code>4^n</code> configurations weighted by coefficients tied to the
|
|
|
|
|
specific topology's branching structure (their eq. 32–34), with branch
|
|
|
|
|
lengths, <code>α</code>, and <code>θ</code> jointly optimized by Newton-Raphson (Hessian over all
|
|
|
|
|
free parameters). The site-rate integral has the same closed form
|
|
|
|
|
<code>E = θ + (1-θ)(1+D/α)^{-α}</code> that underlies <code>+Γ</code> distance corrections
|
|
|
|
|
generally (the same mechanism as <code>corrected_log</code>, not a coincidence — Jin &
|
|
|
|
|
Nei's own formula is the <code>θ=0</code> special case applied to a pairwise <code>D</code>
|
|
|
|
|
rather than a whole-tree one). Explicitly stated by the authors as
|
|
|
|
|
infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
|
|
|
|
|
candidate for obikmer's typical genome counts. Notable contribution worth
|
|
|
|
|
keeping in mind regardless: <code>ρ = (1+θα)/(1+α)</code> is shown to be a more robust
|
|
|
|
|
summary of rate heterogeneity than <code>α</code> alone whenever <code>θ</code> isn't negligible
|
|
|
|
|
— <code>α</code> and <code>θ</code> are confounded (nearly-invariant sites can come from either a
|
|
|
|
|
small <code>α</code>'s left tail or from a nonzero <code>θ</code>), so <code>α</code> alone can be very
|
|
|
|
|
unstable while <code>ρ</code> stays well-behaved. <code>PartitionDispersion</code> has no <code>θ</code>
|
|
|
|
|
term at all (no invariant-site component), so this confound doesn't arise
|
|
|
|
|
for it the same way, but it's worth remembering if an invariant+Γ variant
|
|
|
|
|
is ever considered.</p>
|
|
|
|
|
<p><strong>Gu & Zhang (1997), read in full — rejected for now.</strong> Their procedure:
|
|
|
|
|
(1) a tree with least-squares branch lengths must already exist; (2)
|
|
|
|
|
ancestral states are reconstructed at every internal node (they use Zhang &
|
|
|
|
|
Nei 1997's likelihood method, but note plain parsimony would work, just
|
|
|
|
|
less accurately); (3) per site, branches are split into "changed"/"unchanged"
|
|
|
|
|
given the ancestral reconstruction, and the <em>expected</em> substitution count
|
|
|
|
|
<code>k̂</code> (corrected for multiple hits) is obtained by solving their eq. (7)
|
|
|
|
|
(or eq. 12 for the generalized model) numerically — a root-find per site,
|
|
|
|
|
not a closed form; (4) the site-level <code>k̂</code> values (real-valued, not
|
|
|
|
|
integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to
|
|
|
|
|
get α — again no closed form, numerical optimization. This is a real
|
|
|
|
|
per-site pipeline requiring a tree <em>before</em> distances/α can be computed —
|
|
|
|
|
the reverse of obikmer's current dependency order (<code>snp_distance</code> computes
|
|
|
|
|
distances, which <code>--nj</code>/<code>--upgma</code> only turn into a tree <em>afterward</em>). Their
|
|
|
|
|
own answer to this chicken-and-egg problem is an iterative
|
|
|
|
|
distance→tree→α→distance loop, which they themselves flag as expensive.
|
|
|
|
|
Adopting this would mean a new ancestral-reconstruction subsystem (even a
|
|
|
|
|
parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
|
|
|
|
|
ML fit — a much bigger addition than <code>PartitionDispersion</code>, not something
|
|
|
|
|
to build without a concrete need beyond "closer to Gu&Zhang's own
|
|
|
|
|
simulated accuracy than our moment estimator."</p>
|
|
|
|
|
<p><strong>Guindon & Gascuel (2002), read in full — plausible future addition, not implemented.</strong>
|
|
|
|
|
Their <strong>Q criterion</strong>: for a candidate α, build a tree (they use BIONJ) from
|
|
|
|
|
the α-corrected distance matrix; for every internal branch, group taxa into
|
|
|
|
|
the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
|
|
|
|
|
distances <span class="arithmatex">\(\bar\delta_{AB}\)</span>, <span class="arithmatex">\(\bar\delta_{CD}\)</span>, etc., and let <code>S ≤ M ≤ L</code>
|
|
|
|
|
be the three pairings' sums (<code>{AB,CD}</code>, <code>{AC,BD}</code>, <code>{AD,BC}</code>); the branch's
|
|
|
|
|
reliability score is <code>Q_branch = L − M</code> (zero when the four points are
|
|
|
|
|
perfectly tree-additive). The whole-tree <code>Q</code> is the mean over internal
|
|
|
|
|
branches (negative/zero branches excluded). <code>α* = argmin_α Q(α)</code> over a
|
|
|
|
|
grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
|
|
|
|
|
sensitivity is highest). Complexity: <code>O(n²l)</code> once for the base
|
|
|
|
|
frequencies, then <code>O(n²r)</code> for the <code>r</code> candidate distance matrices, then
|
|
|
|
|
<code>O(n³r)</code> worst case for building <code>r</code> trees and evaluating <code>Q</code> on each
|
|
|
|
|
(same order as the tree-building step itself, so "free" relative to it).</p>
|
|
|
|
|
<p>Central finding, <strong>the opposite direction from an initial misreading in
|
|
|
|
|
this discussion</strong>: <code>α_opt</code> (Guindon-Gascuel's topology-optimal value) is
|
|
|
|
|
<strong>always ≥ the true α</strong>, not the reverse — underestimating rate
|
|
|
|
|
heterogeneity (i.e., picking a larger α than reality) reduces the variance
|
|
|
|
|
of distance estimates and thereby improves NJ/BIONJ topological accuracy,
|
|
|
|
|
especially when the molecular clock roughly holds (where <code>α_opt → ∞</code>, i.e.
|
|
|
|
|
no correction at all is topologically best). This is an <em>empirical
|
|
|
|
|
regularity</em> observed across their tested conditions (η ∈ {0.5, 2.0}, true
|
|
|
|
|
α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
|
|
|
|
|
guarantee would be another unverified-claim mistake of the kind this whole
|
|
|
|
|
discussion has been correcting.</p>
|
|
|
|
|
<p><strong>Proposed integration (not yet implemented)</strong>: since <code>α_true ≤ α_opt</code>
|
|
|
|
|
empirically, <code>PartitionDispersion::estimate_alpha()</code>'s output could seed
|
|
|
|
|
the <em>lower bound</em> of Guindon-Gascuel's grid (search <code>[α̂, ∞)</code> instead of
|
|
|
|
|
<code>[0.1, 5000]</code>), narrowing the grid substantially and reusing the existing
|
|
|
|
|
estimator rather than needing an independent "true α" source — this also
|
|
|
|
|
sidesteps Gu & Zhang's tree-first dependency problem entirely, since
|
|
|
|
|
Guindon-Gascuel's own grid already builds a tree per candidate. Still
|
|
|
|
|
requires: (a) a <strong>BIONJ</strong> implementation (see below), (b) the Q-criterion
|
|
|
|
|
subtree-grouping logic, (c) a grid-search driver. Not started.</p>
|
|
|
|
|
<p><strong>BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.</strong>
|
|
|
|
|
Searched crates.io directly (API query for "bionj": 0 results) and read the
|
|
|
|
|
docs.rs pages of the most plausible candidates: <code>phylotree</code> (builds/reads/
|
|
|
|
|
manipulates trees but has no distance-matrix reconstruction — no NJ, no
|
|
|
|
|
BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
|
|
|
|
|
after checking the actual docs), <code>speedytree</code> (NJ only — Canonical/RapidNJ/
|
|
|
|
|
Hybrid variants, explicitly no BIONJ), <code>nj</code> (plain NJ only), <code>phylo</code> (no
|
|
|
|
|
distance-based reconstruction at all). <code>obikphylo::neighbor_joining</code>
|
|
|
|
|
(<code>tree.rs</code>) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
|
|
|
|
|
how branch lengths are weighted during agglomeration (accounts for
|
|
|
|
|
distance-estimate variance, not just the sum), a modest delta over an
|
|
|
|
|
existing NJ implementation rather than a rewrite, but would need to be
|
|
|
|
|
written in-house if the Guindon-Gascuel integration above is ever pursued
|
|
|
|
|
(their own paper notes other tree-building methods gave similar results in
|
|
|
|
|
their simulations, so plain NJ may be an acceptable substitute if BIONJ
|
|
|
|
|
proper is judged not worth the effort).</p>
|
|
|
|
|
<h3 id="sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11">Sharing one sample across algorithms (<code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/<code>--iqtree</code> + <code>snp-*</code> <code>--distance</code>) — implemented 2026-09-11</h3>
|
|
|
|
|
<p><strong>Correctness bug, not just a performance one.</strong> <code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/
|
|
|
|
|
<code>--iqtree</code> (via <code>SiblingExt::sankoff_bundle</code>) and a <code>snp-*</code> <code>--distance</code>
|
|
|
|
|
(via <code>SiblingExt::snp_distance</code>) always consume the <em>exact same</em> selection
|
|
|
|
|
parameters when requested together in one <code>obikmer phylo</code> invocation
|
|
|
|
|
(<code>n</code>/<code>free_loss</code>/<code>no_ambiguity</code>/excluded-set/<code>entropy_bias</code> — the CLI has
|
|
|
|
|
no way to give them different values in one run, see <code>cmd/phylo/args.rs</code>).
|
|
|
|
|
Before this fix, each independently called <code>sample_index</code>, and
|
|
|
|
|
<code>sample_layer</code>'s <code>rand::rng()</code> (<code>subsample.rs:235</code>) is a <strong>thread-local
|
|
|
|
|
generator that advances across calls, not reseeded each time</strong> — so the
|
|
|
|
|
second call silently drew a <em>different</em> random sample of sites than the
|
|
|
|
|
first, even with identical parameters. Verified empirically (throwaway
|
|
|
|
|
8-genome index): running <code>snp-jc</code> alone vs. combined with <code>--sankoff</code>
|
|
|
|
|
produced visibly different distance matrices from the same index/params.
|
|
|
|
|
This defeated the actual point of combining these flags — running several
|
|
|
|
|
algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a <code>snp-*</code> distance) on
|
|
|
|
|
one <em>identical</em> site selection for direct comparison.</p>
|
|
|
|
|
<p><strong>Fix</strong>: <code>SankoffBundle</code> (<code>sankoff.rs</code>) now retains its <code>PairwiseTally</code>/
|
|
|
|
|
<code>PartitionDispersion</code> internally (previously consumed into <code>raw</code>/
|
|
|
|
|
<code>base_pair_tally</code>/<code>cardinality_tally</code> and dropped) and exposes
|
|
|
|
|
<code>SankoffBundle::snp_distance(kind, gamma_shape)</code>, which computes the
|
|
|
|
|
matrix from that <em>same</em> already-sampled tally — no second <code>sample_index</code>
|
|
|
|
|
call. The shared post-sampling logic (the <code>--gamma-shape</code> support check,
|
|
|
|
|
<code>alpha</code> resolution, final matrix build — previously all inline in the
|
|
|
|
|
<code>snp_distance()</code> free function) was factored into
|
|
|
|
|
<code>pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)</code> in
|
|
|
|
|
<code>snp_distance.rs</code>, called by both the standalone <code>snp_distance()</code> (after
|
|
|
|
|
its own fresh sampling) and <code>SankoffBundle::snp_distance</code> (reusing the
|
|
|
|
|
bundle's). <code>cmd/phylo/mod.rs</code> keeps the <code>Option<SankoffBundle></code> alive past
|
|
|
|
|
the Sankoff-family <code>if</code> block and, when a <code>snp-*</code> <code>--distance</code> is also
|
|
|
|
|
requested, calls <code>bundle.snp_distance(...)</code> instead of
|
|
|
|
|
<code>cache.snp_distance(...)</code> whenever a bundle was built — since both branches
|
|
|
|
|
are driven by the same <code>args.*</code> fields, the parameters trivially always
|
|
|
|
|
match when both fire; no runtime "do the params match" check needed.
|
|
|
|
|
Verified end-to-end: the reused path logs <code>(reusing the Sankoff bundle's
|
|
|
|
|
sample)</code> and the <code>snp_distance</code> stage timer reads <code>0ms</code> (<code>formula(kind)</code>
|
|
|
|
|
is <code>O(n²)</code> post-processing, no I/O), vs. ~150-230ms for a fresh sample on
|
|
|
|
|
the same tiny test index.</p>
|
|
|
|
|
<h4 id="explicit-session-implemented-2026-09-12">Explicit <code>--session</code> (implemented 2026-09-12)</h4>
|
|
|
|
|
<p>The in-process fix above only covered reuse <em>within one command</em>.
|
|
|
|
|
<code>--session DIR</code> (<code>cmd/phylo/args.rs</code>) names a directory (outside the
|
|
|
|
|
index) that persists the <code>snp-*</code> <code>--distance</code> sample — the
|
|
|
|
|
<code>PairwiseTally</code>/<code>PartitionDispersion</code> pair <code>sample_index</code> would otherwise
|
|
|
|
|
rebuild from scratch — across <em>separate</em> <code>obikmer phylo</code> invocations.</p>
|
|
|
|
|
<p><strong>Crate placement, as planned</strong>: a new <strong><code>obiksession</code></strong> crate
|
|
|
|
|
(<code>src/obiksession/</code>), domain-agnostic — no <code>PairwiseTally</code>/"site" concept
|
|
|
|
|
anywhere in it. <code>Session::open(dir, params: &[u8], force: bool)</code> handles
|
|
|
|
|
directory lifecycle and the params-conflict check against an opaque byte
|
|
|
|
|
blob; <code>Session::store(name, bytes)</code>/<code>Session::restore(name) -> Option<Mmap></code>
|
|
|
|
|
handle a checksummed artifact cache. <code>obikphylo</code>
|
|
|
|
|
(<code>siblings/algorithms/session_cache.rs</code>, shared by <code>snp_distance.rs</code> and
|
|
|
|
|
<code>sankoff.rs</code> — see below) is the consumer: it serializes its own
|
|
|
|
|
<code>PairwiseTally</code>/<code>PartitionDispersion</code>/<code>SnpAlignment</code> to bytes (via <code>rkyv</code>)
|
|
|
|
|
and hands them to <code>obiksession</code>, which never sees their type.</p>
|
|
|
|
|
<p><strong>Serialization</strong>: <code>rkyv</code> 0.8.18, added as planned. One deviation from the
|
|
|
|
|
original zero-copy pitch, scoped down deliberately: artifacts are restored
|
|
|
|
|
via <code>rkyv::from_bytes</code> (full owned deserialize) rather than <code>rkyv::access</code>
|
|
|
|
|
(zero-copy over the <code>mmap</code>). True zero-copy would require every
|
|
|
|
|
<code>PairwiseTally</code> query method (<code>categories</code>, <code>base_freq</code>, <code>pair</code>, ...) to
|
|
|
|
|
work generically over <code>Archived<PairwiseTally></code> as well as the owned type
|
|
|
|
|
— a separate, larger change not needed to get the actual win (skipping
|
|
|
|
|
<code>sample_index</code>'s expensive re-scan of the sibling annex; deserializing an
|
|
|
|
|
already-in-memory-sized count array is comparatively cheap CPU, not I/O).
|
|
|
|
|
<code>#[derive(Archive, Serialize, Deserialize)]</code> was added to <code>PairStats</code>,
|
|
|
|
|
<code>PairwiseTally</code>, and <code>PartitionDispersion</code> (<code>pairwise.rs</code>) for this.</p>
|
|
|
|
|
<p><strong>Startup behavior, as designed</strong>: <code>Session::open</code> creates <code>DIR</code> if
|
|
|
|
|
missing (fresh, params recorded); if <code>DIR</code> already holds different saved
|
|
|
|
|
params, <code>cmd/phylo</code> reports a hard error and exits rather than silently
|
|
|
|
|
preferring one side, unless <code>--session-force</code> is given (which discards the
|
|
|
|
|
directory's cached artifacts and starts over under the new params) —
|
|
|
|
|
implements exactly the "explicit escape hatch, not silent override"
|
|
|
|
|
decision from the design discussion. <code>obiksession</code> itself doesn't decode
|
|
|
|
|
either side's bytes to produce a field-by-field diff (true to "opaque
|
|
|
|
|
blob," left to whoever needs it); <code>cmd/phylo</code>'s conflict message is
|
|
|
|
|
currently a single generic sentence listing which flags could be the
|
|
|
|
|
cause, not a computed diff — an honest v1 simplification, not a limitation
|
|
|
|
|
of <code>obiksession</code> itself.</p>
|
|
|
|
|
<p><strong>Per-artifact caching, as designed</strong>: only <code>PairwiseTally</code>/
|
|
|
|
|
<code>PartitionDispersion</code> are cached (<code>snp_distance.rs</code>'s <code>restore_tally</code>/
|
|
|
|
|
<code>store_tally</code>) — <code>ratio_ceiling</code>/<code>gamma_shape</code> remain pure post-processing
|
|
|
|
|
over whatever tally is in hand, cached or fresh, exactly as planned.</p>
|
|
|
|
|
<p><strong>Locking, reconsidered and kept as the whole-session <code>DirLock</code>.</strong> First
|
|
|
|
|
written up as a corner cut ("per-entry locking was sketched, not
|
|
|
|
|
implemented"). On actually working through the concrete race, whole-session
|
|
|
|
|
locking turned out to be the <em>correct</em> choice, not a simplification:
|
|
|
|
|
<code>--session-force</code> resets a session by deleting its whole <code>artifacts/</code>
|
|
|
|
|
directory. If the directory lock were only held briefly (during
|
|
|
|
|
<code>Session::open</code>, then released so a long-running <code>Session</code> doesn't block
|
|
|
|
|
other processes from touching <em>different</em> artifacts), a second process
|
|
|
|
|
calling <code>open(..., force: true)</code> could acquire the lock, see mismatched
|
|
|
|
|
params, and wipe <code>artifacts/</code> <strong>while a first process's already-open
|
|
|
|
|
<code>Session</code> is mid-checkpoint</strong> — genuine corruption risk, not just wasted
|
|
|
|
|
work. Holding the lock for the <code>Session</code>'s entire lifetime makes that
|
|
|
|
|
sequence impossible: a <code>force</code> reset must wait for every currently-open
|
|
|
|
|
<code>Session</code> on that directory to be dropped first. The concurrency this
|
|
|
|
|
gives up (two <em>different</em> processes reading/writing <em>different</em> artifacts
|
|
|
|
|
of the same session at once) isn't exercised by anything today — only one
|
|
|
|
|
code path (<code>snp_distance</code>/<code>sankoff_bundle</code>, sequentially per invocation)
|
|
|
|
|
uses sessions at all.</p>
|
|
|
|
|
<p><strong>Per-layer chunked/resumable dumps — implemented.</strong> <code>subsample::sample_index</code>
|
|
|
|
|
gained a <code>skip_layers: usize</code> parameter (skip the first N (partition,
|
|
|
|
|
layer) pairs, in its own deterministic iteration order, entirely — no
|
|
|
|
|
I/O, no <code>on_layer</code> call) and its <code>on_layer</code> callback gained a <code>raw_index</code>
|
|
|
|
|
argument (that pair's 0-based position in the same order), so a caller can
|
|
|
|
|
tell exactly how far a run has gotten. <code>snp_distance</code>/<code>sankoff_bundle</code>
|
|
|
|
|
checkpoint their tally/dispersion (and, for <code>sankoff_bundle</code>, the
|
|
|
|
|
in-progress <code>SnpAlignment</code>) to the session every
|
|
|
|
|
<code>session_cache::CHECKPOINT_INTERVAL_LAYERS</code> (8) layers or
|
|
|
|
|
<code>CHECKPOINT_INTERVAL</code> (30s) of wall time, whichever comes first — bounding
|
|
|
|
|
how much completed scanning a crash can lose, without paying a full
|
|
|
|
|
tally-rewrite on every one of what can be thousands of layers. A <code>progress</code>
|
|
|
|
|
artifact records how many layers are reflected in the checkpoint, with a
|
|
|
|
|
<code>PROGRESS_COMPLETE</code> (<code>u64::MAX</code>) sentinel once every layer has been
|
|
|
|
|
processed (distinct from any real layer count, so "finished" is never
|
|
|
|
|
confused with "stopped after N layers where N happens to equal the total").</p>
|
|
|
|
|
<p>Verified with a real kill: built a 10-genome/64-partition index, ran
|
|
|
|
|
<code>--distance snp-jc --session DIR</code> under <code>timeout 0.4</code>, confirmed a
|
|
|
|
|
checkpoint at <code>progress=8</code> (exactly <code>CHECKPOINT_INTERVAL_LAYERS</code>) was on
|
|
|
|
|
disk, then re-ran the same command and confirmed it resumed from layer 8
|
|
|
|
|
and completed successfully — 0-effort verification that the mechanism
|
|
|
|
|
itself is real, not just plausible-sounding.</p>
|
|
|
|
|
<p><strong>Important correction from that same test — resume is <em>not</em> bit-reproducible,
|
|
|
|
|
and shouldn't be.</strong> The output matrix from the resumed run differed from
|
|
|
|
|
an uninterrupted control run against the same index/params. Root cause:
|
|
|
|
|
<code>sample_layer</code>'s random draws come from <code>rand::rng()</code>, a per-process,
|
|
|
|
|
OS-seeded generator with no continuity across a process boundary — the
|
|
|
|
|
layers processed before a kill and the layers processed after a resume
|
|
|
|
|
come from two independent random streams. An earlier revision of this doc
|
|
|
|
|
(and of the code comments) claimed resume "reproduces the exact same
|
|
|
|
|
sample an uninterrupted run would have" — that's wrong, and was corrected
|
|
|
|
|
after the user pointed out the actual reasoning error: true randomness has
|
|
|
|
|
no notion of continuity in the first place, so there was never a
|
|
|
|
|
correctness requirement for a resumed run to match an uninterrupted one
|
|
|
|
|
bit-for-bit. The sample produced after a resume is exactly as legitimate a
|
|
|
|
|
random draw as one from an uninterrupted run — just a <em>different</em> one,
|
|
|
|
|
same as running the command twice without <code>--session</code> already gives two
|
|
|
|
|
different samples today. That variance is explicitly wanted (see below),
|
|
|
|
|
not a defect to fix. Comments in <code>subsample.rs</code>/<code>session_cache.rs</code> were
|
|
|
|
|
corrected to state this plainly instead of the false reproducibility claim.</p>
|
|
|
|
|
<p><strong>Explicitly declined: deterministic/seeded sampling.</strong> Raised as a
|
|
|
|
|
possible fix for the above (seed <code>sample_layer</code>'s RNG from a per-session
|
|
|
|
|
value plus <code>(partition, layer)</code>, making a resume reproduce bit-identical
|
|
|
|
|
results, and, if applied everywhere, making <em>any</em> two runs with identical
|
|
|
|
|
parameters produce identical samples). The user explicitly declined this
|
|
|
|
|
for now: repeated runs are relied on to measure sampling variance, and a
|
|
|
|
|
<code>--seed</code> flag would need its own explicit design if wanted later — this is
|
|
|
|
|
not an accidental gap, it's a stated preference to revisit only if asked.</p>
|
|
|
|
|
<p><strong><code>SankoffBundle</code> integration — implemented.</strong> <code>sankoff_bundle</code> accepts
|
|
|
|
|
the same <code>session: Option<&Session></code> and uses the <em>same</em> artifact names
|
|
|
|
|
(<code>session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}</code>)
|
|
|
|
|
as <code>snp_distance</code>, plus its own <code>ALIGNMENT_ARTIFACT</code> (<code>sankoff.rs</code>) folded
|
|
|
|
|
in lockstep with the tally at every checkpoint (same <code>on_layer</code> call
|
|
|
|
|
produces both, so they can never drift out of sync). Consequence verified
|
|
|
|
|
directly: a tally checkpointed by <code>--sankoff --session DIR</code> is restored by
|
|
|
|
|
a <em>later, separate</em> <code>--distance snp-k2p --session DIR</code> invocation with no
|
|
|
|
|
<code>--sankoff</code> at all, and vice versa — cross-consumer reuse, not just
|
|
|
|
|
same-command reuse, exactly the synergy this design aimed for.</p>
|
|
|
|
|
<p><strong>Per-artifact caching, as designed</strong>: only the tally/dispersion (plus,
|
|
|
|
|
for <code>sankoff_bundle</code>, the alignment) are cached — <code>ratio_ceiling</code>/
|
|
|
|
|
<code>gamma_shape</code> remain pure post-processing over whatever tally is in hand,
|
|
|
|
|
cached or fresh, exactly as planned.</p>
|
|
|
|
|
<p><strong>Still not done — genuine follow-up, not a justified tradeoff:</strong>
|
|
|
|
|
<strong>exhaustive-sample caching for <code>--subsample</code>-varying exploration.</strong> The
|
|
|
|
|
design discussion's suggestion — cache the exhaustive (<code>n = None</code>) sample
|
|
|
|
|
once and subsample <em>from</em> it on each run instead of one session per exact
|
|
|
|
|
<code>n</code> — wasn't implemented; a session today is still scoped to one exact
|
|
|
|
|
parameter tuple, <code>--subsample</code> included.</p>
|
|
|
|
|
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
|
|
|
|
|
<p><strong>Implemented.</strong> The primary distance-matrix output
|
|
|
|
|
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
|
|
|
|
|