<p>Low-complexity kmers (polyA, polyT, tandem repeats) are detected and excluded during phase 1. The filter computes a <strong>normalized Shannon entropy</strong> over sub-words of multiple sizes, corrected for one source of bias: the small number of observations within a single kmer relative to the number of possible sub-words.</p>
<p>For a kmer of length k and a sub-word size ws (1 ≤ ws ≤ ws_max, typically ws_max = 6), extract the <spanclass="arithmatex">\(n_{\text{words}} = k - ws + 1\)</span> overlapping sub-words by sliding a window of length ws:</p>
<p>Each sub-word is tallied under its own raw 2-bit-packed value — <strong>no canonicalization</strong>. Let <spanclass="arithmatex">\(f_j\)</span> be the count of raw word <spanclass="arithmatex">\(j\)</span> among the <spanclass="arithmatex">\(n_{\text{words}}\)</span> sub-words (<spanclass="arithmatex">\(\sum_j f_j = n_{\text{words}}\)</span>), over the <spanclass="arithmatex">\(4^{ws}\)</span> possible raw words.</p>
<p>An earlier version of this filter first folded each sub-word into a circular+reverse-complement equivalence class, then "unfolded" the observed class frequency back onto its members to correct for unequal class sizes. That machinery bought nothing it was claimed for — see <em>Why no equivalence classes</em> below — while measurably weakening detection of the very sequences the filter exists to catch, so it was removed.</p>
<h2id="maximum-entropy-correction-for-small-samples">Maximum entropy correction for small samples</h2>
<p>With only <spanclass="arithmatex">\(n_{\text{words}}\)</span> observations over <spanclass="arithmatex">\(4^{ws}\)</span> possible raw words, the achievable maximum entropy is bounded by the most uniform integer distribution over <spanclass="arithmatex">\(4^{ws}\)</span> categories.</p>
<p>Let <spanclass="arithmatex">\(c = \lfloor n_{\text{words}} / 4^{ws} \rfloor\)</span> and <spanclass="arithmatex">\(r = n_{\text{words}} \bmod 4^{ws}\)</span>. The most uniform integer distribution assigns frequency <spanclass="arithmatex">\(c+1\)</span> to <spanclass="arithmatex">\(r\)</span> categories and <spanclass="arithmatex">\(c\)</span> to the remaining <spanclass="arithmatex">\(4^{ws} - r\)</span>, with the convention <spanclass="arithmatex">\(0 \log 0 = 0\)</span>:</p>
<p>When <spanclass="arithmatex">\(n_{\text{words}} < 4^{ws}\)</span>: <spanclass="arithmatex">\(c=0\)</span>, <spanclass="arithmatex">\(r=n_{\text{words}}\)</span>, and the formula reduces to <spanclass="arithmatex">\(H_{\max} = \log(n_{\text{words}})\)</span> — a single unified expression covers both regimes. A truly random sequence achieves <spanclass="arithmatex">\(H_{\text{corr}} \approx H_{\max}\)</span>.</p>
<p>The filter computes <spanclass="arithmatex">\(\hat{H}(ws)\)</span> for each word size ws from 1 to ws_max and returns the <strong>minimum</strong>:</p>
<p>A value near 0 indicates low complexity (e.g. AAAA…); near 1 indicates high complexity. A kmer is rejected if <spanclass="arithmatex">\(\text{entropy}(kmer) < \theta\)</span>, where <spanclass="arithmatex">\(\theta\)</span> is a collection parameter (default 0.7). The minimum across word sizes ensures that any scale of repetition is detected independently: polyA is caught at ws=1, dinucleotide repeats at ws=2, etc.</p>
<h2id="why-no-equivalence-classes">Why no equivalence classes</h2>
<p>A prior design folded each sub-word into the canonical form of its circular-rotation + reverse-complement equivalence class before tallying, on the reasoning that (a) it guarantees <spanclass="arithmatex">\(\text{entropy}(K) = \text{entropy}(\text{revcomp}(K))\)</span>, and (b) collapsing phase-shifted repeats (e.g. <code>ATG</code> ≡ <code>TGA</code> ≡ <code>GAT</code>) into one class better reflects that they are "the same" low-complexity pattern.</p>
<p>Both properties already hold for the raw, unfolded entropy above, without any class machinery:</p>
<ul>
<li><strong>Reverse complement</strong>: for any K of length n, window <spanclass="arithmatex">\(j\)</span> of <spanclass="arithmatex">\(\text{revcomp}(K)\)</span> equals <spanclass="arithmatex">\(\text{revcomp}\)</span> of window <spanclass="arithmatex">\((n{-}ws{-}j)\)</span> of K. This is a bijection between the window sets under which each window maps to its own revcomp — and revcomp is itself a bijection (involution) on the space of raw ws-mers. So the multiset of raw-word frequencies for <spanclass="arithmatex">\(\text{revcomp}(K)\)</span> is exactly a relabeling of the multiset for K, and Shannon entropy — a function of the frequency multiset alone — is exactly invariant. No folding required, for any K.</li>
<li><strong>Tandem repeats</strong>: a period-p repeat sampled by a stride-1 sliding window naturally cycles through its own rotations as raw tokens (e.g. <code>ATGATGATG…</code> yields the raw words <code>ATG</code>, <code>TGA</code>, <code>GAT</code> in rotation as the window slides). The low diversity this represents (few distinct raw words out of <spanclass="arithmatex">\(4^{ws}\)</span> possible) is already visible in the raw frequency distribution — no folding needed to detect it.</li>
</ul>
<p>What the fold-then-unfold step actually did was credit each observed class with the frequency of equivalence-class members that were <strong>never observed on the read strand</strong>, inflating <spanclass="arithmatex">\(H_{\text{corr}}\)</span> for genuine repeats. Worked example: k=31, ws=3, kmer = <code>ATG</code> repeated (<spanclass="arithmatex">\(n_{\text{words}}=29\)</span>, all 29 windows fall into one class of size 6 under the old scheme — 3 rotations × forward/revcomp):</p>
<p>The gap is not a rounding artifact: per sub-word order, the folded score for this same repeat swings from 0.53 (ws=3, aligned with the period) up to <strong>1.03</strong> (ws=5, misaligned with the period) — i.e. a period-3 repeat could score <em>above</em> the theoretical maximum for a random sequence, depending on which ws happens to divide the repeat's period. The raw formula stays flat at ≈0.33–0.40 across ws=2..6 regardless of alignment, which is the robustness the "minimum across ws" design was meant to provide in the first place.</p>
<p><spanclass="arithmatex">\(H_{\text{corr}}\)</span> is a standard Shannon entropy over raw words, so the classical perplexity interpretation holds directly: <spanclass="arithmatex">\(N_{\text{eff}} = e^{H_{\text{corr}}}\)</span> is the number of equiprobable raw words that would yield the same entropy.</p>
<p>For the normalised score <spanclass="arithmatex">\(\hat{H}\)</span>, dividing by <spanclass="arithmatex">\(H_{\max}\)</span> changes the logarithm base:</p>
<p>The property is preserved: <spanclass="arithmatex">\(\hat{H}\)</span> is the logarithm (in base <spanclass="arithmatex">\(N_{\max}\)</span>) of the effective number of equi-represented raw words.</p>
<p>This has a clean interpretation: <spanclass="arithmatex">\(ws \cdot \hat{H}\)</span> is the <strong>effective word length</strong> (in bases) of a perfectly uniform distribution that would produce the same entropy. At <spanclass="arithmatex">\(\hat{H} = 1\)</span> the full space of <spanclass="arithmatex">\(4^{ws}\)</span> words is used; at <spanclass="arithmatex">\(\hat{H} = 0.5\)</span> with ws=2, only <spanclass="arithmatex">\(4^1 = 4\)</span> effective words out of 16 are occupied.</p>
<p>In our actual regime, <spanclass="arithmatex">\(n_{\text{words}}\)</span> is small and <spanclass="arithmatex">\(4^{ws}\)</span> can exceed <spanclass="arithmatex">\(n_{\text{words}}\)</span>, so <spanclass="arithmatex">\(H_{\max} < \log(4^{ws})\)</span> due to the small-sample correction. The exact effective count is <spanclass="arithmatex">\(N_{\max}^{\hat{H}}\)</span>, not <spanclass="arithmatex">\(4^{ws \cdot \hat{H}}\)</span>.</p>
<p>The entropy score is a function of the kmer sequence alone — it does not depend on the surrounding context or on the position within any genome. Two consequences:</p>
<li><strong>Orientation invariance</strong>: <spanclass="arithmatex">\(\text{entropy}(K) = \text{entropy}(\text{revcomp}(K))\)</span> — see <em>Why no equivalence classes</em> above for why this holds without any explicit strand-folding step.</li>
<li><strong>Context independence</strong>: the same kmer is always rejected or always kept, regardless of which genome it occurs in, where in that genome it appears, or which strand is considered. The filter defines a fixed partition of the kmer space into low-complexity and valid kmers.</li>
<scriptid="__config"type="application/json">{"annotate":null,"base":"../..","features":[],"search":"../../assets/javascripts/workers/search.2c215733.min.js","tags":null,"translations":{"clipboard.copied":"Copied to clipboard","clipboard.copy":"Copy to clipboard","search.result.more.one":"1 more on this page","search.result.more.other":"# more on this page","search.result.none":"No matching documents","search.result.one":"1 matching document","search.result.other":"# matching documents","search.result.placeholder":"Type to start searching","search.result.term.missing":"Missing","select.version":"Select version"},"version":null}</script>
