Refactor indexing logic using hash-based minimizer selection.

Updates indexing architecture to use hashing for minimizer distribution and introduces a new hash-based selection mechanism. Additionally, updates documentation references and implements a new script for converting MkDocs into a flat wiki structure.
This commit is contained in:
Eric Coissac committed 2026-10-13 08:34:51 +02:00
1 parent 755cf85334
commit 9da0876ee6
5 files changed
+686 -23

No files matched your search

+1
View File
@@ -44,3 +44,4 @@ biblio
*.parstree *.parstree
*.ckp.gz *.ckp.gz
*.model *.model
scripts/__pycache__/mk_wiki.cpython-314.pyc
+86
View File
@@ -259,3 +259,89 @@
url = {https://doi.org/10.1186/s12864-015-1647-5}, url = {https://doi.org/10.1186/s12864-015-1647-5},
volume = 16, volume = 16,
year = 2015} year = 2015}
@article{Roberts2004-rz,
title = "{Reducing storage requirements for biological sequence
comparison}",
author = "Roberts, Michael and Hayes, Wayne and Hunt, Brian R and Mount,
Stephen M and Yorke, James A",
journal = "Bioinformatics (Oxford, England)",
publisher = "Oxford University Press (OUP)",
volume = 20,
number = 18,
pages = "3363--3369",
abstract = "MOTIVATION: Comparison of nucleic acid and protein sequences is a
fundamental tool of modern bioinformatics. A dominant method of
such string matching is the 'seed-and-extend' approach, in which
occurrences of short subsequences called 'seeds' are used to
search for potentially longer matches in a large database of
sequences. Each such potential match is then checked to see if it
extends beyond the seed. To be effective, the seed-and-extend
approach needs to catalogue seeds from virtually every substring
in the database of search strings. Projects such as mammalian
genome assemblies and large-scale protein matching, however, have
such large sequence databases that the resulting list of seeds
cannot be stored in RAM on a single computer. This significantly
slows the matching process. RESULTS: We present a simple and
elegant method in which only a small fraction of seeds, called
'minimizers', needs to be stored. Using minimizers can speed up
string-matching computations by a large factor while missing only
a small fraction of the matches found using all seeds.",
month = dec,
year = 2004,
url = "https://dx.doi.org/10.1093/bioinformatics/bth408",
doi = "10.1093/bioinformatics/bth408",
pmid = 15256412,
issn = "1367-4803,1367-4811",
language = "en"
}
@INPROCEEDINGS{Steele2014-cz,
title = "{Fast splittable pseudorandom number generators}",
author = "Steele, Jr, Guy L and Lea, Doug and Flood, Christine H",
booktitle = "{Proceedings of the 2014 ACM International Conference on Object
Oriented Programming Systems Languages \& Applications}",
publisher = "ACM",
address = "New York, NY, USA",
abstract = "We describe a new algorithm SplitMix for an object-oriented and
splittable pseudorandom number generator (PRNG) that is quite
fast: 9 64-bit arithmetic/logical operations per 64 bits
generated. A conventional linear PRNG object provides a generate
method that returns one pseudorandom value and updates the state
of the PRNG, but a splittable PRNG object also has a second
operation, split, that replaces the original PRNG object with two
(seemingly) independent PRNG objects, by creating and returning a
new such object and updating the state of the original object.
Splittable PRNG objects make it easy to organize the use of
pseudorandom numbers in multithreaded programs structured using
fork-join parallelism. No locking or synchronization is required
(other than the usual memory fence immediately after object
creation). Because the generate method has no loops or
conditionals, it is suitable for SIMD or GPU implementation. We
derive SplitMix from the DotMix algorithm of Leiserson, Schardl,
and Sukha by making a series of program transformations and
engineering improvements. The end result is an object-oriented
version of the purely functional API used in the Haskell library
for over a decade, but SplitMix is faster and produces
pseudorandom sequences of higher quality; it is also far superior
in quality and speed to java.util.Random, and has been included
in Java JDK8 as the class java.util.SplittableRandom. We have
tested the pseudorandom sequences produced by SplitMix using two
standard statistical test suites (DieHarder and TestU01) and they
appear to be adequate for ``everyday'' use, such as in Monte
Carlo algorithms and randomized data structures where speed is
important.",
month = oct,
year = 2014,
url = "http://dx.doi.org/10.1145/2660193.2660195",
doi = "10.1145/2660193.2660195",
isbn = 9781450325851
}
@online{Stafford2011,
author = {Stafford, David},
title = {Better Bit Mixing - Improving on MurmurHash3's 64-bit Finalizer},
date = {2011-09-28},
url = {https://zimbry.blogspot.com/2011/09/better-bit-mixing-improving-on.html},
urldate = {2026-09-12}
}
+2 -8
View File
@@ -10,18 +10,12 @@ The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_sel
canonical minimizer → hash(minimizer) → p-bit value → partition index canonical minimizer → hash(minimizer) → p-bit value → partition index
``` ```
The routing value is recomputed whenever it is needed (during construction and again at query time) rather than stored — it is not part of the on-disk super-kmer representation.
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition. Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
## Why hashing is necessary
A canonical minimizer is an m-mer ($m \in \{9, 11, 13, 15\}$), and its distribution over all possible m-mer values is not uniform — as the lexicographic minimum of a window, small values are systematically over-represented [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf]. Routing directly on the raw minimizer value would therefore produce badly unbalanced partitions.
Hashing the minimizer before routing redistributes this skewed distribution uniformly across partitions. This works reliably because the number of partition-index bits $p$ is chosen well below the number of bits available in the minimizer ($2m$): even with strong bias in the minimizer distribution, the hash has enough entropy margin to absorb it, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
## Parameter guidance ## Parameter guidance
Even though $H$ already makes minimizer values well-distributed (see [Minimizer selection](minimizer_selection.md)), choosing $p$ well below the number of bits available in the minimizer ($2m$) leaves a comfortable entropy margin, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
| Minimizer size $m$ | Minimizer bits ($2m$) | Typical partition-index bits $p$ | Partitions | | Minimizer size $m$ | Minimizer bits ($2m$) | Typical partition-index bits $p$ | Partitions |
|----|-----------|-----------|------------| |----|-----------|-----------|------------|
| 9 | 18 | 6–8 | 64–256 | | 9 | 18 | 6–8 | 64–256 |
+69 -15
View File
@@ -2,27 +2,58 @@
## Definition ## Definition
A **minimizer** of a kmer window is the m-mer ($m < k$) that is smallest, among all $k - m + 1$ overlapping m-mers in the window, under a chosen ordering. The minimizer is always taken in canonical form (lexicographic minimum of forward and reverse complement) so that selection is strand-independent. The **minimizer** of a k-mer is the canonical form of the m-mer that is smallest according to a chosen ordering among the $k-m+1$ overlapping m-mers contained in the k-mer, with $m<k$ [@Roberts2004-rz].
The minimizer partitions a sequence into super-kmers: maximal runs of overlapping kmers that share the same minimizer (see [Kmers and super-kmers](kmers_and_superkmers.md)). For an m-mer $s$, its **canonical form** is defined as
## Hash-based ("random") minimizer $$
s^c = \min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right),
$$
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions. where $\operatorname{RC}(s)$ denotes the reverse complement of $s$.
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijection with good avalanche properties, every distinct m-mer in a window has an equal chance of holding the minimum hash value, independent of its nucleotide composition. In **OBIkmer**, canonical m-mers are ordered according to a deterministic hash function $H$ applied to their canonical form:
$$
s_1^c <_h s_2^c
\quad\Longleftrightarrow\quad
H(s_1^c) < H(s_2^c).
$$
Thus, for a k-mer $K$, let
$$
M(K)=(s_1,\ldots,s_{k-m+1})
$$
be its sequence of overlapping m-mers. The minimizer is
$$
\operatorname{minimizer}(K) = s_j^c,
\qquad
j=\underset{i}{\operatorname{argmin}}\;H(s_i^c).
$$
In other words, the hash function defines the ordering of canonical m-mers, while the minimizer itself is the canonical form of the m-mer selected by that ordering.
Applying the minimizer to consecutive k-mers partitions a sequence into super-kmers, defined as maximal runs of consecutive k-mers sharing the same minimizer (see [Kmers and super-kmers](kmers_and_superkmers.md)).
## Hash-based minimizer ordering
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf].
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijective mixing function with good avalanche properties [@Steele2014-cz], its output behaves approximately as a composition-independent pseudorandom ordering of distinct m-mers. Under the usual random-hash assumption, each distinct m-mer in a window has approximately the same probability of being the minimum, independently of nucleotide composition.
The canonical form used as input to $H$ is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the *distribution of hash values themselves* toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down. The canonical form used as input to $H$ is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the *distribution of hash values themselves* toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down.
### Hash function ### Hash function
The hash function is a 64-bit mixing function (splitmix64-style finalizer) applied to the m-mer XORed with a fixed non-zero seed: The hash function \(H\) applies Stafford's 64-bit mixing function, variant 13 [@Stafford2011], to the m-mer encoding XORed with a seed $\sigma$:
$$H(x) = \text{mix64}(x \oplus s), \quad s = \lfloor 2^{64}/\varphi \rfloor = \texttt{0x9e3779b97f4a7c15}$$
``` ```text
H(x): H(x):
x ← x ⊕ 0x9e3779b97f4a7c15 x ← x ⊕ σ
x ← x ⊕ (x >> 30) x ← x ⊕ (x >> 30)
x ← x × 0xbf58476d1ce4e5b9 x ← x × 0xbf58476d1ce4e5b9
x ← x ⊕ (x >> 27) x ← x ⊕ (x >> 27)
@@ -30,13 +61,36 @@ H(x):
return x ⊕ (x >> 31) return x ⊕ (x >> 31)
``` ```
The XOR seed avoids the finalizer's fixed point at 0 ($\text{mix64}(0) = 0$), which would otherwise make an all-A m-mer (canonical value 0) win every window comparison. The hash uses a fixed non-zero seed
## Partition routing is independent of minimizer selection $$ \sigma=\texttt{0x9e3779b97f4a7c15}. $$
The hash used to select a minimizer within a window (the minimum of several hash values) and the hash used to route a super-kmer to a storage partition are computed separately: The choice of $\sigma$ is not arbitrary. Low-complexity m-mers (homopolymers and short tandem repeats) are disproportionately abundant in real genomes. If one of them happened to be the global argmin of $H$, it could therefore occur as the minimizer in far more windows than expected from the composition-uniform behavior described above. This is not a bias of the mixing function itself, but a consequence of the repeated occurrence of the same input in real sequence data.
- **Selection** uses $H$ applied to every candidate m-mer in the window, keeping the minimum. In particular, with $\sigma=0$, the all-A m-mer is encoded as $0$, and
- **Partition routing** recomputes $H$ on the single selected minimizer only, once its position is fixed. This is a hash of one specific value, not the minimum of several, so it is uniformly distributed and safe to use directly for routing.
See [Partitioning and indexing architecture](indexing_architecture.md) for how the routing value is turned into a partition index. $$ \operatorname{Mix13}(0)=0, $$
This makes it the global argmin of the hash function. The non-zero seed eliminates the particular pathological correspondence between the all-A m-mer and the zero output of the mixer.
Exhaustive enumeration of all $4^m$ m-mers confirms that, with $\sigma=\texttt{0x9e3779b97f4a7c15}$, the global argmin is neither a homopolymer nor a periodic repeat for any tested \(m\):
| $m$ | argmin (canonical) | decoded sequence | minimal period |
|---|---|---|---|
| 3 | 16 | `CAA` | 3 |
| 5 | 78 | `ACATG` | 5 |
| 7 | 5512 | `CCCGAGA` | 7 |
| 9 | 108760 | `CGGGATCGA` | 9 |
| 11 | 179014 | `AAGGTGTCACG` | 11 |
| 13 | 33759044 | `GAAATACTTCACA` | 13 |
| 15 | 29869313 | `AACTACTTACCAAAC` | 15 |
When the minimal period of the sequence is $m$, as observed in the table above, the sequence is primitive: it cannot be represented as repetitions of a shorter sequence.
## Partition routing
A super-kmer's partition is the low $p$ bits of its minimizer's hash:
$$\text{partition} = H(\text{minimizer}) \bmod 2^p$$
See [Partitioning and indexing architecture](indexing_architecture.md) for more details.
+528
View File
@@ -0,0 +1,528 @@
#!/usr/bin/env python3
"""Convert the MkDocs user documentation (UserDocMD/) into a Gitea/Forgejo wiki tree.
Gitea/Forgejo wikis use a different link syntax than MkDocs and cannot render
multi-line `$$...$$` display math (a known Goldmark/KaTeX limitation: the
parser only recognizes the delimiters when the whole block sits on one
physical line). This script rewrites the three constructs that would
otherwise break, flattens the tree onto page basenames, and copies everything
else verbatim:
* Markdown links [text](path.md) -> [text](PageName)
Relative paths (including MkDocs root-relative `/path.md` links) are
resolved against the source tree and then flattened to a wiki page id;
only the target is rewritten, the label is left untouched. Gitea/Forgejo's
own `[[page|label]]` shortlink syntax is deliberately not used: it is
post-processed on rendered HTML text nodes, so it cannot match when the
label contains inline formatting (a `` `code` `` span splits the text
into separate DOM nodes around it) — confirmed empirically on the target
instance, where such labels never became links at all, and even a plain
label put target and display text in the position swapped from what the
MediaWiki-style `[[target|label]]` convention would suggest. A native
Markdown link is a real AST link node, so it has neither problem.
External links (http(s)://, mailto:) and links pointing outside the
source tree are left untouched.
* Pandoc citations [@key; @key] -> formatted, CSL-styled citation
Gitea/Forgejo's Goldmark does not support the [@key] citation syntax, and
a bare `(@key)` fallback is meaningless to a reader without the .bib file
open. Files containing citations are instead piped through `pandoc
--citeproc` with the project's own `references.bib` and `ecology-letters
.csl` — the same files `mkdocs-bibtex` uses for the real site — so the
wiki gets the identical citation style, e.g. `[@Zheng2020-ji]` becomes
`(Zheng et al. 2020)`, with a formatted bibliography (one `<div id="ref-
KEY">` per entry, real title/journal/DOI) appended to that page. This
duplicates entries across pages that cite the same source rather than
centralizing them on one wiki page — an acceptable tradeoff for a
handful of references, and far simpler than inventing a cross-page
anchor scheme. Files with no `[@key]` at all skip pandoc entirely.
* Multi-line display math $$\\n...\\n$$ -> single-line $$...$$
Collapsed onto one line (interior newlines replaced by spaces; `\\\\`
row separators are preserved) so Gitea/Forgejo's renderer recognizes it.
Inline `$...$` and already single-line `$$...$$` need no change and pass
through as-is.
None of these rewrites touch fenced code blocks (```` ``` ```` or `~~~`,
including openings longer than three characters, e.g. ```` ```` ```` wrapping
a literal triple-backtick example).
Wiki page ids are flat (no directories), so a source file living in a
sub-directory gets its directory path folded into the page id: a file at
`theory/encoding.md` becomes the page `theory-encoding`. Files at the root keep
their bare stem, except `index.md` which maps to `Home` — the page name
Gitea/Forgejo treats as the wiki's landing page. The same mapping is used to
resolve internal links, so cross-directory references stay consistent. If
folding still produces a collision the script refuses to write rather than
silently dropping a page.
The `nav:` section of the MkDocs config is reproduced as a `_Sidebar.md` so the
wiki keeps the same top-level grouping (Home / Installation / Theory / Usage /
Formats / Architecture). Nav entries are usually `{label: path}` mappings, but
a bare `path` entry (no explicit label) is also honored: the label is taken
from the target file's first `# ` heading, falling back to a prettified
filename. The sidebar is static and shown on every page, which is the only
way Gitea/Forgejo expose a structured menu; the file is hidden from the page
list because its name starts with `_`.
Usage:
mk_wiki.py OUTPUT [OPTIONS]
Options:
--source DIR Source tree (default: UserDocMD next to this script's parent)
--config FILE MkDocs user config (default: <repo>/mkdocs-user.yml)
--csl FILE CSL citation style (default: <source>/ecology-letters.csl)
--bib FILE BibTeX bibliography (default: <source>/references.bib)
--no-sidebar do not generate _Sidebar.md
--verbose print every conversion
"""
from __future__ import annotations
import argparse
import re
import subprocess
import sys
from pathlib import Path
import yaml
class ConversionError(Exception):
"""Raised when a source file cannot be converted faithfully."""
def _make_lenient_loader() -> type[yaml.SafeLoader]:
"""A SafeLoader that ignores unknown YAML tags instead of erroring.
The MkDocs config uses `!!python/name:pymdownx.superfences.fence_code_format`
on a custom-fence entry; PyYAML cannot resolve that tag without an unsafe
loader, but the script only needs the `nav:` section, which is plain YAML.
Treating unknown tags as their literal scalar value is enough.
"""
class _LenientLoader(yaml.SafeLoader):
pass
def _unknown_constructor(loader: yaml.Loader, node: yaml.Node): # type: ignore[no-untyped-def]
if isinstance(node, yaml.ScalarNode):
return loader.construct_scalar(node)
if isinstance(node, yaml.SequenceNode):
return loader.construct_sequence(node)
if isinstance(node, yaml.MappingNode):
return loader.construct_mapping(node)
return None
_LenientLoader.add_constructor(None, _unknown_constructor)
return _LenientLoader
def find_repo_root(start: Path) -> Path:
"""Walk up until we find a directory containing UserDocMD/."""
for candidate in [start.resolve(), *start.resolve().parents]:
if (candidate / "UserDocMD").is_dir():
return candidate
return start.resolve()
def is_external(target: str) -> bool:
return bool(re.match(r"^[a-zA-Z][a-zA-Z0-9+.-]*://", target)) or target.startswith("mailto:")
def split_anchor(target: str) -> tuple[str, str]:
if "#" in target:
path, anchor = target.split("#", 1)
return path, "#" + anchor
return target, ""
def wiki_page_id(src_root: Path, abs_target: Path) -> str:
"""Map a source file to its wiki page id.
Wikis are flat, so the directory path is folded into the id: a file at
`theory/encoding.md` becomes `theory-encoding`, while a root file keeps its
bare stem, except `index.md` which maps to `Home` — the page name
Gitea/Forgejo renders as the wiki's landing page.
"""
rel = abs_target.relative_to(src_root.resolve())
parts = list(rel.parts)
if len(parts) > 1:
# Drop the .md suffix from the last part, join the rest with '-'.
return "-".join([*parts[:-1], parts[-1][:-3]])
stem = parts[0][:-3]
return "Home" if stem == "index" else stem
def resolve_wiki_path(src_root: Path, src_file: Path, target: str) -> str | None:
"""Turn a relative markdown target into a wiki page id.
Returns None for targets we should leave as plain Markdown links
(external URLs, non-markdown files, missing files, or targets outside the
source tree).
"""
path, anchor = split_anchor(target)
if is_external(path) or path == "":
return None
if path.startswith("/"):
# MkDocs root-relative link: resolved against the docs root, not the
# current file's directory (Path("a") / "/b" would silently discard
# "a" and resolve "/b" from the filesystem root, which is wrong here).
abs_target = (src_root / path.lstrip("/")).resolve()
else:
abs_target = (src_file.parent / path).resolve()
try:
abs_target.relative_to(src_root.resolve())
except ValueError:
return None
if abs_target.suffix != ".md":
return None
if not abs_target.is_file():
return None
return wiki_page_id(src_root, abs_target) + anchor
LINK_RE = re.compile(r"\[([^\]]*)\]\(([^)\s]+)\)")
FENCE_RE = re.compile(r"^(`{3,}|~{3,})")
# Pandoc citation syntax, e.g. [@Zheng2020-ji; @Golan2025-xf] — just used to
# detect whether a file has any citation worth running through pandoc at all.
HAS_CITATION_RE = re.compile(r"\[@[^\]]+\]")
# pandoc's markdown writer must be told, relative to its own defaults, to:
# -smart leave dashes/quotes as written (no em-dash -> "---")
# -fenced_divs emit the bibliography as raw HTML <div> (Goldmark
# has no ``:::`` fenced-div extension)
# -simple_tables, -multiline_tables, -grid_tables, +pipe_tables
# keep GFM pipe-table syntax, the only kind Goldmark
# parses, instead of pandoc's space-aligned tables
# +fenced_code_blocks (pandoc still emits an indented block for a fence
# with no language tag regardless of this extension —
# such fences need a language hint in the source)
# -header_attributes Goldmark has no `{#id .class}` heading-attribute
# syntax; without this the "Bibliography" heading pandoc
# adds (see BIBLIOGRAPHY_TITLE below) would render with
# a literal `{#bibliography .unnumbered}` after it
PANDOC_MARKDOWN_FORMAT = (
"markdown-smart-fenced_divs-simple_tables-multiline_tables-grid_tables"
"-header_attributes+pipe_tables+fenced_code_blocks"
)
# Title pandoc gives the generated bibliography section (always emitted as a
# level-1 `# ` heading, with no option to pick the level) — demoted to `##`
# below to match the rest of this doc set's heading hierarchy.
BIBLIOGRAPHY_TITLE = "Bibliography"
class FenceState:
"""Tracks whether the current line is inside a fenced code block.
A closing fence must use the same character and be at least as long as
the opening one (CommonMark rule) — this is what lets a source file wrap
a literal ``` example inside a longer ```` fence.
"""
def __init__(self) -> None:
self.marker = ""
def toggle(self, line: str) -> bool:
"""Update state for a fence delimiter line; return True if it is one."""
m = FENCE_RE.match(line.rstrip("\n"))
if not m:
return False
run = m.group(1)
if not self.marker:
self.marker = run
elif run[0] == self.marker[0] and len(run) >= len(self.marker):
self.marker = ""
return True
@property
def active(self) -> bool:
return bool(self.marker)
def convert_links(text: str, src_root: Path, src_file: Path, verbose: bool) -> str:
out_lines: list[str] = []
fence = FenceState()
for line in text.splitlines(keepends=True):
if fence.toggle(line) or fence.active:
out_lines.append(line)
continue
def repl(m: re.Match) -> str:
label, target = m.group(1), m.group(2)
wiki = resolve_wiki_path(src_root, src_file, target)
if wiki is None:
return m.group(0) # external or non-md: leave as Markdown
if verbose:
print(f" {src_file.relative_to(src_root)}: [{label}]({target}) -> [{label}]({wiki})")
return f"[{label}]({wiki})"
out_lines.append(LINK_RE.sub(repl, line))
return "".join(out_lines)
def convert_citations(
text: str, src_root: Path, src_file: Path, csl_path: Path, bib_path: Path, verbose: bool
) -> str:
"""Render Pandoc `[@key; @key]` citations with the project's real CSL style.
Skips files with no citation entirely, so a missing `--csl`/`--bib` or a
missing `pandoc` binary is only an error for files that actually need it.
"""
if not HAS_CITATION_RE.search(text):
return text
if not csl_path.is_file():
raise ConversionError(f"{src_file.relative_to(src_root)}: CSL style not found: {csl_path}")
if not bib_path.is_file():
raise ConversionError(f"{src_file.relative_to(src_root)}: bibliography not found: {bib_path}")
try:
result = subprocess.run(
[
"pandoc",
"--citeproc",
f"--csl={csl_path}",
f"--bibliography={bib_path}",
"--wrap=none",
f"--metadata=link-citations=true",
f"--metadata=reference-section-title={BIBLIOGRAPHY_TITLE}",
"-f",
"markdown-smart",
"-t",
PANDOC_MARKDOWN_FORMAT,
],
input=text,
capture_output=True,
text=True,
)
except FileNotFoundError:
raise ConversionError(
f"{src_file.relative_to(src_root)}: 'pandoc' not found on PATH, needed to render citations"
) from None
if result.returncode != 0:
raise ConversionError(
f"{src_file.relative_to(src_root)}: pandoc failed to render citations:\n{result.stderr}"
)
# pandoc always emits the bibliography title as a level-1 heading; demote
# it to `##` to match the rest of this doc set's hierarchy.
out = result.stdout.replace(f"\n# {BIBLIOGRAPHY_TITLE}\n", f"\n## {BIBLIOGRAPHY_TITLE}\n")
if verbose:
n = len(HAS_CITATION_RE.findall(text))
print(f" {src_file.relative_to(src_root)}: rendered {n} citation group(s) via pandoc/{csl_path.name}")
return out
def convert_math(text: str, src_root: Path, src_file: Path, verbose: bool) -> str:
"""Collapse multi-line `$$...$$` display blocks onto a single line.
Gitea/Forgejo's math renderer only recognizes `$$...$$` when it sits on
one physical line; a block opened and closed on their own bare `$$` line
(the natural Pandoc/MkDocs style) renders as literal text otherwise.
Interior line breaks are replaced by spaces (insignificant in math mode);
`\\\\` row separators inside `aligned`/`cases` environments are untouched.
Already single-line `$$...$$` and inline `$...$` need no change.
"""
lines = text.split("\n")
out: list[str] = []
fence = FenceState()
i = 0
n = len(lines)
while i < n:
line = lines[i]
if fence.toggle(line) or fence.active:
out.append(line)
i += 1
continue
if line.strip() == "$$":
body: list[str] = []
i += 1
closed = False
while i < n:
if lines[i].strip() == "$$":
closed = True
i += 1
break
body.append(lines[i].strip())
i += 1
if not closed:
raise ConversionError(
f"{src_file.relative_to(src_root)}: unterminated $$ display math block"
)
collapsed = "$$" + " ".join(part for part in body if part) + "$$"
if verbose:
print(f" {src_file.relative_to(src_root)}: collapsed {len(body)}-line $$ block")
out.append(collapsed)
continue
out.append(line)
i += 1
return "\n".join(out)
def convert_file(
src_root: Path, src_file: Path, csl_path: Path, bib_path: Path, verbose: bool
) -> str:
text = src_file.read_text(encoding="utf-8")
# Citations first, before pandoc has seen any of our own wiki-specific
# rewrites: it needs to parse plain, unmodified Markdown.
text = convert_citations(text, src_root, src_file, csl_path, bib_path, verbose)
text = convert_math(text, src_root, src_file, verbose)
text = convert_links(text, src_root, src_file, verbose)
return text
def _default_label(src_root: Path, rel_path: str) -> str:
"""Derive a nav label for a bare `path.md` entry (no explicit label).
Mirrors MkDocs' own fallback: use the page's first `# ` heading if it has
one, otherwise prettify the filename.
"""
target = src_root / rel_path
if target.is_file():
for line in target.read_text(encoding="utf-8").splitlines():
if line.startswith("# "):
return line[2:].strip()
return Path(rel_path).stem.replace("_", " ").replace("-", " ").title()
def build_sidebar(src_root: Path, config_path: Path) -> str | None:
"""Render the MkDocs `nav:` section as a Gitea/Forgejo `_Sidebar.md`.
Returns None if the config is missing or has no usable `nav:` section.
Each nav entry is either a bare `path` string or a `{label: path}` pair
(or a nested list for a group); the path is mapped through `wiki_page_id`
so the sidebar links to the flattened page ids.
"""
if not config_path.is_file():
print(f"warning: config not found, skipping sidebar: {config_path}")
return None
try:
with config_path.open(encoding="utf-8") as fh:
cfg = yaml.safe_load(fh)
except yaml.YAMLError:
# The MkDocs config carries `!!python/name:...` tags that SafeLoader
# cannot resolve; fall back to a lenient loader that treats unknown
# tags as plain scalars so the `nav:` section is still readable.
try:
with config_path.open(encoding="utf-8") as fh:
cfg = yaml.load(fh, Loader=_make_lenient_loader())
except yaml.YAMLError as exc2:
print(f"warning: could not parse {config_path}: {exc2}")
return None
nav = cfg.get("nav") if isinstance(cfg, dict) else None
if not nav:
print(f"warning: no `nav:` section in {config_path}, skipping sidebar")
return None
def link_line(depth: int, rel_path: str, label: str) -> str:
page = wiki_page_id(src_root, (src_root / rel_path).resolve())
return f"{' ' * depth}- [{label}]({page})"
def render_entry(entry: object, depth: int) -> list[str]:
if isinstance(entry, str):
return [link_line(depth, entry, _default_label(src_root, entry))]
if not isinstance(entry, dict):
return []
lines: list[str] = []
for label, target in entry.items():
if isinstance(target, str):
lines.append(link_line(depth, target, label))
elif isinstance(target, list):
# Headings must never be indented: CommonMark treats a line
# indented by 4+ spaces as a code block, not an ATX heading,
# which a depth-scaled indent would trigger from depth 2 on.
lines.append(f"## {label}")
lines.extend(render_subtree(target, depth + 1))
return lines
def render_subtree(nodes: list[object], depth: int) -> list[str]:
out: list[str] = []
for node in nodes:
out.extend(render_entry(node, depth))
return out
body = render_subtree(nav, 0)
if not body:
return None
return "# Wiki sidebar\n\n" + "\n".join(body) + "\n"
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("output", help="destination wiki directory (created if missing)")
ap.add_argument("--source", default=None, help="source Markdown tree (default: <repo>/UserDocMD)")
ap.add_argument("--config", default=None,
help="MkDocs user config (default: <repo>/mkdocs-user.yml)")
ap.add_argument("--csl", default=None,
help="CSL citation style (default: <source>/ecology-letters.csl)")
ap.add_argument("--bib", default=None,
help="BibTeX bibliography (default: <source>/references.bib)")
ap.add_argument("--no-sidebar", action="store_true", help="do not generate _Sidebar.md")
ap.add_argument("--verbose", action="store_true", help="print every conversion")
args = ap.parse_args()
script_dir = Path(__file__).resolve().parent
repo_root = find_repo_root(script_dir)
src_root = Path(args.source).resolve() if args.source else (repo_root / "UserDocMD")
out_root = Path(args.output).resolve()
config_path = Path(args.config).resolve() if args.config else (repo_root / "mkdocs-user.yml")
csl_path = Path(args.csl).resolve() if args.csl else (src_root / "ecology-letters.csl")
bib_path = Path(args.bib).resolve() if args.bib else (src_root / "references.bib")
if not src_root.is_dir():
print(f"error: source tree not found: {src_root}", file=sys.stderr)
return 2
md_files = sorted(p for p in src_root.rglob("*.md") if p.is_file())
if not md_files:
print(f"error: no Markdown files under {src_root}", file=sys.stderr)
return 2
# A wiki is flat, so the destination filename is the source path folded into
# a page id. Detect collisions up front so a silent overwrite does not drop
# a page.
by_name: dict[str, list[Path]] = {}
for src in md_files:
by_name.setdefault(wiki_page_id(src_root, src), []).append(src)
collisions = {name: paths for name, paths in by_name.items() if len(paths) > 1}
if collisions:
for name, paths in sorted(collisions.items()):
print(f"error: wiki page id collision '{name}':", file=sys.stderr)
for p in paths:
print(f" {p.relative_to(src_root)}", file=sys.stderr)
return 2
out_root.mkdir(parents=True, exist_ok=True)
written = 0
for src in md_files:
dst = out_root / (wiki_page_id(src_root, src) + ".md")
try:
content = convert_file(src_root, src, csl_path, bib_path, args.verbose)
except ConversionError as exc:
print(f"error: {exc}", file=sys.stderr)
return 2
dst.write_text(content, encoding="utf-8")
written += 1
if not args.no_sidebar:
sidebar = build_sidebar(src_root, config_path)
if sidebar is not None:
(out_root / "_Sidebar.md").write_text(sidebar, encoding="utf-8")
written += 1
print(f"converted {written} Markdown file(s) from {src_root} -> {out_root}")
return 0
if __name__ == "__main__":
sys.exit(main())