obitools4/prompt_documentation_globale.md

# Objective

Fully document OBITools (version 4, written in Go) in English, using a 4‑phase incremental pipeline.

You **MUST** use the available MCP servers:

- `cclsp` – exact definitions, references, diagnostics  
- `jcodemunch` – code indexing, symbol extraction  
- `treesitter` – AST and CLI parsing  
- `context7` – external documentation  

All tool calls must follow the exact API described in the MCP server documentation. If a required tool is unavailable, you **MUST** log the error and stop execution.

### Tool call format (CRITICAL)

Tool calls **MUST** use this exact XML format — no spaces inside the angle brackets:

```
<function=tool_name>
{"param": "value"}
</function>
```

**FORBIDDEN** — these variants will cause parse errors and must NEVER be used:
- `< function=tool_name >` (spaces around the tag name)
- `< function = tool_name>` (spaces around `=`)
- `<function = tool_name>` (space before `=`)

The opening tag is `<function=tool_name>` with **zero spaces** inside `<` and `>`.

---

# Global rules

** You are not allowed to read twice the same file in a row. **

## Language

- All generated documentation **MUST** be in English.  
- If an existing documentation file is in French:  
  1. Translate it to English  
  2. Save the original as `.fr.md` **before** overwriting  
  3. Write the new English version  

---

## Execution mode (STRICT)

You are operating in **STRICT TOOL MODE**:

- If a file must be written, you **MUST** use the `Shell` tool.
- You **MUST NOT** read entire directory listings into memory.
- You **MUST** work with **one item at a time** using a simple text file as a task queue.

### Reading files before writing

- **Before writing to an existing documentation file**, you must first read it using the `Read` tool.
- **When documenting a single Go source file**, you only need to read that one file (plus up to 4-5 helper files if needed for context).
- Do NOT read the entire codebase - only what is necessary to document the current file.

---

### Rules

- Always write the **full** file (no partial updates).  
- Paths are relative to the project root; directories are created implicitly.  
- Content must be valid UTF‑8; use `\n` line endings.  
- Do **not** wrap content in backticks.  

---

## Progress tracking: task queue files

We use **line‑oriented task files** to avoid loading large lists into memory. Each phase has its own task file:

- `docs/todo/phase1.txt` – list of Go files (one per line) to document.  
- `docs/todo/phase1bis.txt` – same list, but after phase1 is done.  
- `docs/todo/phase2.txt` – list of packages.  
- `docs/todo/phase3.txt` – list of tools.

**How it works:**

1. At the start of a phase, if the task file does not exist, it is created by scanning the codebase once (Phase 0 or Phase X init).  
2. **Each run of the LLM processes only the first line of the task file.**  
3. After processing the item (success or permanent failure), the line is removed from the task file.  
   - On success, the line is deleted (no extra sentinel file needed).  
   - On transient failure (retry < 3), we keep the line but increment a retry counter stored in a separate file.  
   - On permanent failure (retry ≥ 3), we move the line to a `failed.txt` file and log the error.  
4. The LLM then exits (or continues if the task file is still non‑empty, but it must never load more than one line).

This way, the LLM’s context never holds more than a single task at a time.

### Retry mechanism

For each item (e.g., `internal/align/align.go`), we maintain a retry counter in:

- `docs/retry/phase1/internal/align/align.go.count`

If the file does not exist, retries = 0.  
Each time processing fails, we increment the counter (write the new number).  
If after increment the counter < 3, we keep the line in the task file.  
If counter reaches 3, we **remove the line from the task file**, add it to `docs/failed/phase1/internal/align/align.go.failed` (just a marker), and log the error.

---

## Documentation quality requirements (CRITICAL)

Documentation MUST NOT be superficial. For each documented element (file, function, struct, package):

###  You MUST explain:

- what it does
- why it exists (context, problem solved)
- how it is used
- assumptions and preconditions
- possible edge cases

### Forbidden patterns

- Vague phrases like “This function handles…”, “Utility for…”, “Helper function…”.
- Generic descriptions that could apply to any project.

### Required content per element type

- Functions:
  - Purpose
  - Parameter meaning
  - Return values
  - Notable behaviour (panic conditions, side effects, concurrency)
- Structs:
  - Role in the system
  - Meaning of key fields
- Files:
  - Role within the package
  - Interactions with other files

### Anti‑generic rule

If the description could apply to any project, it is INVALID. You MUST include domain‑specific context (bioinformatics, sequence processing, etc.) and concrete behaviour.

### Quality validation

Before marking an item as done (i.e., creating the .done sentinel), you MUST perform a self‑validation:

- Check that all required sections are present.
- Verify that no forbidden patterns remain.

If validation fails, increment the retry counter and keep the item pending.


---

# Directory structure

```
docs/
  todo/                          # task queues
    phase1.txt
    phase1bis.txt
    phase2.txt
    phase3.txt
  retry/                         # retry counters
    phase1/                      # mirrors file structure
      internal/align/align.go.count
    phase1bis/
    phase2/
    phase3/
  failed/                        # permanent failure markers
    phase1/
      internal/align/align.go.failed
    phase1bis/
    phase2/
    phase3/
  phase1/                        # actual documentation
    <relative_path>/<file>.go.md
  phase2/
    <package>.md
  phase3/
    <tool>.md
  error.log
```

---

# Phase 0: Initialization

1. Ensure required directories exist: `docs/todo`, `docs/retry`, `docs/failed`, `docs/phase1`, `docs/phase2`, `docs/phase3`.  
2. **If `docs/todo/phase1.txt` does not exist**:  
   - Use `find pkg -name "*.go" ! -name "*_test.go" ! -path "*/cmd/*"` to list all Go files (excluding tests and main.go).  
   - Write the list (one relative path per line, e.g., `internal/align/align.go`) to `docs/todo/phase1.txt`.  
3. Do the same for phase2 and phase3 later when those phases start.  
4. **No other state is stored.**

---

# Phase 1: File documentation

**Processing rule:**
- Read the **first line** of `docs/todo/phase1.txt` (using `head -n 1`).  
- If the file is empty, Phase 1 is complete → proceed to Phase 1bis initialization.  
- Otherwise, process that single file.

**Processing a file:**

1. Let `relpath` be the line content (e.g., `internal/align/align.go`).  
2. Check if a permanent failure marker exists at `docs/failed/phase1/${relpath}.failed`. If yes, remove the line from the task file and skip (line will be deleted). 
3. If the documentation file `docs/phase1/${relpath}.go.md` exists go directly to its validation (step 6).
4. Otherwise, generate documentation for that file (using MCP tools as before).  
5. Write the documentation to `docs/phase1/${relpath}.go.md`.  
6. Validate quality.  
7. If validation succeeds:  
   - Remove the line from the task file.  
   - Remove any retry counter file for this item.  
   - (No sentinel needed; the removal from todo indicates completion.)  
8. If validation fails:  
   - Increment retry counter:  
     - If `docs/retry/phase1/${relpath}.count` does not exist, set to 1.  
     - Else read it, add 1, write back.  
   - If new counter >= 3:  
     - Remove line from task file.  
     - Create `docs/failed/phase1/${relpath}.failed`.  
     - Log error.  
   - If new counter < 3:  
     - Keep the line in the task file (do nothing, it stays as first line for next run).  
9. **Exit** (or stop if this was a single run). The next invocation will read the first line again (same if retry, or next if removed).

**Important:**  
- Do **not** read more than one line.  
- Do **not** attempt to process multiple items in one run.  
- The LLM should finish after handling one item.

---

# Phase 1bis: Review and harmonization

When Phase 1 is complete (i.e., `docs/todo/phase1.txt` empty), we initialize `docs/todo/phase1bis.txt` with the same list of files (the ones that succeeded).  
But note: we need to know which files were successfully documented. Since we removed lines from `phase1.txt` on success, we need a record. The simplest is to reuse the same list but we can generate it by listing the existing `.go.md` files in `docs/phase1/` (since every successful file has a `.go.md`).  
Thus, Phase 1bis initialization:

- If `docs/todo/phase1bis.txt` does not exist, create it by listing all `.go.md` files under `docs/phase1/`, stripping the `docs/phase1/` prefix and the `.go.md` suffix, and writing the relative path (same format as phase1).  

Then processing is identical to Phase 1, but using `docs/todo/phase1bis.txt` and output is overwriting the same `.go.md` files (with improvements). Retry counters go in `docs/retry/phase1bis/`.

---

# Phase 2: Package documentation

When Phase 1bis is complete (`docs/todo/phase1bis.txt` empty), initialize `docs/todo/phase2.txt`:

- List all packages: unique directories under `pkg/` that contain at least one `.go` file and are not tools.  
- Write each package identifier (e.g., `align`, `internal/align`) as a line.

Processing: read first line, generate `docs/phase2/<package>.md`, validate, remove line on success, retry logic in `docs/retry/phase2/`.

---

# Phase 3: Tool documentation

When Phase 2 complete, initialize `docs/todo/phase3.txt`:

- List all directories under `cmd/` that contain a `main.go`. Write each tool name as a line.

Processing: read first line, generate `docs/phase3/<tool>.md`, validate, remove line on success, retry logic in `docs/retry/phase3/`.

---

# Finalization

When all task files are empty and no pending phases, generate `docs/README.md` by:

- Listing all package docs (files in `docs/phase2/`) and linking.  
- Listing all tool docs (files in `docs/phase3/`) and linking.  

Write using `Shell`.

---

# Execution flow summary

1. **Phase 0**: Create directories and initial `todo/phase1.txt` if missing. Exit.  
2. **Phase 1**:  
   - If `todo/phase1.txt` exists and non‑empty → process first line.  
   - Else → move to Phase 1bis initialization.  
3. **Phase 1bis**:  
   - If `todo/phase1bis.txt` does not exist → create from successful phase1 docs.  
   - If non‑empty → process first line.  
   - Else → move to Phase 2 initialization.  
4. **Phase 2**: similar.  
5. **Phase 3**: similar.  
6. **Finalization**: generate README.

The LLM should be invoked repeatedly (e.g., by a scheduler) until all phases are done. Each invocation processes exactly one item.

---

# Important reminders

- Always call `Shell` to write files; never output content in plain text.  
- Validate quality before removing a line from the task file.  
- Log all failures to `docs/error.log` in JSON lines format.  
- If any MCP tool fails, treat as failure and increment retry counter.  
- Never read more than one line from a task file in a single run.