The Indexing Pipeline

Indexing turns a repository into a queryable SQLite index. It is a pure, deterministic pipeline — walk the tree, detect each file's language, parse it once with tree-sitter, chunk along definition boundaries, extract symbols and references, then resolve the call graph. Every stage is content-addressed, so re-indexing an unchanged tree is nearly free.

The whole pipeline runs behind a single call, engine.index(). It is idempotent: running it again on an unchanged repository does no work, and on a changed one it re-parses only what moved. engine.refresh() is the same idempotent re-index under a more intention-revealing name.

from rce.code import CodeEngine

engine = CodeEngine.open_or_create("/path/to/repo")
summary = engine.index()             # walk -> parse -> chunk -> symbols -> refs -> resolve
print(summary.files_indexed, summary.files_deleted, summary.skipped)

# Later, after edits — same idempotent re-index, only changed files do work.
engine.refresh()

The stages

A file flows left to right. Walking and language detection decide what gets indexed; parsing, chunking, and symbol/reference extraction are pure per-file work that produces FileArtifacts; resolution is a single cross-file pass that wires references to the symbols they name. Only the final write touches SQLite.

1. Walk (gitignore-aware)

The Walker recurses the repo in sorted, deterministic order and filters out everything that should not be indexed. A file is skipped when it is matched by .gitignore (or the built-in deny patterns and any extra_ignore globs), is larger than max_file_size (1 MB default), looks binary (a NUL byte in the first 8 KB), or has no detectable language. The reason for every skip is counted and surfaced on the IndexSummary.skipped dict.

2. Detect language

detect_language maps a file to a canonical language id in three steps: exact filename first (so Dockerfile and Makefile resolve), then file extension (.py → python, .ts/.tsx → typescript, and so on), and finally a shebang scan of the first line for files without a recognised extension. A file whose language cannot be determined is dropped from the corpus.

3. Parse (tree-sitter)

Each file is parsed exactly once by a pooled tree-sitter parser. From that single tree RCE Code extracts symbol definitions and, on a clean parse, references. Parsing happens in parse_file, a pure function with no database handle — it reads bytes, detects, decodes, parses, and returns FileArtifacts, which is what lets parsing move to worker processes without touching the writer.

4. AST chunk (definition-aligned)

The AST chunker tiles a file so every line lands in exactly one chunk, aligned to definition boundaries. Its guarantees:

• Never splits mid-definition. Each top-level function, method, class, interface, enum, or type becomes one chunk; the gaps between definitions become BLOCK chunks windowed by max_chunk_lines (60 by default).
• Comments attached. A comment immediately preceding a definition is chained onto that definition's chunk, so a function and its doc comment stay together.
• Oversize definitions kept whole. A definition longer than max_chunk_lines is not split — it is emitted as one chunk and flagged truncated=True rather than cut mid-body.
• Line-chunk fallback. If tree-sitter reports significant parse errors (a badly-broken or unsupported file), RCE Code skips AST chunking and falls back to plain line chunking, recording parse_status = FALLBACK_LINE.

5. Symbols & references

Symbols (definitions: their name, kind, line span, and parent qualified name) are extracted from the parse tree. On a clean parse, references (the names a file uses — calls and the like) are extracted too. Both are then linked back into the index: each symbol is attached to the chunk whose line span contains it, and each reference is attached to the innermost enclosing symbol by line.

6. Resolve references

After all files for an index pass are written, a single cross-file resolver maps each reference's target name to a concrete symbol id with a confidence score. It is a name-based heuristic, not a type checker: a same-file definition wins (confidence 0.9), a project-unique name resolves (0.5), and ambiguous or unmatched names are left unresolved. The pass is idempotent and only runs when something was actually indexed or deleted.

Incremental indexing

RCE Code is content-addressed end to end. Before parsing, the walker's file bytes are hashed and compared against the content_hash already stored for that path. If they match, the file is an unchanged no-op — it is never re-read past the hash, never re-parsed, never re-chunked. This is why engine.index() on an untouched tree costs little more than a directory walk plus a hash per file.

When a file has changed, chunk rows are reused where their content hash still matches (only the line/byte offsets are refreshed), genuinely new chunks are inserted, and chunks whose content disappeared are deleted. Files that vanished from the repo entirely are removed from the index. Because chunks, symbols, and embeddings are keyed by content hash, nothing downstream — including embeddings — is recomputed unless the underlying code actually changed.

The write model: single writer, per-file transaction, WAL

Parsing is pure and parallelizable, but writing is deliberately serial. The parent process is the sole writer. Files are written in sorted-path order, and each file's artifacts — chunks, symbols, and references — are committed in one atomic transaction per file. If a write fails mid-file it rolls back cleanly, leaving the index consistent.

The database runs in WAL (write-ahead logging) mode with synchronous = NORMAL, so readers never block the single writer and an interrupted run cannot tear a half-written file into the index. The reference-resolution pass and the bookkeeping that records the last sync time and skip counts are likewise committed as their own transactions.

Where to go next

Once chunks exist, you embed them for semantic and hybrid search. See Embedders for the embedder protocol and the built-in adapters, and Vector Backends for where the cosine KNN actually runs.