Lexical Search

Lexical search is keyword retrieval over indexed chunks, powered by SQLite's FTS5 full-text index and ranked with BM25. Identifiers are split into sub-tokens at index time, so a query for user finds getUserById even though the two never share a literal substring boundary.

It is the fastest ranked modality and needs nothing beyond a built index — no embeddings, no network. It is also the leg that hybrid search always has available, which is why hybrid still works before you embed anything.

BM25 ranking

Chunks are mirrored into an FTS5 virtual table. At query time RCE Code runs a MATCH and orders by SQLite's bm25() function. BM25 is the standard term-frequency / inverse-document-frequency ranking function: a chunk scores higher when it contains the query terms more often, but rarer terms across the corpus count for more than common ones.

The ranking is column-weighted. Matches in a chunk's name and in its expanded search-terms column count for more than matches in raw body content, so a query that hits a function's name outranks one that merely appears somewhere in its body. SQLite returns a BM25 score where more negative means more relevant; RCE Code orders ascending and negates it, so the .score you see is positive and larger-is-better.

Identifier sub-token expansion

Code identifiers are dense: getUserById, user_id, and UserID all encode the words “user” and “id” but a naive full-text index would treat each as one opaque token. RCE Code expands every identifier into its camelCase and snake_case sub-tokens at index time and stores them, space-joined and de-duplicated, in a dedicated search_terms column on each chunk.

The expansion keeps the original lowercased token and adds each sub-token of length two or more. So getUserById contributes getuserbyid, get, user, by, and id. A query for user or id then matches that chunk through its search_terms column even though neither word appears as a standalone token in the source.

getUserById   ->  getuserbyid get user by id
parse_file    ->  parse_file parse file
HTTPServer    ->  httpserver http server

The query side mirrors this: your query string is itself expanded into the same sub-tokens and turned into an OR of the resulting terms before it hits FTS5. That is what makes a one-word query land on the right multi-word identifiers.

Python API

The facade method is search_lexical(query, top_k=10), returning a list[SearchResult] ordered best-first.

from rce.code import CodeEngine

engine = CodeEngine.open_or_create("/path/to/repo")
engine.index()

results = engine.search_lexical("user id lookup", top_k=10)
for r in results:
    print(round(r.score, 3), r.source, r.chunk.file_path,
          f"{r.chunk.start_line}-{r.chunk.end_line}")
engine.close()

Each SearchResult carries:

• .score — the negated BM25 score; larger is more relevant.
• .chunk — the matched Chunk (file path, line span, kind, name, content, and the stored search_terms).
• .source — SearchSource.LEXICAL for this modality.

Results also expose .matched_terms (the expanded query terms that were used) and a .freshness record indicating whether the file changed since it was indexed.

CLI

The search subcommand runs lexical search and prints one line per hit with the file, line span, and score.

# lexical search in the current repo
rce-code search "user id lookup"

# point at another repo and cap the result count
rce-code search "retry backoff" --repo /path/to/repo --top-k 5

src/auth/session.py:42-71  (score 8.314)
src/users/repo.py:10-28    (score 6.902)