Adding a Language

RCE Code parses source with tree-sitter. Because every language-specific detail lives in data — a grammar wheel, a small declarative profile, and two query files — adding a language is a data-driven change: you do not touch the extractors, the indexer, or any other engine code.

There are five touchpoints:

1. The grammar wheel — a PyPI package shipping the compiled tree-sitter grammar.
2. A factory entry in src/rce/code/parse/treesitter.py so the parser pool can build the grammar.
3. A LanguageProfile in src/rce/code/languages.py (namespace separator + comment node types).
4. A <lang>.scm query that captures definitions.
5. A <lang>.refs.scm query that captures references (calls + imports).

A sixth thing, detection (mapping a file to a language id), is usually already wired — you only add an entry if the file extension/name/shebang isn’t recognized yet.

1. Add the grammar wheel

tree-sitter grammars are distributed as standalone wheels. Add the dependency to [project] dependencies in pyproject.toml:

dependencies = [
    # ...
    "tree-sitter-java>=0.23",
    "tree-sitter-ruby>=0.23",
    # ...
]

These wheels ship no type stubs, so also add the import name to the [[tool.mypy.overrides]] module list so mypy doesn’t complain about a missing stub:

[[tool.mypy.overrides]]
module = [
    # ...
    "tree_sitter_java",
    "tree_sitter_ruby",
    # ...
]

Then install:

uv pip install -e ".[dev]"

Each wheel exposes a zero-argument language() that returns the grammar capsule — e.g. tree_sitter_ruby.language(). A few wheels that bundle several grammars use a named variant instead (TypeScript’s wheel exposes tree_sitter_typescript.language_typescript), so check the wheel before wiring step 2.

2. Register the grammar factory

src/rce/code/parse/treesitter.py keeps a registry, _FACTORIES, mapping a language id to the zero-arg callable that produces the grammar capsule. Import the wheel and add an entry:

import tree_sitter_ruby
# ...

_FACTORIES: dict[str, Callable[[], object]] = {
    "python": tree_sitter_python.language,
    "javascript": tree_sitter_javascript.language,
    "typescript": tree_sitter_typescript.language_typescript,
    "go": tree_sitter_go.language,
    "rust": tree_sitter_rust.language,
    "java": tree_sitter_java.language,
    "ruby": tree_sitter_ruby.language,
}

3. Add a LanguageProfile

src/rce/code/languages.py holds the small declarative bits that can’t be expressed as queries. Add an entry to _PROFILES:

_PROFILES: dict[str, LanguageProfile] = {
    # ...
    "ruby": LanguageProfile(
        name="ruby",
        qualified_separator="::",
        comment_node_types=frozenset({"comment"}),
    ),
}

The fields (name mirrors the registry key):

• qualified_separator — the string used to join nested names into a qualified name. Ruby namespaces with :: (so a method total in Billing::Invoice becomes Billing::Invoice::total); Python/Java/Go/JS use ".".
• comment_node_types — the grammar’s node-type name(s) for comments, consumed by the AST chunker. Ruby’s grammar uses a single comment node type; some grammars split them (Rust and Java use frozenset({"line_comment", "block_comment"})). Find the right names with the parse-tree snippet in §6.

4. Detection (usually already done)

src/rce/code/ingest/language.py maps a file to a canonical language id, in order: filename (_FILENAME) → extension (_EXT) → shebang (_SHEBANG). Ruby was already covered — .rb → ruby lives in _EXT and ruby → ruby in _SHEBANG — so Ruby needed no change here.

If your language’s extension isn’t recognized, add it to the relevant table. For example, to recognize a new .foo extension:

_EXT: dict[str, str] = {
    # ...
    ".foo": "yourlang",
}

Add to _FILENAME for extensionless files matched by exact name (e.g. Dockerfile), or to _SHEBANG for interpreter names found in a #! line. The id you map to here must match the key you used in _FACTORIES, _PROFILES, and your .scm filenames.

5. Write the two .scm files

Both query files live in src/rce/code/parse/queries/ and are named after the language id: <lang>.scm (definitions) and <lang>.refs.scm (references). The extractors load them by that name; nothing else references them.

Definitions — ruby.scm

The convention consumed by src/rce/code/parse/symbols.py: capture the definition node with @def.<kind> and its name identifier with @name. Valid kinds (the suffix after def.) are:

function, method, class, interface, enum, type.

A useful shortcut: capture both free functions and methods as @def.function. The extractor walks each definition’s ancestors, and a @def.function whose enclosing definition is a class is auto-promoted to METHOD. So you don’t need a separate method pattern — just capture all defs as functions and let the ancestor walk reclassify the ones inside a class.

The shipped ruby.scm:

(method name: (identifier) @name) @def.function
(singleton_method name: (identifier) @name) @def.function
(class name: (constant) @name) @def.class
(module name: (constant) @name) @def.class

Note: both Ruby class and module are captured as @def.class (both are class-like namespaces), and both method and singleton_method (i.e. def self.create) are captured as @def.function — inside a class/module they auto-promote to METHOD, while a top-level def stays FUNCTION.

References — ruby.refs.scm

The convention consumed by src/rce/code/parse/refs.py: capture a callee with @ref.call and an imported name with @ref.import. The text of an @ref.import capture has its surrounding quotes/backticks stripped automatically, so capture the string literal directly.

Most languages have a distinct import/include node, so you write one pattern for calls and one for imports. Ruby doesn’t — require and require_relative are ordinary method calls, syntactically identical to any other call. To split them out we use tree-sitter predicates (#match? / #not-match?):

((call method: (identifier) @ref.call) (#not-match? @ref.call "^require(_relative)?$"))
((call method: (identifier) @_req arguments: (argument_list (string (string_content) @ref.import)))
 (#match? @_req "^require(_relative)?$"))

Three things make this work, and they’re the subtle part of the whole guide:

1. Predicates are applied by this query path. refs.py runs the query through QueryCursor(query).matches(...), which honors #match? / #not-match?. The first pattern emits every call except requires; the second emits the string argument of a require as an import. The two are mutually exclusive, so each require is bucketed exactly once and nothing is double-counted.
2. The @_-prefixed helper capture. A predicate can only constrain a named capture, so to test the method name in the second pattern we must capture it — as @_req. But that capture is only a helper for the predicate; it is not a reference. By tree-sitter convention, a capture whose name starts with _ is predicate-only scaffolding, and refs.py honors that convention: it skips any capture whose name starts with _. So @_req constrains the predicate but is never emitted as a reference — only @ref.import (the string content) is. If you instead named it @ref.call, the require’s method name would leak out as a call reference.
3. The regex is anchored: ^require(_relative)?$. Anchoring matters. Without the ^...$, #match? would also fire on a user-defined method like require_login or required, wrongly treating it as an import (and, via the #not-match? in the first pattern, dropping it from the calls). With the anchors, only the exact names require and require_relative are routed to imports; everything else stays a normal @ref.call.

6. Finding node types

You’ll need the grammar’s exact node-type names (for the profile’s comment types and for writing the queries). Parse a snippet and print the tree:

import tree_sitter_ruby
from tree_sitter import Language, Parser

parser = Parser(Language(tree_sitter_ruby.language()))
print(parser.parse(b"class A; def m; end; end").root_node)

That prints the S-expression of the parse tree, showing every node type and field name (method, name:, identifier, constant, comment, ...). Iterate on your .scm against the same grammar until the captures are right:

from tree_sitter import Language, Parser, Query, QueryCursor

lang = Language(tree_sitter_ruby.language())
tree = Parser(lang).parse(b'require "set"\ndef f; compute; end\n')
query = Query(lang, open("src/rce/code/parse/queries/ruby.refs.scm").read())
for pattern_index, captures in QueryCursor(query).matches(tree.root_node):
    print(pattern_index, {name: [n.text for n in nodes] for name, nodes in captures.items()})

This Query + QueryCursor(query).matches(root) flow is exactly what the extractors (and the tests below) use, so getting it green here means it’ll work in RCE Code.

7. Tests

Mirror the two Ruby test files. They feed inline source bytes through the public extractors and assert on the result. Create tests/unit/test_symbol_extract_<lang>.py and tests/unit/test_ref_extract_<lang>.py.

Symbols — assert names, kinds (including in-class method promotion), and qualified names:

from rce.code.parse.symbols import extract_symbols
from rce.code.parse.treesitter import ParserPool
from rce.code.types import SymbolKind

RUBY = b"""module Billing
  class Invoice
    def total(items)
      items.sum
    end
    def self.create(data)
      new(data)
    end
  end
end

def top_level(x)
  x + 1
end
"""


def _by_name(src: bytes = RUBY) -> dict[str, object]:
    return {s.name: s for s in extract_symbols(ParserPool(), "ruby", src)}


def test_ruby_module_and_class_are_classlike():
    s = _by_name()
    assert s["Billing"].kind == SymbolKind.CLASS
    assert s["Invoice"].kind == SymbolKind.CLASS


def test_ruby_methods_reclassified_inside_class():
    s = _by_name()
    assert s["total"].kind == SymbolKind.METHOD
    assert s["create"].kind == SymbolKind.METHOD  # def self.create -> singleton_method


def test_ruby_top_level_def_is_function():
    assert _by_name()["top_level"].kind == SymbolKind.FUNCTION


def test_ruby_qualified_name_uses_double_colon():
    assert _by_name()["total"].qualified_name == "Billing::Invoice::total"

References — assert the call/import bucketing, and (because Ruby uses an anchored predicate) that a user method merely named like require stays a call:

from rce.code.parse.refs import extract_references
from rce.code.parse.treesitter import ParserPool

RUBY = b"""require "set"
require_relative "./helper"
def f(x)
  compute(x)
  puts "noise"
end
"""


def _refs(src: bytes = RUBY):
    pool = ParserPool()
    tree = pool.parse("ruby", src)
    return extract_references(pool, tree, src, "ruby")


def test_ruby_captures_calls():
    calls = {r.to_name for r in _refs() if not r.is_import}
    assert "compute" in calls and "puts" in calls


def test_ruby_require_is_import_not_call():
    refs = _refs()
    assert {r.to_name for r in refs if r.is_import} == {"set", "./helper"}
    calls = {r.to_name for r in refs if not r.is_import}
    assert "require" not in calls and "require_relative" not in calls


def test_ruby_non_require_string_arg_is_not_imported():
    # `puts "noise"` must NOT become an import — the require predicate filters it out.
    assert "noise" not in {r.to_name for r in _refs() if r.is_import}


def test_ruby_user_method_named_like_require_is_a_call_not_dropped():
    # The require predicate is anchored (^require(_relative)?$), so a user method whose name
    # merely starts with "require" is still captured as a normal call, not silently dropped.
    src = b"def f(user)\n  require_login(user)\n  required(user)\nend\n"
    pool = ParserPool()
    refs = extract_references(pool, pool.parse("ruby", src), src, "ruby")
    calls = {r.to_name for r in refs if not r.is_import}
    assert "require_login" in calls and "required" in calls
    assert not any(r.is_import for r in refs)  # nothing here is a real require

If a test enumerates the supported languages, add yours. For Ruby that was the single tuple in tests/unit/test_treesitter.py::test_supports_known_languages:

for lang in ("python", "javascript", "typescript", "go", "rust", "java", "ruby"):
    assert pool.supports(lang)

8. Verify

Run your new tests first, then the full gate:

pytest tests/unit/test_*_ruby.py -v
pytest -q && ruff check src tests && mypy

See Supported Languages for the full matrix of what ships today and the documented per-language bounds.