Improving Local Techdocs for Your AI Coding Agent

After crawling and cleaning technical documentation locally, we add page classification, embeddings, and a knowledge graph to make the data much more useful for AI coding agents.

This is the third post in our series on making technical documentation available to AI agents and knowledge bases, based on our work building an AI-first knowledge base. In the first post, we describe how to crawl documentation sites, clean page content, and generate image descriptions. In the second post, we cover practical issues that arise when crawling complete technical-documentation sites.

Here I want to describe what we do afterwards to structure the crawled documentation further and make it available in a more useful form. That can include use in your local AI coding agent. At a high level, we classify pages and embed them with a local model. Then we build a knowledge graph that combines explicit hyperlinks with semantic similarity edges.

Classifying Pages

A lot of pages in technical documentation are not actual content. Many are purely navigation hubs - index pages that just link to other, more concrete pages. Others only show legal terms or changelogs. If you are doing further work on the data, you probably want to ignore most of those.

We do a first rule-based pass to cover as much as possible quickly and cheaply on a local machine, without involving LLMs. To do so, we check the URL against a set of patterns for each category:

LEGAL_PATTERNS = ["/legal/", "/privacy", "/terms", "/eula", "/cookie"]

def classify_by_rules(url: str, title: str, content: str) -> str | None:
    if any(p in url.lower() for p in LEGAL_PATTERNS):
        return "legal"
    # ... same pattern for changelog, reference, etc.
    # Navigation: short content with mostly links
    if len(content.split()) < 200 and "[" in content:
        return "navigation"
    return None  # needs LLM classification

Everything the rules cannot classify gets sent to a local LLM. We give it the URL, title, the first 200 words of the page, and a list of its headings, then ask it to classify the page by primary intent. It uses conceptual, tutorial, how-to, and example for the main content pages, based on the Diátaxis framework for documentation, as well as navigation, reference, legal, and changelog for structural pages, plus broken and misc.

With this two-pass approach, the rule-based step handles the easy cases cheaply, and the LLM only sees what it actually needs to. We can then filter out legal, changelog, navigation, and reference pages when doing further work on the data, and focus only on actual content pages.

Embedding Pages

With pages classified, the next step is embedding them. We use a local sentence transformer model to avoid api costs and make the process faster. This seems okay for this use case so far.

If a page exceeds the token limit, we split it at heading boundaries and average the resulting chunk embeddings:

def embed_page(content: str) -> list[float]:
    chunks = re.split(r'(?m)^#{1,3} ', content)
    if len(chunks) == 1:
        return model.encode(content).tolist()
    embeddings = [model.encode(chunk) for chunk in chunks if chunk.strip()]
    avg = np.mean(embeddings, axis=0)
    return (avg / np.linalg.norm(avg)).tolist()

We embed from the cleaned markdown rather than plain text. Headings and code blocks have semantic meaning that helps the model understand page structure. List structure does too.

Building a Knowledge Graph

With the embeddings, we can build a graph that includes semantic similarity in combination with the explicit hyperlinks in the documentation. We store two types of edges: link edges (explicit hyperlinks from page A to page B as directed edges with no weight) and semantic edges (between pages with high embedding similarity, we store two directed edges with the cosine similarity as the edge weight).

We only include actual content pages in the semantic graph with pages classified as navigation, legal, reference etc. excluded. We also set a configurable similarity threshold and only add edges between pages with higher similarity than that (currently 0.75). Finally, we cap the number of neighbors per page (currently 20) so you don’t end up with a few massively connected hubs.

All of this page data, classifications, embeddings, and graph edges ends up in the same SQLite database.

The Result

First, crawl the technical documentation as described in the second post. Then extract the page content and describe images as in the first post. Finally, run these post-processing steps on the crawled data.

You end up with a completely self-contained, local SQLite database with the full documentation stored in a form your AI can use easily. Agents can query it with plain SQL and models are quite good at that. They can use the classification to filter out noise and read only actual content pages, or for more fine-grained queries like “Show me all how-to pages related to topic X”. And they can use the embeddings to find semantically similar pages, or use the knowledge graph by following both the explicit hyperlinks written by the documentation authors and the implicit similarity edges we derived from the content.

If you have built something similar or think this is interesting, I would love to hear about it. How are you using local documentation with your coding agents? What would you do differently?