Case study: Using large language models and structured document navigation to make complex technical standards easier to interpret.

The challenge

Technical standards and regulatory documents can be extraordinarily difficult to navigate. They often contain hundreds of pages of interconnected requirements, definitions, exceptions, tables, and references to other sections.

Finding the correct answer to a technical question can require consulting several parts of a document, understanding how different provisions relate to one another, and identifying exceptions that substantially change the interpretation of a rule.

Large language models offer a promising way to make this information more accessible. However, applying a conventional AI chatbot to this kind of material presents a serious problem: plausible answers aren’t necessarily correct answers.

A system intended to help professionals interpret technical requirements needs to do more than produce convincing explanations. It needs to find the relevant source material, navigate relationships between provisions, and provide answers that can be verified.

Why conventional RAG wasn’t enough

An increasingly common approach to building AI systems around proprietary documents is Retrieval-Augmented Generation (RAG).

Typically, documents are divided into chunks of text, converted into vector embeddings, and stored in a searchable index. When a user asks a question, the system retrieves semantically similar chunks and supplies them to a language model.

This works well for many applications, but technical standards present particular challenges.

An individual provision may depend on definitions elsewhere in the document, exceptions in a neighbouring section, or requirements inherited from a parent section. Extracting a relevant paragraph without its structural context can result in an incomplete or misleading answer.

The problem isn’t necessarily that the model can’t reason about the material. It’s that the retrieval process may not provide the right information.

A different approach: preserving document structure

For this project, I developed an alternative approach based on representing technical documents as structured, hierarchical data rather than collections of independent text fragments.

The document is decomposed into a navigable tree that preserves its organization, from major sections down to individual provisions.

Instead of relying entirely on similarity search to select passages, the AI can explore this structure, locating relevant material, examining surrounding context, and following references to related provisions.

The idea is to give the language model something closer to the experience of navigating the original technical document, while retaining the flexibility of a natural-language interface.

This approach has several advantages:

  • Structural context: Individual provisions remain connected to the sections and subsections that give them meaning.
  • Cross-references: The model can investigate related requirements rather than relying on isolated excerpts.
  • Targeted retrieval: The system can navigate to relevant material without placing the entire document into the model’s context.
  • Verifiable answers: Responses can be grounded in specific source provisions, allowing users to inspect the underlying material.

The approach combines conventional document processing and data structures with the language understanding and reasoning capabilities of modern LLMs.

Testing and refinement

Accuracy is especially important when working with technical requirements.

During development, I tested the system with challenging questions designed to expose weaknesses in document retrieval and interpretation.

These tests helped identify cases where the system missed relevant context, failed to account for exceptions, or otherwise produced incomplete answers.

I refined the document representation and navigation approach in response to these failures.

This iterative process proved important. Building a useful AI system involves more than connecting a language model to a data source. It requires understanding how and why the system fails, then engineering solutions that address those failure modes.

The result

The project produced a working AI-assisted reference system capable of answering specialized questions by navigating structured technical documentation.

Testing demonstrated the promise of the approach, including its ability to handle questions that require consulting multiple related provisions.

The system also demonstrated an important principle: effective AI applications often depend as much on how information is organized and made accessible to the model as on the capabilities of the model itself.

The objective is not to replace professional judgment or authoritative source documents, but to make complex information faster to navigate and easier to work with.

Broader applications

Although this project focused on technical standards, the underlying challenges are common across many industries.

Legal and regulatory materials, engineering specifications, policy manuals, operational procedures, and other complex document collections all contain information whose meaning depends on structure and context.

AI systems that can navigate these relationships offer significant potential for organizations that depend on large collections of specialized knowledge.

For me, the project illustrates the kind of AI engineering I find most interesting: combining established software engineering techniques with emerging AI capabilities to solve problems that conventional software has struggled to address.

Developed by Entra Consulting. Certain project and client details have been omitted.