I wanted to explore how AI could make this process easier.

Instead of manually searching through documents, what if we could upload a file, ask a question in plain English, and get an answer based on its contents?

That idea introduced me to Retrieval-Augmented Generation (RAG).

In this article, I'll walk through how I approached building a document question-answering application, the technologies involved, how the components work together, and the engineering challenges I needed to consider.

The Problem I Wanted to Solve

Imagine a company has hundreds or thousands of documents containing policies, reports, product manuals, and technical documentation.

An employee asks:

"What is the company's annual leave policy?"

A traditional file search may help locate a document, but the employee still needs to open it and find the relevant information.

A general-purpose LLM might explain annual leave, but it cannot automatically know the contents of private company documents it has never received.

I wanted to explore a system that could retrieve the relevant information from documents and use an LLM to generate a useful answer.

That is the problem RAG is designed to address.

Understanding the Architecture Before Writing Code

Before putting the components together, I broke the application into two workflows.

1. Document processing

Uploaded documents → Text extraction → Chunking → Embeddings → Vector database

2. Question answering

User question → Retrieval → Relevant document chunks → LLM → Generated answer

The first workflow prepares the documents for searching. The second uses those prepared documents to answer questions.

Understanding this separation is important because I don't want to process every document from scratch whenever someone asks a new question.

Step 1: Processing the Documents

The first challenge is getting useful text out of uploaded files.

A PDF may contain paragraphs, headings, tables, page numbers, or scanned images. The application needs to extract its text before it can search the content.

For a basic implementation, I would start with text-based PDFs and add OCR support for scanned documents later.

Once the text is extracted, the application can prepare it for retrieval.

Step 2: Splitting the Text Into Chunks

Why not store an entire document as one large piece of text?

Because retrieving a small, relevant passage is often more useful than retrieving a whole document.

For example, an employee handbook might contain separate sections for annual leave, sick leave, and remote work.

I would split the extracted text into smaller chunks, ideally preserving paragraph and section boundaries where possible.

Chunking requires some experimentation. If the chunks are too small, they may lose context. If they are too large, the retrieved content may include irrelevant information.

Overlapping chunks can also help preserve information across boundaries.

Step 3: Generating Embeddings

Next, the application needs a way to find information based on meaning rather than exact word matches.

This is where embeddings come in.

An embedding model converts a piece of text into a numerical vector.

Consider these two sentences:

• "Employees receive 20 days of annual leave."

• "How many vacation days can staff take each year?"

The wording differs, but the sentences express related ideas. An embedding-based search system can help identify that relationship.

I would generate embeddings for the document chunks and later generate an embedding for each user question using a compatible model.

Step 4: Storing the Data in a Vector Database

The application needs to store the embeddings so that relevant chunks can be retrieved efficiently.

For a prototype, ChromaDB is one option worth exploring. Other options include Qdrant and PostgreSQL with pgvector.

Alongside each embedding, I would store the corresponding text and metadata, such as:

• Document ID

• File name

• Page number

• Chunk ID

This information helps the application retrieve the original text and identify where an answer came from.

For a larger application, document ownership and access permissions would also need to be enforced during retrieval.

Step 5: Retrieving Relevant Information

Suppose a user asks:

"How many annual leave days do employees receive?"

The application converts the question into an embedding and searches the vector database for relevant chunks.

The retrieval system might return a passage stating that eligible employees receive 20 days of annual leave each year.

At this stage, the application has found potentially useful information. It has not generated the final answer yet.

Retrieval quality matters: if the relevant passage is missing from the retrieved results, the language model may not have enough evidence to answer correctly.

Step 6: Connecting the Retrieved Context to an LLM

Now comes the generation step.

The application combines the user's question with the retrieved text and passes both to an LLM.

A simplified prompt could look like this:

Question:

How many annual leave days do employees receive?

Context:

Eligible employees receive 20 days of annual leave each year.

Instruction:

Answer using the provided context. If the context does not contain enough information, say so rather than inventing an answer.

The model can then generate a natural-language response based on the supplied information.

The important point is that the application performs retrieval first and provides the resulting context to the LLM. The LLM does not automatically know the contents of the vector database.

Step 7: Making the Application Useful

Generating an answer is only part of the application.

I would also want to provide:

• A document upload interface

• A chat interface for asking questions

• Loading indicators while processing files

• Source references for generated answers

• Clear error messages for unsupported files

• A way to update or delete uploaded documents

For a production system, I would also need authentication, secure file storage, access controls, and reliable handling of document updates.

These features help turn a technical prototype into something people can actually use.

The Challenges I Would Focus On

Building a RAG system is not simply a matter of connecting a vector database to an LLM.

Some important challenges include:

Retrieval accuracy: Does the system find the correct passage for a question?

Chunking strategy: Does each chunk preserve enough information to be understood independently?

Hallucinations: Does the model introduce claims that are not supported by the retrieved text?

Document updates: What happens when a source file changes or gets deleted?

Performance: How do retrieval time, embedding generation, and model response time change as the document collection grows?

These questions need to be investigated through actual testing rather than assumptions.

For example, I would prepare a set of questions with known answers, check which chunks are retrieved, and compare the generated responses with the original documents. That would help identify whether a problem comes from retrieval, document processing, or generation.

What I Learned From Exploring RAG

The most interesting idea behind RAG is that a language model does not have to contain all the information it needs inside its training data.

An application can retrieve external information when a question is asked and use that information as context.

This connects several areas of software engineering:

• Document processing

• Embedding models

• Vector search

• Backend development

• LLM integration

• Data security

• Testing and evaluation

It also highlights an important lesson: a fluent answer does not necessarily mean the system retrieved the right evidence.

The quality of a RAG application depends on the complete pipeline, not just the language model.

What's Next?

I would like to explore how retrieval quality changes with different chunk sizes, how metadata filtering works across document collections, and how source citations can make answers easier to verify.

The next step is to test these ideas in a working application and document the results.

For me, the value of learning AI engineering is not just understanding what a technology does. It is understanding how its components work together, where they fail, and how to improve them.

Thanks for reading! If you're exploring RAG or building document-based AI applications, I'd love to hear what you're learning and which challenges you've encountered.