Designing an End-to-End RAG Architecture from Scratch

Building an AI-powered application is easy to describe. Upload documents → ask a question → get an answer. Actually building that flow is a different story. While working on Guidely, an internal knowledge assistant, I wanted to understand what happens between those two endpoints. More importantly, I wanted to design the system so that each part had a clear responsibility and could be tested independently. The result was an end-to-end Retrieval-Augmented Generation (RAG) system built around a simple pipeline: Documents ↓ Parsing ↓ Chunking ↓ Embeddings ↓ Vector Store ↓ Semantic Search ↓ Response Generation ↓ Citations ↓ React Frontend The interesting part wasn't simply getting an LLM to answer questions. It was designing the architecture that makes the entire pipeline work reliably. What is Guidely? Guidely is an internal knowledge assistant that allows users to ask questions about a collection of organizational documents. Instead of expecting an AI model to already know everything about an organization's internal knowledge, Guidely retrieves relevant information from the organization's documents and uses that information to construct an answer. For example, a user might ask: "What is TrustLayer?" Guidely searches the organization's knowledge base, retrieves the most relevant sections, and uses those sections as context for generating the answer. The response is then presented together with the sources that support it. This is the basic idea behind RAG. But I wanted the architecture to make the flow explicit rather than hiding everything inside one large function. The Architecture The first major decision was to separate the system into distinct stages. ┌─────────────────┐ │ Documents │ └────────┬────────┘ ↓ ┌─────────────────┐ │ Parser │ └────────┬────────┘ ↓ ┌─────────────────┐ │ Chunker │ └────────┬────────┘ ↓ ┌─────────────────┐ │ Embeddings │ └────────┬────────┘ ↓ ┌─────────────────┐ │ Vector Store │ └────────┬────────┘ ↓ User Query ↓ ┌─────────────────┐ │ Semantic Search │ └────────┬────────┘ ↓ ┌─────────────────┐ │ Response │ └────────┬────────┘ ↓ ┌─────────────────┐ │ React Frontend │ └─────────────────┘ Each component answers a different question. Parser: How do I extract text from a document? Chunker: How do I divide that text into useful pieces? Embeddings: How do I represent those pieces numerically? Vector store: How do I efficiently find similar pieces? Search: Which pieces are relevant to this question? Response: How do I turn those pieces into a useful answer? Frontend: How do I present the answer to a human? That separation became one of the most important architectural decisions in the project. 1. Document Ingestion The first stage is getting documents into the system. Guidely supports documents such as: TXT PDF DOCX The upload API receives a document and stores it in the document directory. But uploading a file isn't the same thing as making it searchable. The document needs to go through the ingestion pipeline. Uploaded document ↓ Determine file type ↓ Parse document ↓ Extract text ↓ Chunk text ↓ Generate embeddings ↓ Store vectors + metadata This separation means the upload layer doesn't need to understand embeddings or semantic search. Its responsibility is simply: Get the document into the system. 2. Parsing Once a document exists, Guidely needs to extract its text. The parser provides a common interface: parse_document(file_path) Internally, the appropriate parser can be selected depending on the file type. For example: .txt → text parser .pdf → PDF parser .docx → DOCX parser The important architectural idea here is that the rest of the pipeline doesn't need to care where the text came from. Once parsing is complete, everything downstream works with: text: str This keeps the pipeline format-independent. 3. Chunking A document can be thousands of words long. Sending an entire document into a retrieval system isn't ideal. Instead, Guidely breaks the extracted text into smaller chunks. The chunking strategy uses tokens rather than simply splitting every N characters. For example: Document ──────────────────────────── Paragraph 1 Paragraph 2 Paragraph 3 Paragraph 4 Paragraph 5 ... Chunk 1 ──────────────── Paragraph 1 Paragraph 2 Chunk 2 ──────────────── Paragraph 2 Paragraph 3 Chunk 3 ──────────────── Paragraph 3 Paragraph 4 The overlap is intentional. If a relevant sentence happens to sit near a chunk boundary, overlap reduces the chance that important context gets separated. The chunking function therefore has two important parameters: chunk_size = 800 overlap = 100 The exact values can be tuned later. The important design decision was making chunking its own service rather than embedding the logic inside document ingestion. 4. Embeddings This is where the system starts moving from traditional text processing into semantic search. Each chunk is converted into a vector representation. Conceptually: "TrustLayer is a decentralized protocol..." ↓ Embedding Model ↓ [0.021, -0.143, 0.782, ...] The same process happens when a user asks a question. "What is TrustLayer?" ↓ Embedding Model ↓ Query Vector Now the system can compare the query vector with document vectors. This is the foundation of semantic retrieval. One challenge I encountered here was model selection. I initially explored hosted embedding APIs but ran into API limitations. I eventually moved toward a local Sentence Transformers model. That decision had an architectural benefit beyond simply solving the immediate problem: the embedding layer became independent from the rest of the application. If I change the embedding model later, the search API doesn't need to change. 5. Vector Storage Once chunks have embeddings, the vectors need to be stored somewhere. For Guidely, I used FAISS. The basic relationship looks like: Vector │ ├── FAISS index │ └── Metadata ├── filename ├── chunk information └── original text The vector index handles similarity search. The metadata provides the information needed to understand what a vector represents. This separation is important. FAISS answers: Which vectors are closest to this query? The metadata answers: What do those vectors actually represent? 6. Semantic Search When a user submits a question, the query follows a shorter path: User question ↓ Create embedding ↓ FAISS similarity search ↓ Top K results ↓ Relevant document chunks The search service looks roughly like this: query_embedding = create_embeddings([query])[0] distances, indices = index.search( query_vector, top_k ) The returned vector IDs are then mapped back to document metadata. One useful property of this architecture is that the search service doesn't need to know anything about the frontend. It simply returns structured results. For example: { "filename": "faq.txt", "text": "TrustLayer is a decentralized protocol..." } That keeps the backend boundary clean. 7. Why Similarity Alone Isn't Enough One of the more important problems I encountered was handling irrelevant questions. A vector database will usually return something. Even if the user asks a question completely unrelated to the knowledge base, FAISS can still return the nearest vectors. That creates a dangerous situation: Irrelevant question ↓ Similarity search ↓ Some vaguely similar chunks ↓ AI generates an answer ↓ Citation appears The system can therefore look confident even when it shouldn't be answering. This led to an important architectural requirement: Retrieval needs a relevance boundary. Instead of blindly accepting the top K results, the system needs to determine whether the retrieved results are actually relevant enough to support an answer. This is also where citations need to be handled carefully. A citation should not appear simply because a document happened to be returned by FAISS. It should appear because that document actually contributed relevant context to the answer. 8. Response Generation After retrieval, the relevant chunks become context for the response layer. Conceptually: User Question + Retrieved Context ↓ Response Generator ↓ Answer + Sources The backend returns a structured response rather than exposing internal implementation details. For example: { "answer": "TrustLayer is a decentralized protocol on Solana...", "citations": [ { "source": "faq.txt", "snippet": "It allows clients and talent to collaborate directly..." } ] } This distinction matters. The backend can contain things such as: chunk IDs FAISS indices similarity distances embedding vectors But a user doesn't need to see any of those. The response layer acts as the boundary between the internal retrieval system and the human-facing application. --- 9. Query-Aware Citations Citations became an interesting part of the project. Initially, returning the entire retrieved chunk produced poor results. A citation could contain an entire section of a document even when only one sentence supported the answer. For example: faq.txt TRUSTLAYER FREQUENTLY ASKED QUESTIONS ... TABLE OF CONTENTS ... Q1... Q2... Q3... That's technically a citation, but it isn't particularly useful to a human. The goal became: faq.txt "It allows clients and talent to collaborate directly..." The citation should answer: "Where did this information come from?" not: "Here is a large portion of the document." This led to a query-aware citation strategy where the snippet is selected based on the information relevant to the user's question. 10. The Frontend The frontend is intentionally separated from the retrieval system. I built the interface with React. The user sees: Guidely Ask your organization's knowledge ┌──────────────────────────────────────────┐ │ Ask a question... → │ └──────────────────────────────────────────┘ Answer TrustLayer is a decentralized protocol... Sources ┌──────────────────────────────────────────┐ │ 📄 faq.txt │ │ It allows clients and talent... │ └──────────────────────────────────────────┘ The frontend doesn't need to know how embeddings work. It doesn't know what FAISS is. It doesn't need to understand chunking. It simply consumes the response contract from the API. That separation makes the system much easier to reason about. 11. The Admin Side The second major frontend surface is the knowledge-base administration page. The admin interface allows documents to be uploaded and viewed. The architecture looks like: Admin ↓ Upload document ↓ FastAPI ↓ Document storage ↓ Ingestion pipeline ↓ Embeddings ↓ FAISS The interface intentionally hides implementation details. An administrator doesn't need to know: "Your document has been converted into a 768-dimensional vector and inserted at index 42." They need to know: "Your document has been uploaded and is available." This distinction influenced a lot of the UI decisions. 12. FastAPI as the Backend Boundary FastAPI became the boundary between the frontend and the internal services. The application is organized around responsibilities such as: app/ ├── routers/ │ ├── search.py │ └── documents.py │ ├── services/ │ ├── parser.py │ ├── chunker.py │ ├── embeddings.py │ ├── vector_store.py │ └── response.py │ └── main.py This structure isn't about creating as many files as possible. It's about making the data flow understandable. A search request can be traced through: search router ↓ search service ↓ embedding service ↓ vector store ↓ response service Similarly, document ingestion has its own path. That makes debugging considerably easier. 13. One of the Most Important Lessons: Architecture Before Features The biggest challenge wasn't writing the individual functions. It was deciding where each responsibility belonged. For example, it would have been possible to create one large function: def ask_question(query): # create embedding # search FAISS # retrieve documents # generate response # format citations # return result It would probably work. But it would also become difficult to test and modify. Instead, Guidely separates those responsibilities. That gives me the ability to change: the embedding model the vector database the chunking strategy the response formatting the frontend without necessarily rewriting the entire system. That was the architectural goal. 14. The Complete Pipeline After putting everything together, the final architecture looks like this: DOCUMENT INGESTION Document ↓ Parser ↓ Text ↓ Chunker ↓ Chunks ↓ Embedding Model ↓ Vectors ↓ FAISS + Metadata │ │ │ ▼ QUERY PIPELINE User Question ↓ Embedding Model ↓ Query Vector ↓ FAISS Similarity Search ↓ Relevant Chunks ↓ Relevance Filtering ↓ Response Generation ↓ Answer + Query-Aware Citations ↓ React UI This is what made Guidely feel like a complete system rather than just an AI chatbot. Next steps The current architecture works, but there are several areas I would improve as the project evolves. Better retrieval evaluation Similarity scores alone aren't enough. I'd like to build a proper evaluation dataset containing: questions expected relevant documents expected chunks expected answers This would make retrieval quality measurable rather than something I evaluate manually. Better chunking Different documents have different structures. A fixed token-based chunk size isn't necessarily optimal for: FAQs API documentation policies technical manuals A future version could use structure-aware chunking. Better citation extraction Citation snippets could be selected more intelligently based on the query and the generated answer. Persistent vector infrastructure FAISS works well for a project like this, but a production deployment could benefit from a persistent vector database depending on scale and operational requirements. Authentication and permissions The current admin functionality is primarily focused on document management. A production knowledge assistant would also need authentication, authorization, document ownership, and potentially per-user or per-team knowledge bases. As I conclude! The most valuable part of this project wasn't getting an AI model to answer a question. It was learning to think about the system as a collection of independent stages. A useful mental model is: Don't start with: "How do I make an AI answer questions?" Start with: "How does information move through the system?" Once that question is answered, the architecture becomes much clearer. Documents become text. Text becomes chunks. Chunks become vectors. Vectors become searchable knowledge. Search results become context. Context becomes an answer. And the answer becomes something a human can actually use. That is the architecture behind Guidely: a small but complete end-to-end RAG system designed not just to work, but to make each stage understandable, replaceable, and testable. The next challenge is measuring how well each stage works.