Your AI Application Is Only as Good as the Data Behind It

I've been working with AI applications and RAG architectures, and one thing keeps coming up: We spend a lot of time discussing LLMs, embeddings, vector databases, agents, and prompts. But we don't always spend the same amount of time thinking about the data. And honestly, that can become a problem. A simple example Imagine building an AI assistant for an enterprise. You connect it to company documentation and ask: "What is our vendor onboarding process?" The knowledge base contains: An old vendor onboarding policy. A newer version with some changes. A separate document describing vendor security checks. Another document explaining vendor offboarding. Duplicate sections with slightly different wording. The AI retrieves a few passages and generates an answer. The answer might sound convincing. But did it retrieve the correct version? Did it find all the relevant sections? Did it miss an important requirement? This isn't necessarily an LLM problem. It could be a data organization and retrieval problem. Why structure matters When building AI applications, the way information is stored can have a direct impact on retrieval quality. For example, consider a document containing: Vendor Management Onboarding: Vendor approval is required. Security review must be completed. Offboarding: Access must be revoked. Company assets must be returned. Now imagine the same information stored as an unstructured block of text with no clear headings or metadata. The content hasn't changed, but the ability to identify and retrieve specific pieces of information may be affected. Good document structure gives retrieval systems useful signals. Some things worth getting right: Meaningful chunking: Split documents around logical sections rather than arbitrary text lengths alone. Metadata: Preserve document names, section names, versions, timestamps, and ownership. Consistency: Use consistent terminology across documents. Data quality: Remove duplicates, outdated content, and conflicting information where appropriate. Traceability: Make it possible to identify where retrieved information originated. The RAG connection In a RAG architecture, we generally have a pipeline: Source Documents | v Data Cleaning | v Document Processing | v Chunking + Metadata | v Embeddings + Indexing | v Retrieval | v Context Assembly | v LLM Generation | v Grounded Answer Every stage matters. If the source data is poor, chunking won't magically make it accurate. If metadata is missing, retrieval loses useful context. If the wrong sections are retrieved, even a strong LLM may struggle to produce a complete answer. And if the information itself is outdated, a perfectly functioning pipeline can still return an outdated answer. One distinction I find important There is a difference between retrieving information that is similar to a question and retrieving information that is sufficient to answer the question. For example, asking about vendor onboarding and offboarding may require two different sections. Retrieving only the onboarding section might look like a successful semantic search. But it doesn't provide enough evidence for the complete question. This is why I think evidence coverage, data organization, and retrieval quality deserve much more attention in enterprise AI engineering. Final thought We are entering a phase where building an AI application is becoming easier. But building one that consistently provides accurate, grounded, and useful answers is a different challenge. I don't think every AI problem needs a bigger model or a more complicated prompt. Sometimes, we need to go back to basics: Get the data right. Structure it properly. Preserve its context. Keep it current. Then build AI on top of it. The foundation matters. About the Author I'm Ashwin Giridharan, a Cloud & Enterprise Solutions Architect focused on building scalable systems and exploring the architecture of AI. My interests include enterprise architecture, cloud platforms, distributed systems, Generative AI, LLMs, RAG, Agentic AI, and taking AI solutions from experimentation to production. I share practical insights, technical explorations, and lessons learned while building and understanding modern technology systems. Connect and explore my work: Website: https://ashwingiridharan.com/ LinkedIn: https://www.linkedin.com/in/ashwingiridharan7/ GitHub: https://github.com/giridharan-ashwin