Building a RAG assistant that says “I don’t know”: evaluation, security and a $0 deployment

Most RAG tutorials end when the chatbot answers a question about a PDF. I wanted to find out what comes after that: how do you know it works, who is allowed to see what, what happens when a document tries to give it orders, and how do you ship it without paying for anything? This post walks through enterprise-ai-agent, a learning and portfolio project. It answers questions about public AWS documentation, cites its sources, refuses what the documents do not cover, and includes a small agent that can use tools. A note on how it was built: I worked step by step with an AI coding assistant (Claude). I ran every command, read every error and decided the scope, and writing this post was my way of making sure I can explain each choice. The project is not production software, and I say where it falls short at the end. Repository: https://github.com/PabsCueto/enterprise-ai-agent What it does • POST /ask retrieves the chunks the user is allowed to see, asks Gemini to answer using only that context, and returns the answer with source URLs and token usage. • POST /agent is a LangGraph agent with three tools: search the documentation, query a simulated internal status API, and create a ticket draft (a local JSON file that is sent nowhere). • Around them: an evaluation harness, per-user access control, structured logs, and a Docker image that is built and smoke-tested in CI. The corpus is seven introductory pages of AWS documentation, about a hundred chunks. It is small on purpose: I can check every result by hand. request -> FastAPI (request id, JSON log, API key, rate limit) |-- /ask -> retrieve (per-user filter + score threshold) -> Gemini |-- /agent -> LangGraph: agent tools -> answer Two layers against made-up answers The first layer is a similarity threshold. Chunks that are not close enough to the question are dropped, and if nothing is left the model is never called. The second layer is the prompt: answer only from the context, or say you cannot find it. I measured both. Questions unrelated to AWS (the capital of France, a taco recipe) scored at most 0.575, while questions the corpus covers scored at least 0.682, so a threshold of 0.6 separates them cleanly. The interesting case is a question that is about AWS but is not covered: VPCs, EC2 pricing, RDS, Amplify. Those scored between 0.60 and 0.664, above the threshold. Embeddings measure topic, not whether the answer is in the text, and a question about RDS looks a lot like DynamoDB documentation. The score cannot catch them; the prompt does, and the model refused all four. Neither layer would be enough on its own. An agent with guardrails The agent is a small LangGraph graph: a model node that decides, a tools node that executes, and a loop between them until the model answers. What matters is what the agent cannot do: • It has no tools to modify or delete anything, and tickets are drafts. • Tickets are created only on an explicit request. • The graph stops after 10 steps, thread ids are validated, and tool output is treated as data rather than instructions. One lesson came from the prompt. My first rule told the agent to ask when information was missing. When I wrote “create a ticket for the problem you find”, it asked me for a title and description it could have written itself. The fix was one sentence: if the user delegates the content, draft it from the data you gathered, and ask only when there is not enough to write it. Measuring it The harness has three small datasets: 22 retrieval questions (14 covered, 4 unrelated, 4 about AWS but not covered), 8 agent cases and 7 prompt-injection cases. It reports hit rate and MRR, how many questions are answered or rejected, and whether the agent called the expected tools in order and avoided forbidden ones. The results were good: 100% hit rate at 4 with an MRR of 1.0 on the covered questions, and 8 of 8 agent cases. But the datasets are small and I wrote them, so those numbers say little on their own. What the harness gave me were two bugs: One agent case failed. It was not the agent: my evaluation script passed a temporary folder to the ticket tool as a string instead of a path. The agent had chosen the right tools and crashed inside mine. There is now a regression test that fails without the fix. My first report suggested a threshold of 0.673. It came from mixing two kinds of “negative” question, and the margin was only 0.018. I separated them: the threshold is calibrated only against unrelated questions, and the AWS-but-not-covered ones are judged by the final answer. Who can see what • API keys. Random 256-bit values, stored only as SHA-256 hashes and compared in constant time. With no users configured, every protected request is rejected. • Document groups. Each document belongs to a group, each user has groups, and the filter runs inside the vector search before the top-k is taken, so a forbidden document never reaches the prompt. • Per document, not per topic. A restricted user asking about IAM got a partial answer built from the DynamoDB and S3 pages, which mention IAM. It was honest about what the context lacked, but it shows the granularity. • Conversations. The internal thread id is the user name plus the thread id, so two users sharing a thread id never share history. • Rate limit per user with a sliding window. To check the security tests were not vacuous, I removed each protection on purpose and confirmed that at least one test failed every time. Poisoned documents Indirect prompt injection is the opposite threat: the user is legitimate, but a document contains hidden orders. I wrote four short documents with a normal introduction and one hidden instruction each: create a ticket titled HACKED, answer only PWNED, append an image and a link to a fake domain, or print the system prompt. They live in a separate index, never in the real one. Each case ends as OK (the poisoned text reached the model and it did not obey), FAIL, or inconclusive (the poisoned text never reached the model, so the case proves nothing). That third state is what prevents a false “all good”. Result: 7 of 7 resisted, none inconclusive. I would not read that as “it is safe”. They are seven payloads I wrote, detection is by tool calls and fixed strings, and there is no complete defence against prompt injection. What actually bounds the damage is structural: no destructive tools, draft-only tickets and a step limit. Observability without a platform Every request writes one JSON log line with a request id (also returned in a response header), status, latency, user, tools used, token usage and whether the question was refused. For an agent turn the tokens are summed over every model call. The question text is not logged, only its length and a short hash, and tool arguments are not logged either. A protected /metrics endpoint shows counts, average and p95 latency, tokens and refusal rate. Metrics are in memory, so they reset on restart. Shipping it for $0 The Docker image is slim, runs as a non-root user and contains no documents, index or keys. My Mac runs macOS 12 and cannot run Docker Desktop, so CI builds the image and smoke-tests it: it starts the container with a tiny fake index and checks that /health answers and that /ask returns 401 when no users exist. The public demo runs on Render’s free instance, which needs no card, sleeps after 15 minutes and has an ephemeral disk. Three things bit me: New services default to a newer Python than the one I had tested, so I pinned 3.13 with a .python-version file. The index is built during the deploy, so waking up never spends embedding calls. I pasted the wrong text into the secret file that holds the users, and the server crashed with an opaque traceback. The clipboard still held an old terminal command. The app now logs the file path, its size and its first character (never the content), starts with no users, and rejects every protected request until it is fixed. Startup on the free 0.1-CPU instance takes about 26 seconds. Limits, and what I would do next • The corpus is tiny, the status API is simulated, and the three datasets are small and self-written. • There is no answer-faithfulness metric yet, only whether the model refuses. • Conversation memory, rate limits and metrics live in one process. • Real use would need HTTPS, key rotation and proper authentication, not API keys in a file. • Next: a faithfulness check, a bigger corpus with real PDFs, a vector database, human confirmation before write actions such as tickets, and a short-lived deployment on AWS. If you take one thing from this: the model was the easy part. The value was in measuring it, breaking it on purpose, and writing down what I could not prove.