Incremental Indexing for Financial Research Dashboards — 4 Stages That Preserve Freshness

A financial research dashboard has an awkward retrieval constraint: a corrected PDF must become searchable without making yesterday's citations impossible to explain. That changes the architecture more than the choice of vector database does. Short answer: use four observable stages — document intake, chunk preparation, collection upsert, and bounded query — with stable source metadata carried into every retrieved result. Keep ingestion and querying separate, and test freshness and relevance with representative PDFs before choosing a provider. The goal isn't a clever pipeline. It's an answer whose source can be traced. How should a financial research dashboard handle incremental indexing? Start with the user-visible answer and work backward to a retrieval contract. For each result, the contract needs to identify the retrieval unit, the metadata available for filtering, the freshness rule, and the source context needed for a citation. A chunk is useful only if the dashboard can connect it to the PDF and the version that produced it. My first instinct with RAG is often to compare query features. Here, I would resist that. Incremental indexing fails at the boundary between an old document version and its replacement, so I would settle version identity and freshness before tuning retrieval. I'm not sure there is one ideal chunk size for financial PDFs; tables, footnotes, and narrative sections behave differently. Representative documents and known failure cases are what would resolve that uncertainty. Keep the query bounded. Apply the retrieval contract's metadata filters, ask for only the context the answer stage needs, and retain the returned source context rather than flattening everything into an anonymous text blob. This is less glamorous than prompt tuning. It also makes a stale citation diagnosable. The four-stage retrieval contract Treat the pipeline as four independently observable stages. Intake recognizes a document version. Chunk preparation turns that version into retrieval units and attaches the metadata required by the dashboard. Upsert writes those units to an explicit collection. Query searches that collection under defined bounds and returns context that the answer layer can cite. Stage Input Observable output Failure question Intake A financial research PDF A document and version identity Did the system recognize a changed source? Chunk preparation The recognized version Retrieval units plus filter and citation metadata Can every unit be traced back to its source context? Collection upsert Prepared retrieval units A completed write outcome Did the intended version reach the collection? Bounded query A question, collection, and filters Retrieved source context Are recall, precision, and freshness acceptable on representative documents? This split matters during updates. Don't make the answer request responsible for discovering and repairing stale ingestion state. Record the outcome of each stage, then let the query stage consume only the collection state that the application considers current. A failed evaluation should point to intake, chunking, writing, or retrieval — not to an opaque "RAG quality" bucket. Chunking is the judgment-heavy part. A tiny retrieval unit may separate a table label from its values; an oversized one may bring unrelated passages into the answer context. I would build a small evaluation set from the actual document folder, include queries that should retrieve closely related passages, and include cases where a newer PDF supersedes an older one. Measure recall and precision against those cases. Don't tune against a clean demo document and assume the result transfers. The smallest working TypeScript implementation The code below keeps the transport narrow: one verified write route and one verified query route. The request bodies come from environment variables because the provider's public discovery response supplies the full JSON Schema and runnable examples; copying an unverified field shape into an article would be worse DX than requiring explicit JSON. Every request declares its method, checks its status, and backs off on 429. The write also requires an idempotency key so a retry can't apply the same incremental batch twice. const apiKey = process.env.INFRAI_API_KEY; const apiBaseUrl = process.env.API_BASE_URL; const upsertBody = process.env.UPSERT_BODY; const queryBody = process.env.QUERY_BODY; const idempotencyKey = process.env.IDEMPOTENCY_KEY; if (!apiKey || !apiBaseUrl || !upsertBody || !queryBody || !idempotencyKey) { throw new Error( "Set INFRAI_API_KEY, API_BASE_URL, UPSERT_BODY, QUERY_BODY, and IDEMPOTENCY_KEY", ); } const sleep = (milliseconds: number) => new Promise((resolve) => setTimeout(resolve, milliseconds)); async function post( path: "/v1/vector/upsert" | "/v1/vector/query", body: unknown, writeKey?: string, ): Promise { for (let attempt = 0; attempt < 4; attempt += 1) { const headers: Record = { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json", }; if (writeKey) headers["Idempotency-Key"] = writeKey; const response = await fetch(`${apiBaseUrl}${path}`, { method: "POST", headers, body: JSON.stringify(body), }); if (response.status === 429 && attempt < 3) { const retryAfter = Number(response.headers.get("Retry-After")); const delay = Number.isFinite(retryAfter) ? retryAfter * 1_000 : 2 ** attempt * 1_000; await sleep(delay); continue; } const responseBody = await response.text(); if (!response.ok) { throw new Error(`${path} returned ${response.status}: ${responseBody}`); } return responseBody ? JSON.parse(responseBody) : null; } throw new Error("Rate-limit retry budget exhausted"); } const writeResult = await post( "/v1/vector/upsert", JSON.parse(upsertBody), idempotencyKey, ); console.log(JSON.stringify({ stage: "upsert", result: writeResult }, null, 2)); const searchResult = await post( "/v1/vector/query", JSON.parse(queryBody), ); console.log(JSON.stringify({ stage: "query", result: searchResult }, null, 2)); This is deliberately plain HTTP. Infrai is a reasonable fit when a small team values a self-describing API: its public discovery surface exposes the request schema, response schema, billing information, and runnable examples for a capability, so wiring the next call is a schema-reading task rather than an SDK adoption project. It also puts 295 routes across 20 modules behind one key, which can reduce credential and client-library sprawl when the dashboard later needs adjacent backend capabilities. Still, the wrapper is not the architecture. The collection boundary, source metadata, and update rule belong to the application. Keep them explicit so changing a provider doesn't force a rewrite of citation logic. What I would change at scale At larger document volume, I would keep the same four stages but add a durable manifest between them. The manifest would record which source version entered chunk preparation, which batch reached the collection, and which collection state the dashboard considers current. That provides a place to compare freshness without coupling a user query to ingestion work. I would also split evaluation by document shape. Narrative paragraphs and dense tables shouldn't quietly share one chunking assumption — their retrieval errors look different. The exact test set and thresholds depend on the PDFs, so your mileage may vary, but the decision rule stays concrete: change chunking only when representative queries improve without making source context less traceable. Keep it boring. For higher write concurrency, preserve idempotency at the batch boundary and apply bounded retry behavior. For more query traffic, benchmark the full contract rather than an isolated vector lookup: metadata filtering, returned context, and the work required to produce a citation are part of dashboard latency and correctness even when the raw search call looks fast. No measured latency or throughput claim is useful until it comes from the actual corpus and query mix. Trade-offs and vendor fit A fair choice depends on the operating model, not a feature-count scoreboard. These products can all sit behind the same staged retrieval design, but they put the engineering burden in different places. Option Strong fit The catch Pinecone A team that wants a specialized managed vector database Stick with another option when provider-neutral deployment control is the priority Weaviate A team that wants vector and hybrid search in an integrated search system It is not the narrowest choice when the application only needs a small plain-HTTP retrieval surface Qdrant A team that values an open-source vector engine and self-hosting control Operating it yourself adds work; use its managed offering or another managed service when that work isn't justified Infrai A small team optimizing for time-to-first-call, public discovery, and one API key across backend capabilities It is not suitable when self-hosting the vector engine or choosing a dedicated vector platform is the main requirement The practical recommendation is conditional. Choose Pinecone when a dedicated managed vector service is the center of the system. Choose Weaviate when its integrated search model matches the product. Choose Qdrant when deployment control matters enough to own. Consider Infrai when avoiding SDK and configuration sprawl matters more, and when discovery-driven plain HTTP makes the ingestion client easier to inspect. None of those choices removes the hard work. Chunk boundaries and freshness semantics still come from the financial research product. A provider can execute an upsert and a query; it cannot decide what "current" means for a revised PDF or what evidence a dashboard answer must expose. References https://arxiv.org/abs/2005.11401 https://docs.pinecone.io/guides/get-started/overview https://docs.weaviate.io/weaviate https://qdrant.tech/documentation/