Automated RAG Index Refresh with EventBridge Scheduler: Keep Your Vector Store Fresh in Node.js 22

Stale embeddings silently sabotage your retrieval‑augmented generation answers, and most teams only notice after a user complaint. By wiring EventBridge Scheduler to a lightweight Node.js 22 service you can refresh your vector store on a strict timetable without manual intervention. This post shows exactly how to set it up and avoid the hidden pitfalls. Why RAG Staleness Is a Silent Killer RAG (retrieval‑augmented generation) is a pattern where a language model first retrieves relevant pieces of text from a vector store and then generates a response using those pieces. Embedding – a list of numbers that captures the meaning of a piece of text. Vector store – a database that holds embeddings and lets you quickly find the ones closest to a query vector. When the underlying documents change (e.g., a product spec is updated, a FAQ is edited, or a legal clause is revised) the old embeddings become stale. The retrieval step will still return the outdated chunk because the vector store hasn’t been refreshed. The model then generates an answer that looks confident but is factually wrong. In plain English: If your knowledge base is a library, stale embeddings are like dusty old books that never get replaced—readers keep quoting outdated information. A tiny demonstration // Imagine we have a simple in‑memory vector store for illustration type Vector = number[]; type Doc = { id: string; text: string; embedding: Vector }; let store: Doc[] = [ { id: "1", text: "The API returns 200 OK on success.", embedding: [0.12, 0.34, 0.56] // generated months ago } ]; // A user asks about the new status code (it changed to 201) const queryEmbedding = [0.12, 0.34, 0.56]; // same as old doc const nearest = store.reduce((best, doc) => { // naive cosine similarity placeholder const sim = doc.embedding.reduce((s, v, i) => s + v * queryEmbedding[i], 0); return sim > best.sim ? { doc, sim } : best; }, { doc: null as Doc | null, sim: -Infinity }); console.log("Returned doc:", nearest.doc?.text); // → "The API returns 200 OK on success." Even though the source document now says “201 Created”, the system still serves the old sentence because we never recomputed the embedding. Why refresh matters User trust erodes the moment a bot gives outdated facts. Compliance requirements often demand that published information be current. Search relevance drops as the gap widens between the store and reality. The fix is simple: periodically recompute embeddings for all source documents and overwrite the old vectors. The challenge is doing it reliably, on schedule, and without manual steps. Introducing EventBridge Scheduler for Automated Workflows EventBridge Scheduler is a fully managed service that fires an event (HTTP request, Lambda invocation, etc.) at a regular interval you define. Think of it as a digital alarm clock that can call any URL on a schedule you set. Key concepts: Schedule – the definition of “when” (e.g., rate(24 hours) or a cron expression). Target – the destination that receives the event (an HTTPS endpoint in our case). scheduleTimezone – the time zone the schedule interprets the expression in. The default time zone is UTC. If you forget to set scheduleTimezone to your local zone, a “midnight” job will actually run at 5 am EST, 10 pm PST, etc., causing missed refreshes and the illusion that the scheduler is broken. Tip: Always set scheduleTimezone explicitly, even if you are fine with UTC, so the intent is crystal clear. Rate vs. cron Rate expression (rate(24 hours)) is great for “every N units”. It guarantees a minimum 1‑second granularity and is easy to read. Cron expression (cron(0 2 * * ? *)) gives you fine‑grained control (specific hour, day of week). It’s more error‑prone, especially around daylight‑saving‑time (DST) transitions. For a daily refresh we’ll stick with the rate expression. Creating a schedule with the AWS SDK We will use the @aws-sdk/client-scheduler package, the official JavaScript/TypeScript client for EventBridge Scheduler. import { SchedulerClient, CreateScheduleCommand, DeleteScheduleCommand, } from "@aws-sdk/client-scheduler"; const client = new SchedulerClient({ region: "us-east-1" }); async function createDailyRefreshSchedule() { const cmd = new CreateScheduleCommand({ Name: "RagEmbeddingRefresh", // Rate of once per day ScheduleExpression: "rate(24 hours)", // Explicitly set to your local time zone; change as needed ScheduleExpressionTimezone: "America/New_York", // The target is an HTTPS endpoint we’ll deploy later Target: { Arn: "arn:aws:scheduler:::aws-sdk:apprunner:CreateService", // placeholder HttpParameters: { PathParameterValues: [], // not needed for static path HeaderParameters: { "Content-Type": "application/json", }, // Optional auth can be added here (e.g., OIDC token) }, // The URL the Scheduler will call // Using an App Runner HTTPS endpoint // Replace with your actual endpoint Uri: "https://my-rag-refresh.service.aws-region.amazonaws.com/refresh", }, // Optional description helps with ops visibility Description: "Refreshes embeddings for the RAG vector store each night", // Prevent accidental duplicate creation FlexibleTimeWindow: { Mode: "OFF" }, }); const response = await client.send(cmd); console.log("Schedule created:", response); } // Run once during deployment createDailyRefreshSchedule().catch(console.error); Gotchas to watch out for Gotcha Why it matters Time‑zone/DST edge cases A schedule created at 23:00 UTC may shift an hour forward/backward when DST changes, causing two runs in a row or a 23‑hour gap. Rate vs. cron confusion Choosing a cron expression for “every 24 hours” can accidentally create a 25‑hour gap on DST transitions. Minimum resolution of 1 second Scheduler cannot fire more often than once per second; sub‑second triggers need a different approach. Setting Up a Secure HTTP Target with Node.js 22 Our schedule will invoke an HTTPS endpoint hosted on AWS App Runner. App Runner automatically provisions a load balancer, TLS termination, and a container runtime, so we only need to ship a tiny Node.js 22 service. Why a lightweight HTTP service? Stateless – each request can recompute embeddings without relying on previous state. Scalable – App Runner can spin up more instances if a refresh takes longer than expected. Secure – TLS is handled for us, and we can add IAM‑based authentication later if needed. Minimal Express app (no build step) We will write the service in TypeScript but run it directly with the Node 22 flag --experimental-strip-types. This flag strips TypeScript type annotations at runtime, letting us avoid a separate compilation step. // src/server.ts import express, { Request, Response } from "express"; // Create an Express application const app = express(); app.use(express.json()); // parse JSON bodies // Health‑check endpoint (useful for monitoring) app.get("/health", (_req: Request, res: Response) => { res.json({ status: "ok", timestamp: new Date().toISOString() }); }); /** * Main refresh endpoint that EventBridge Scheduler calls. * It runs the embedding refresh logic (see next section) and returns a short status. */ app.post("/refresh", async (_req: Request, res: Response) => { try { // Import the refresh function lazily so the HTTP server starts fast const { refreshEmbeddings } = await import("./refresh"); await refreshEmbeddings(); res.json({ result: "success", refreshedAt: new Date().toISOString() }); } catch (err) { console.error("Refresh failed:", err); res.status(500).json({ result: "error", message: (err as Error).message }); } }); // Export the Express app for App Runner to invoke export default app; // If this file is executed directly (node --experimental-strip-types src/server.ts) // start the HTTP listener on the port App Runner expects. if (require.main === module) { const PORT = process.env.PORT ? parseInt(process.env.PORT) : 8080; app.listen(PORT, () => { console.log(`RAG refresh service listening on http://0.0.0.0:${PORT}`); }); } Running locally node --experimental-strip-types src/server.ts The flag removes all type syntax, so the file is plain JavaScript at runtime. This keeps the deployment package tiny (no node_modules/.bin/tsc or dist/ folder). Key point: Using --experimental-strip-types gives you the readability of TypeScript while still delivering a single‑file bundle that App Runner can run instantly. Writing the Embedding Refresh Logic in TypeScript (No Build Step) Now we flesh out refreshEmbeddings. The steps are: Fetch fresh source documents – could be S3 objects, a CMS API, or a database. Generate embeddings – we call an Amazon Bedrock model (e.g., amazon.titan-embed-text-v1). Batch write the new vectors to DynamoDB. Why batch writes? DynamoDB charges per write capacity unit (WCU). Using BatchWriteItem groups up to 25 items per request, reducing round‑trip overhead and staying within the service’s limits. The code – all in one file for simplicity // src/refresh.ts import { DynamoDBDocumentClient, BatchWriteCommand, BatchWriteCommandInput, } from "@aws-sdk/lib-dynamodb"; import { DynamoDBClient } from "@aws-sdk/client-dynamodb"; import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime"; import fetch from "node-fetch"; // built‑in in Node 22, but keep for clarity // ---------- Configuration ---------- const DDB_TABLE = process.env.DDB_TABLE ?? "RagEmbeddings"; const BEDROCK_MODEL_ID = "amazon.titan-embed-text-v1"; // example embedding model const SOURCE_DOCS_URL = process.env.SOURCE_DOCS_URL ?? "https://my-cms.example.com/docs.json"; // ---------- Clients ---------- const ddbClient = new DynamoDBClient({}); const ddbDocClient = DynamoDBDocumentClient.from(ddbClient); const bedrockClient = new BedrockRuntimeClient({}); // Helper: call Bedrock to get an embedding for a piece of text async function getEmbedding(text: string): Promise { const payload = JSON.stringify({ inputText: text }); const command = new InvokeModelCommand({ ModelId: BEDROCK_MODEL_ID, ContentType: "application/json", Accept: "application/json", Body: Buffer.from(payload), }); const response = await bedrockClient.send(command); const bodyString = Buffer.from(response.Body as Uint8Array).toString("utf-8"); const parsed = JSON.parse(bodyString); // Bedrock returns an array of floats under `embedding` return parsed.embedding as number[]; } // Helper: fetch the latest documents (JSON array of { id, text }) async function loadSourceDocuments(): Promise { const resp = await fetch(SOURCE_DOCS_URL); if (!resp.ok) { throw new Error(`Failed to fetch source docs: ${resp.statusText}`); } return (await resp.json()) as Array; } // Main function that ties everything together export async function refreshEmbeddings(): Promise { console.log("Starting embedding refresh…"); const docs = await loadSourceDocuments(); // Process documents in chunks of 25 (max batch size) const BATCH_SIZE = 25; for (let i = 0; i < docs.length; i += BATCH_SIZE) { const batch = docs.slice(i, i + BATCH_SIZE); const writeRequests = await Promise.all( batch.map(async (doc) => { const embedding = await getEmbedding(doc.text); // DynamoDB expects a map; we store the embedding as a list of numbers return { PutRequest: { Item: { pk: `DOC#${doc.id}`, // primary key (partition key) sk: "EMBEDDING", // sort key – allows future extensions text: doc.text, embedding, refreshedAt: new Date().toISOString(), // Optional TTL (time‑to‑live) attribute for automatic cleanup ttl: Math.floor(Date.now() / 1000) + 60 * 60 * 24 * 30, // 30 days }, }, }; }) ); const batchInput: BatchWriteCommandInput = { RequestItems: { [DDB_TABLE]: writeRequests, }, }; // DynamoDB may return unprocessed items; retry logic is essential at scale let attempts = 0; let unprocessed = writeRequests; while (unprocessed.length && attempts < 3) { const cmd = new BatchWriteCommand({ RequestItems: { [DDB_TABLE]: unprocessed }, }); const result = await ddbDocClient.send(cmd); unprocessed = result.UnprocessedItems?.[DDB_TABLE] ?? []; if (unprocessed.length) { attempts++; console.warn(`Retry ${attempts}: ${unprocessed.length} items unprocessed`); // Exponential back‑off await new Promise((r) => setTimeout(r, 1000 * attempts)); } } if (unprocessed.length) { console.error("Failed to write some items after retries:", unprocessed); // In a production system you might push these to a dead‑letter queue } else { console.log(`Batch ${i / BATCH_SIZE + 1} written successfully`); } } console.log("Embedding refresh completed"); } Why we batch and retry Hot partitions – writing many items with the same partition key can overwhelm a single physical partition. By sharding (pk: DOC#${doc.id}) we spread writes across many keys. BatchWriteItem limit – DynamoDB caps each batch at 25 items and a total payload of 16 MiB. Exceeding either triggers a ValidationException. Unprocessed items – under heavy load DynamoDB may return items it couldn’t write; a simple exponential back‑off retry loop resolves most cases. Tip: If your collection grows beyond a few thousand documents, consider adding a random hash prefix to the partition key (e.g., pk: \DOC#${hash(doc.id)}#${doc.id}``) to guarantee even distribution. Connecting Scheduler to DynamoDB: The Update Flow Now that we have a working HTTP endpoint and refresh logic, let’s describe the end‑to‑end flow triggered by EventBridge Scheduler. Scheduler fires – at the configured time it sends an HTTPS POST to /refresh. App Runner receives the request – the Express server routes it to refreshEmbeddings. refreshEmbeddings pulls fresh docs – from the source URL (could be S3, a headless CMS, etc.). Each document gets an embedding – via Bedrock’s InvokeModel. Embeddings are batch‑written – to DynamoDB using BatchWriteCommand. DynamoDB gotchas to keep in mind Gotcha Impact Mitigation Hot partitions still exist in 2025. A single partition key that receives a burst of writes can throttle the table. Requests slow down or get throttled. Use a sharded key pattern (DOC##). TransactWriteItems limit of 100 items – useful for atomic multi‑item updates, but not for bulk refresh. Attempting a transaction with >100 items throws an error. Stick with BatchWriteItem for bulk refresh; reserve transactions for critical single‑record updates. GSI eventually consistent reads – a Global Secondary Index (GSI) may not reflect the latest write instantly. A downstream query might miss the newest vectors for a few seconds. For the refresh path, consistency isn’t required; the retrieval path should use the primary index with strong consistency if you need immediate visibility. TTL deletion lag (up to 48 h) – items with a ttl attribute are removed asynchronously. Old vectors linger longer than expected, consuming space and WCU. Accept the lag or implement a manual cleanup job that runs after the TTL window. Pricing at scale – writes cost $0.25 per WCU per month. A daily full refresh can become expensive if you write many megabytes each time. Unexpected bill shock. Monitor WCU consumption; consider incremental updates (only changed docs) once the pipeline stabilizes. Example: Creating the DynamoDB table (once) `typescript import { DynamoDBClient, CreateTableCommand, } from "@aws-sdk/client-dynamodb"; const ddb = new DynamoDBClient({ region: "us-east-1" }); async function createRagTable() { const cmd = new CreateTableCommand({ TableName: DDB_TABLE, AttributeDefinitions: [ { AttributeName: "pk", AttributeType: "S" }, // partition key (string) { AttributeName: "sk", AttributeType: "S" }, // sort key (string) ], KeySchema: [ { AttributeName: "pk", KeyType: "HASH" }, { AttributeName: "sk", KeyType: "RANGE" }, ], BillingMode: "PAY_PER_REQUEST", // avoids manual WCU provisioning // Optional TTL attribute declaration TimeToLiveSpecification: { AttributeName: "ttl", Enabled: true, }, }); const result = await ddb.send(cmd); console.log("Table created:", result.TableDescription?.TableName); } // Run once during initial provisioning createRagTable().catch(console.error); ` In plain English: The table has two keys – pk groups all data for a single document, while sk lets us store different item types (e.g., the embedding vs. metadata) under the same document ID. The Takeaway Key recap: Automating your RAG embedding refresh removes the hidden risk of stale answers, and EventBridge Scheduler combined with a tiny Node.js 22 service gives you a reliable, server‑less solution. Stale embeddings cause wrong answers; a daily refresh keeps the vector store aligned with source data. EventBridge Scheduler is the glue that fires an HTTPS call on a strict timetable; always set scheduleTimezone to avoid DST surprises. Node.js 22 with --experimental-strip-types lets you write clean TypeScript without a separate build step, keeping the container image small. Embedding generation uses an Amazon Bedrock model; batch the calls if you have many documents to stay within rate limits. DynamoDB batch writes are efficient but require sharding to avoid hot partitions and retry logic for unprocessed items. Operational gotchas (time‑zone handling, hot partitions, TTL lag, pricing) are easy to miss but have simple mitigations. Final deployment checklist [ ] Create the DynamoDB table (pk, sk, TTL enabled). [ ] Store environment variables (DDB_TABLE, SOURCE_DOCS_URL, optional BEDROCK_MODEL_ID). [ ] Build a Docker image that runs node --experimental-strip-types src/server.ts. [ ] Deploy the image to AWS App Runner (or any container service that provides an HTTPS endpoint). [ ] Verify the /health endpoint returns { status: "ok" }. [ ] Run the /refresh endpoint manually once to confirm embeddings are written. [ ] Use the @aws-sdk/client-scheduler script to create the daily rate(24 hours) schedule, setting ScheduleExpressionTimezone to your local zone. [ ] Inspect CloudWatch logs for the first scheduled run; ensure no unprocessed items remain. [ ] Set up an alert on DynamoDB write throttling or on Scheduler failures. Add IAM permissions for the App Runner service role to call Bedrock and DynamoDB. With these steps in place, your RAG system will stay fresh, your users will receive up‑to‑date answers, and you’ll avoid the quiet nightmare of stale embeddings. Happy building! Transparency notice This article was written with the help of an AI system — Groq (GPT OSS 120B). Published: 2026-09-09 · Primary focus: Scheduler All code blocks are intended to be correct and runnable, but please verify them against the official docs for the tools mentioned before using in production. Find an error? Drop a comment — corrections are always welcome.