How do you run AI agent evals in parallel on cloud GPUs?

Give every eval episode its own isolated sandbox, cap what each episode may spend, cap the whole batch, and let a separate grader score the answers. On Nodus that is one AgentGroup with a concurrency limit, a per-run model cap and a group cap, grading a fixed, versioned task set in sandboxes with no network. Parallelism shortens the wall clock but not the bill, since the same episodes use the same sandbox-minutes either way. Most agent evals do not need a GPU at all. The agent thinks on a hosted model and runs commands and tests on a CPU. You need a GPU only when the task executes GPU code, or when you serve the model under test yourself. What does a parallel agent eval need? A parallel eval needs four things: isolation per episode, a fixed task set, a grader the agent cannot reach, and limits that stop spend before it happens. Isolation keeps one episode's mistakes out of the next, and a separate grader keeps the agent away from the answer key. Nodus covers each piece with a documented object. An Agent is a definition: a system prompt, the inference source and a cost cap. Each AgentRun is one conversation in its own sandbox. An AgentGroup runs many runs of one agent at once, with maxActive between 1 and 1,000 (default 10) and one maxCostUSD for the group. An Environment is a versioned set of tasks with a grader, and the same name, version and seed always give the same tasks in the same order. Grading is the part to get right. In a Nodus Environment, tasks carry a prompt and non-secret metadata only, and expected answers stay with the grader. Program graders run in grading sandboxes with no network access, and code from a completion runs as a separate user that cannot read the answers. Every verdict is Correct, Incorrect, InvalidOutput or InfrastructureFailure, and an infrastructure failure does not count toward the pass rate. How do you launch the episodes and gather graded results on Nodus? Define the agent with a per-run cap, then create an evaluation group that names an Environment at an explicit version. This is the example worker agent from the Agents guide, applied with nodus apply -f agent.yaml: apiVersion: nodus.dev/v1 kind: Agent metadata: name: parallel-agents-worker spec: system: | Follow the task's requested answer format; otherwise answer in one short sentence. # Nodus's Claude, limited to Haiku so each run stays far below its cap. model: access: Nodus families: [haiku] maxTokens: 1024 perRunMaxCostUSD: "0.10" The evaluation itself, copied from the Evaluate an agent section of the same guide, runs two held-out tasks from a catalog Environment, two at a time: """Evaluate the deployed example agent on a fixed, versioned task batch.""" import nodus def main() -> None: # Model caps exclude Sandbox compute; set a project Budget before running this example. evaluation = nodus.AgentGroup.create( "parallel-agents-eval", agent="parallel-agents-worker", max_active=2, max_cost="0.20", evaluation={ "environment": "nodus/arithmetic-v2@2.0.0", "split": "test", "tasks": 2, "repetitions": 1, "seed": 42, "timeout": "10m", }, ) try: status = evaluation.wait(timeout=660) results = evaluation.results() print( f"{status.phase}: cases={len(results)}, pending={status.evaluation.pending}" ) print("pass rate:", status.evaluation.get("passRate")) print("mean reward:", status.evaluation.get("meanReward")) for result in results: print(result.task_id, result.state, result.get("verdict")) finally: evaluation.delete() if __name__ == "__main__": main() Nodus generates the task batch, runs it through ordinary AgentRuns, and grades completed answers with the Environment's isolated grader. There is no submit or seal step. A batch holds at most 100 cases including repetitions, and every member pins the same Agent revision. Results record the Environment version, image digest, split and seed, so you can rerun the same eval later. Agent evaluations are in Beta. Before you build on this example, confirm that your installed SDK version accepts evaluation= and group.results(). The console's Results view shows the same scored cases, pass rate and pending cases for each group. Plain parallel groups with submit_many and agent.map are documented on the Python Agents page if you grade answers yourself. A failed run, missing answer or grader failure stays unscored instead of becoming a zero. That is the right call, and it is why pass rate should be read next to coverage, as we argue in Can an agent's pass rate rise while it completes fewer tasks?. How do the per-run cap and the batch cap work together? The per-run cap bounds one episode, and the group cap bounds the batch. perRunMaxCostUSD (default $1.00) is checked before every model call. When the next call would not fit, the run waits with MaxCostReached and makes no more calls. A runaway episode therefore stops at its cap instead of eating the batch. The group cap is stricter than it looks. A run starts only when its own cap fits under what is left: the group cap, minus what the group has spent, minus the caps of runs still going. That gives a simple sizing rule we use: set the group cap to at least the number of tasks times the per-run cap, and every case is guaranteed room to start. Set it lower and you bound spend, but some cases may wait with MaxCostReached and end the batch unscored. Both caps cover model usage only. The sandbox each run works in, and the grading sandboxes, are billed on their own. For one ceiling over compute and managed inference together, the Agents guide recommends a project-scoped Budget with period: Total and action: Block. This is the Budget example from the docs; change period to Total for an eval campaign: apiVersion: nodus.dev/v1 kind: Budget metadata: name: research-monthly spec: limitUSD: "500.00" period: Monthly # a UTC calendar month; Total counts from creation scope: project: research # leave out for the whole org selector: matchLabels: {team: nlp} action: Block # Notify only sends notices thresholds: [50, 80, 100] notify: emails: [research-leads@example.com] With Block, new work in scope is refused with 402 BudgetExceeded and running work stops gracefully. We cover the same pattern for coding agents in How do you cap the GPU spending of an AI coding agent?. What does a 100-episode eval cost, and does parallelism change it? Price the sandbox time and the model ceiling separately, because they behave differently. Take one full batch of 100 cases, each running about 10 minutes. That is 1,000 sandbox-minutes, or 16.67 sandbox-hours, whatever max_active you pick. Where each episode runs Rate on the Nodus pricing page 100 episodes x 10 min CPU, 1 vCPU and 1 GiB $0.0075/hr $0.13 NVIDIA L4 GPU $0.430/hr $7.17 NVIDIA H100 SXM GPU $2.600/hr $43.33 Model ceiling, 100 runs x $0.10 cap per-run cap $10.00 at most Prices checked November 7, 2026, on the Nodus pricing page; GPU rates are listed as hourly supply prices. The sum is 16.67 hours times the hourly rate, before any minimum charge per sandbox, which the Sandboxes dry-run estimate shows. New agent sandboxes start at one vCPU and 1 GiB under the Agents guide's Auto setting. On a CPU sandbox the model is the bill; on an H100 the GPU is. The model cap is a ceiling, not a forecast. Your real model spend is whatever the runs use, shown in status.cost. The lesson from the table is still clear: for a CPU-only agent eval, the sandbox is a rounding error and the model is the bill. Put an H100 under every episode and the GPU costs more than four times the model ceiling. Parallelism moves the wall clock. With 100 ten-minute episodes, max_active of 1 takes 1,000 minutes, 10 takes 100 minutes, 25 takes 40, 50 takes 20, and 100 takes 10, ignoring startup and queueing. The sandbox-minutes stay at 1,000 in every case. Same 1,000 sandbox-minutes in every bar; only the waiting changes. Your model provider's rate limits can become the real ceiling on parallelism. If you bring your own key, Nodus charges nothing for the model, and your provider's rate and spending limits apply. When does an agent eval need a GPU sandbox? Use the cheapest place that can actually run the task. This is the decision rule we apply: The agent calls a hosted model and the task runs on CPU (tests, scripts, file edits, API calls): run it as an AgentGroup on CPU sandboxes. Model tokens dominate, so cap them tightly. The task executes GPU code, such as a CUDA kernel, a small training run or a benchmark: give that step a GPU. GPU Sandboxes on Nodus run on dedicated on-demand machines, never Spot, and the agent evals changelog entry describes running a harness either in an agent sandbox or as a GPU workload for batch evaluation. You serve the model under test yourself: run the harness as a GPU Job, or as a Sweep when you want the same eval across GPU types or shards. A Sweep shares one maxCostUSD across its cells, runs maxParallel at once and supports up to 64 repetitions. GPU sandboxes keep billing while idle, so give each one a short idleTimeout, a maxLifetime and its own maxCostUSD. The Sandboxes guide's quick start sets all three. From Python, nodus.Sandbox.create takes the same limits (Python Sandboxes). For the GPU sandbox options themselves, see Which sandboxes give AI agents GPUs?. How many episodes and repeats do you need for a trustworthy score? Size the batch by the error bar you can accept, then prefer new tasks over repeats. For a pass rate, the Central Limit Theorem standard error in Adding Error Bars to Evals (Miller, 2024) is the square root of p(1 - p) / n for binary scores. At a 50% pass rate over 100 independent tasks, the 95% interval is 1.96 x 0.05, or plus or minus 9.8 points. Quadrupling to 400 tasks halves it to plus or minus 4.9. Repeats help less than new tasks. In the paper's worked example (binary scores, task difficulty uniform), K answers per task cut the variance to (1 + 2/K)/3 of a single answer. With one 100-case batch, that gives these trade-offs: Allocation of 100 cases Variance vs 100 x 1 Standard error vs 100 x 1 100 tasks x 1 answer 1.00 1.00 50 tasks x 2 answers 1.33 1.15 25 tasks x 4 answers 2.00 1.41 Spending cases on repeats instead of new tasks widens the error bar. The arithmetic: 50 x 2 gives (1/50) x (2/3) = 1/75 against 1/100, a ratio of 1.33. 25 x 4 gives (1/25) x (1/2) = 1/50, a ratio of 2.00. The paper also warns that a pooled standard error across all repeated answers is inconsistent, so average each task's repeats first. Use repetitions when you have run out of distinct tasks, or when you want each task's own pass rate. Two more points from the same paper are worth applying. When comparing two agents, analyze per-task paired differences rather than two headline numbers. And do not lower the sampling temperature just to make the score less noisy. Where do Inspect AI and the SWE-bench harness fit? Keep the harness you have if it already works. The question is where it runs. Inspect AI runs samples in parallel and limits them with max_sandboxes, which for its Docker provider defaults to twice the processors available. The SWE-bench harness recommends at least 120 GB of free storage, 16 GB of RAM and 8 CPU cores, and fewer than min(0.75 * os.cpu_count(), 24) for --max_workers. Those limits come from one machine. On a large workstation, a local Docker run is the simplest choice, and you should use it. Move to per-episode cloud sandboxes when one machine's cores cap your concurrency, or when you need hard spending caps and a grader isolated from the agent. If you compare sandbox providers, read their billing rules. Modal, for example, bills Sandboxes at non-preemptible rates of 3x list price, $0.00003942 per physical core per second for CPU. FAQ Does a higher max_active make an eval more expensive? Not for the same episodes: each still gets one sandbox, the same minutes and the same model usage. What changes is how hard you push your model provider's rate limits. What happens when one episode hits its cost cap? It waits with MaxCostReached and makes no more model calls. The cap is copied when the run is created, so to continue you cancel it and start a new run with a higher cap. Can the agent read the expected answers? Not in a Nodus Environment. Only task prompts reach the agent, and program graders run as a separate user in a sandbox without network access. How long can one evaluation batch take? The timeout defaults to 30 minutes and can be set from 1 minute to 24 hours. It includes preparation, queueing and funding waits, and answers that could not be graded by then are reported as evaluation timeouts, outside the pass rate. Should I report one number per eval? Report the pass rate with its standard error, plus how many cases were scored. A bare percentage hides both the noise and the missing cases. Originally published on the Nodus blog.