How to Generate Karate API Tests from OpenAPI — Without Creating a Maintenance Nightmare

How to Generate Karate API Tests from OpenAPI — Without Creating a Maintenance Nightmare Generating API tests from an OpenAPI file sounds like an easy automation win. Parse the specification. Create one test per endpoint. Add a few assertions. Commit the generated files. That approach works surprisingly well in a demo — and often becomes painful as soon as the API or test suite starts evolving. The hard problem is not producing Karate syntax. The hard problem is producing tests that fit the way your project already works and remain useful after the first generation run. The tempting approach Imagine an API contract containing: paths: /orders: post: operationId: createOrder /orders/{id}: get: operationId: getOrder A generator can easily produce something like: Feature: Orders API Scenario: Create an order Given url baseUrl And path 'orders' And request { productId: 123, quantity: 1 } When method post Then status 201 Scenario: Get an order Given url baseUrl And path 'orders', 123 When method get Then status 200 It is valid Karate. It may even pass. But several important questions are still unanswered: Where did baseUrl come from? Does the project already have authentication helpers? Is productId: 123 valid test data? What should the response schema contain? What happens for an invalid product? What happens when quantity is missing? Is the endpoint already covered by another scenario? Does the team have naming, tagging or reusable-feature conventions? A generated test that ignores those questions creates code, but not necessarily useful coverage. Treat OpenAPI as a contract, not a test suite OpenAPI tells us a lot about the API surface: paths and HTTP methods parameters request schemas response schemas required fields enums and formats documented response codes security schemes That information is excellent input for test generation. But the specification usually does not contain all the information needed for an executable test suite. Environment configuration, reusable authentication, realistic data creation and project conventions typically live elsewhere. A good generator therefore needs two kinds of context: Contract context — what the API says it accepts and returns. Repository context — how this project already executes Karate tests. Ignoring the second is where many generated suites become difficult to maintain. Generate deterministic coverage first I prefer to begin with rules that do not need an LLM. For each operation, deterministic generation can derive candidate scenarios from the contract. For example, a required field gives us at least two obvious cases: quantity: type: integer minimum: 1 Positive case: And request { productId: 123, quantity: 1 } When method post Then status 201 Boundary or negative candidates can be derived from the same schema: And request { productId: 123, quantity: 0 } When method post Then status 400 and: And request { productId: 123 } When method post Then status 400 This has an important advantage: the reasoning is explainable. We know exactly why each scenario exists. AI can still improve the suite later, but it should not be the only mechanism deciding what needs to be tested. Reuse the project before inventing configuration Before generating a new feature, inspect the repository. A Karate project may already contain: src/test/java/ karate-config.js auth/ token.feature common/ create-user.feature orders/ orders.feature If authentication already comes from karate-config.js, generated tests should use it. If the team creates test users through a reusable feature, the generator should reuse that pattern instead of inventing credentials. If scenarios consistently use tags such as: @orders @smoke new tests should follow the same convention. Repository awareness is the difference between generating a standalone example and adding a maintainable test to an existing suite. Detect coverage before generating more tests Another mistake is assuming every OpenAPI operation needs a new feature file. First compare the API contract with the existing Karate scenarios. Think of the API surface as a matrix: POST /orders covered GET /orders/{id} covered DELETE /orders/{id} missing PATCH /orders/{id} missing Now generation becomes targeted. Instead of producing another complete test suite, generate tests only for the gaps. That makes the workflow: OpenAPI ↓ Existing Karate suite ↓ Coverage mapping ↓ Missing operations ↓ Generate only what is missing This is much more useful in a real repository than repeatedly regenerating everything. Use AI as an enhancement layer There are places where AI is genuinely useful. For example: understanding domain intent from descriptions suggesting edge cases that are not encoded in the schema adapting generated scenarios to an existing style explaining a failed test proposing a repair after an API change But those workflows should sit on top of deterministic evidence. A useful architecture looks like this: OpenAPI specification ↓ Deterministic parser ↓ Coverage + schema rules ↓ Repository conventions ↓ Candidate Karate scenarios ↓ Optional AI enhancement ↓ Validation / review The key word is optional. You should still be able to generate, execute and analyse tests when an AI provider is unavailable. Validate what was generated Generation is not finished when a .feature file exists. At minimum, generated output should be checked for: valid Karate syntax valid referenced configuration duplicate scenarios unresolved variables incorrect paths or HTTP methods missing required request data assertions that are too weak The last item deserves particular attention. This test can pass while proving very little: When method get Then status 200 Where the contract supports it, stronger assertions might verify: Then status 200 And match response.id == '#number' And match response.status == '#string' And match response.items == '#array' The goal is not to maximize assertion count. It is to make sure the test can detect the failures it was created to protect against. This is the workflow I built into Karate Test Management I originally built Karate Test Management, an open-source VS Code extension, around test generation. As the project grew, the interesting problem became everything around generation. The current workflow can: generate Karate tests from OpenAPI import Postman, HAR and GraphQL sources index an existing Karate test library execute features and individual scenarios compare OpenAPI operations with existing test coverage identify missing tests track quality findings analyse failures and flakiness optionally use AI for generation and repair The important architectural decision is that deterministic generation, execution and coverage analysis do not depend on AI. AI is there to enhance test-engineering work, not to replace the evidence underneath it. A better mental model Instead of thinking: OpenAPI → generated test files I find this model more useful: Discover ↓ Understand the contract and repository ↓ Measure existing coverage ↓ Generate missing scenarios ↓ Execute ↓ Analyse evidence ↓ Maintain That turns test generation from a one-off code-generation trick into part of a sustainable API testing workflow. If you use Karate and want to experiment with this approach, the project is open source: GitHub: https://github.com/mov2day/KaratePlugin VS Code Marketplace: https://marketplace.visualstudio.com/items?itemName=MuthuKumarKoodalingam.karate-test-generator The next article in this series will look at a related problem: your Karate tests may all be green while large parts of your API are still untested.