Comparing Planning Models for an AI Agent for Test Automation: Qwen, Gemma, MiniMax and Luna

Every AI agent for test automation runs on a model. Your choice of model affects the quality of your tests and the price. If you choose the wrong model, you pay more for worse results. We tested four models in Explorbot’s most important slot. It is the part that plans the tests. In the end, the cheapest tokens cost the most. Explorbot is an open-source AI agent for test automation. You point it at a URL. It reads the page, works out what the application does, invents its own test scenarios, and runs them in a real browser. When something breaks, it reports the problem with steps to reproduce. It uses three model slots, because the three jobs are very different:

• The model slot reads pages. Every step sends it cleaned HTML and an ARIA tree, so the token volume here dwarfs everything else. It has to be cheap and fast.
• The agenticModel slot decides. It writes the test plan and it supervises execution. It never sees raw HTML, only a short summary and the last few actions. Volume is low. We buy quality here rather than price.
• The visionModel slot reads screenshots. Volume is low here too.

The config we run:

ai: {
  model: groq('openai/gpt-oss-20b'),
  agenticModel: openrouter('gpt-5.6-luna'),
  visionModel: openrouter('gpt-5.6-luna'),
}

We are happy with this. It costs cents rather than dollars, and a session finishes in minutes.

Last time we tested models for the reading slot and found that cheaper tokens produced more expensive sessions. This time we tested the planning slot. Its output is a few thousand tokens, and it determines every browser action that follows.

What Models We Tested

We tested four models across five routes. Prices are per million tokens, checked on 7 August 2026.

• GPT-5.6 Luna is our current pick and the baseline.
• Gemma 4 31b is what models.json recommends for OpenRouter.
• Qwen 3.6 27b is what it recommends for Groq.
• MiniMax M2.7 is a reasoning model. We put it on two routes to see what the route alone costs. MiniMax was also meant to run on Groq. Groq does not serve it, so that variant never happened.

What We Ran

A login form is a bad test of a planner. Everyone writes the same five scenarios for it. So we pointed Explorbot at a page with real CRUD behaviour, the test runs list of a local Testomat.io instance. A real Runs page gives a planner something to chew on: manual runs to create, run groups to build, status filters, a status report, pagination, and an archive behind a menu.

explorbot explore /projects/codeceptjs/runs --max-tests 2

Only the agenticModel changed between runs. The reading model stayed on GPT-OSS 20b through Groq. Screenshots stayed on Luna through OpenRouter. Every variant started from the same pre-authenticated browser session, and every variant ran twice.

Here is the plan Luna wrote after researching the page:

1. [critical]  Create a new manual test run and verify it is added to the runs list.
2. [important] Create a new run group and verify the group is displayed on the Runs page.
3. [high]      Apply a status filter from the Filters modal and verify the runs list changes.
4. [important] Open the Runs Status Report and verify it presents a status overview.
5. [high]      Search for a run title that does not match any run and verify the no-results state.
6. [normal]    Select the Manual status tab and verify the list displays manual runs.
7. [normal]    Navigate to the next pagination page and verify a different page is displayed.
8. [low]       Use the More options menu to open the Runs Archive and verify archived runs appear.

It then executed the top two.

Structured Output Decides Who is Eligible

The planner asks for JSON matching a fixed schema. So does the supervisor when it rules on a test. A model that cannot return that JSON has no seat, however good its reasoning is. We tested both real schemas in isolation before running a single session:

Qwen does not fail on Groq. It is refused:

This model does not support response format `json_schema`.

The control run on the same API key proves this is per-model rather than per-account. Plain JSON mode does not rescue it. Groq then requires the word “json” somewhere in the messages, and our prompts do not guarantee it.

Run a session anyway and it dies before it starts. Explorbot tried three planning styles, twice each, and collected six identical refusals. No tests ran. It had already spent 180K reading tokens researching a page nobody would go on to test.

This reaches beyond our benchmark. models.json ships qwen/qwen3.6-27b as the recommended Groq model for this slot. Anyone starting from that default gets a planner that cannot plan.

Gemma is the one soft entry in the table. It plans reliably, and on the supervisor schema it sometimes returns an object it never finishes. Explorbot times those calls out and retries. They did not appear in the sessions at all, so it stays in the comparison.

The Four Models That Worked

Every remaining model planned a sensible suite. All four found the same feature surface: create a manual run, create a run group, filter by status tab, paginate, open the status report. The plans contained six to eight scenarios, and each session executed the top two.

Read the pass column carefully. Each model wrote the scenarios it was then graded on, and it also acted as the judge. Luna failed two scenarios that no other model attempted. One of them required creating a disposable run group, pinning it from its action menu, and verifying the pinned state. A harder plan produces a lower pass rate.

The judging side is partly constrained. A pass verdict must name a claim, which Explorbot turns into CodeceptJS assertions and runs on the reading model. That model was identical in every variant. The confound is still real, and we would not rank on this column alone.

We found a second one while writing this up. Explorbot keeps notes about pages it has already visited, and the flag we used to suppress that only suppresses part of it. So models that ran later in each round saw a little more accumulated experience than the ones that ran first. The page already had a large base of notes from earlier work. The results also do not improve in run order, so we do not think this moved much. We are telling you because it is the kind of thing that quietly flatters whichever model you test last.

What a Session Costs

The planning slot is small. It made nine to twenty calls per session and stayed under 125K tokens. These are the second-round numbers, where every model’s calls could be separated cleanly from the screenshot slot.

MiniMax spends most of its output on reasoning tokens. They cost the full output price. That is how the smallest token here still costs the most.

What Caching is Worth

An agent re-sends its conversation on every call. Most of your input is a repeat. Providers bill repeats at a separate rate. Two things then set your real price. One is the size of the discount. The other is the share of input that reaches it.

Luna is the only model here that wins on both. It pays roughly a third of its list rate. The others each fail on one side. Gemma reaches the cache often. OpenRouter then bills those tokens at the fresh rate anyway. MiniMax has a deep discount and never reaches it, because we routed it through :nitro. That suffix picks a new upstream for every call. The next machine has never seen your prefix.

This is what separates Luna from Gemma. They list at the same input price. They do not pay the same input price. The reading slot has more room than this one. Groq halves cached input on GPT-OSS 20b. Our hit rates there stayed low all session. Check that line before you commit to a provider.

The Full Cost of the Session

Averaged over both rounds. The planning figure for Luna is round two alone. In round one it shared a model name with the screenshot slot, so the two could not be separated.

The planning model is a fraction of what a session costs. Everything else is the reading model working through the plan it was handed.

That reverses the ranking. Gemma has the cheapest published rates in the table on both input and output. It emits no reasoning tokens at all. It still produced the most expensive sessions of the four.

Three questions get three different answers:

1. Which model is cheapest to buy tokens from? Gemma.
2. Which is cheapest to call? Luna, because Gemma wastes every cache hit it gets.
3. Which is cheapest to run? Luna again. Gemma comes last, because its plans drove far more reading than any other model here.

Only the third question is your bill.

How to Choose the Right Model for an AI Agent for Test Automation?

The planning model is cheap on its own. But it controls your bill in an indirect way. It decides how much work the reading model does. So judge each planner by the full session cost. The price per token tells you very little here:

• Qwen 3.6 27b on Groq cannot fill this slot. Groq refuses the structured output the planner needs, and there is no fallback that works from inside Explorbot. The model may be excellent. It is simply not reachable for this job today. Since it is our shipped Groq default, that recommendation needs replacing.
• Gemma 4 31b is the cheapest to buy and the most expensive to run. It passed all four tests and it has the lowest published rates in the comparison. It also gets no discount on cached input, so nearly half its traffic is repeated tokens billed at full price. On top of that, its plans drove the most reading of any model here. That is the whole argument of this post in one model.
• MiniMax M2.7 works on both routes, and the route costs more than we expected. Same model and the same schema compliance either way. SambaNova charges nearly twice as much, mostly on its input rate. Take MiniMax through OpenRouter. Drop the :nitro suffix too, which bought us speed by throwing away the prompt cache.
• GPT-5.6 Luna stays our default. It had the cheapest slot, the cheapest session, and the fastest wall clock, with no structured-output failures across twelve isolated attempts and nineteen live calls. It also planned the hardest scenarios of the four and failed half of them. We read that as ambition rather than incompetence, and ambition is what we want from a planner in an exploratory tool.

Try Explorbot

npm i explorbot
npx explorbot init

import { createGroq } from '@ai-sdk/groq';
import { createOpenRouter } from '@openrouter/ai-sdk-provider';
const groq = createGroq({ apiKey: process.env.GROQ_API_KEY });
const openrouter = createOpenRouter({ apiKey: process.env.OPENROUTER_API_KEY });
ai: {
  model: groq('openai/gpt-oss-20b'),
  agenticModel: openrouter('gpt-5.6-luna'),
  visionModel: openrouter('gpt-5.6-luna'),
}

That is the configuration the numbers above describe. Going straight to OpenAI for the last two slots is one less key and the same model. Grab the source here, swap agenticModel to try one of the other planners, and leave everything else alone. The eight completed sessions behind this post cost about $0.96 in total. If you run a planner we did not test, tell us what the session cost. If you are ready to import your CodeceptJS or Playwright tests, run them, and track every result, group, and report in one dashboard, try Testomat.io for free.