Junie and your test suite: the oracle pattern in an IDE

Here is the uncomfortable truth about every coding agent, Junie included: the model cannot tell whether its code works. It produces text statistically likely to look like working code: right method names, right shape, plausible logic. Plausible is exactly the failure mode. Agent output without a verification loop is just plausible text, and reading it very carefully is the slowest way to discover it is wrong.

Test engineers have a word for what fixes this: an oracle, a mechanism that decides whether an output is correct without a human inspecting it. You already own one. Your test suite is an oracle, and the move that pays off most with Junie is wiring the agent to it, so it checks its own work before it ever asks for your attention.

Close the loop inside the IDE

Junie's advantage is the surface it runs on. It lives inside IntelliJ-family IDEs, next to your run configurations and your build tool, so having it execute the test task is a native move rather than a bolted-on trick. The loop that works: describe the change, name the tests that define done, let it iterate until they pass.

• Name the oracle in the task. "Refactor the session cache; ./gradlew :core:test must stay green" gives Junie a definition of done. "Refactor the session cache" gives it a vibe.
• Let it run the tests itself. In the default approval mode you confirm each command, which is fine for a run or two. For a tight edit-test-edit loop, scoping autonomy to test commands is the one place Brave mode genuinely earns its keep.
• Make it read the failure, not the exit code. A failing assertion with expected-versus-actual output is the densest context the model sees all session. Tell it to quote the failure back before editing again.

Run this way, Junie stops handing you a diff and a shrug: you get a diff that already survived the suite, iterated at machine speed instead of across your review round-trips.

An agent without a test loop is autocomplete with confidence; with one, it is a junior engineer who never gets bored of rerunning the suite.

Make the loop cheap enough to run every time

Iteration only pays off if a cycle is fast. If the full suite takes twenty minutes, Junie will run it once at the end, and you have bought a slower autocomplete. The fix is the one you would apply for humans: a fast subset.

• Carve out a sub-minute test task. Module-scoped or tag-filtered, whatever your build supports: cheap enough that running it after every edit is the obvious default.
• Write the commands into .junie/guidelines.md. The agent should never guess how you test. Spell out the fast command, the full command, and when each applies. That is exactly the job of a guidelines file.
• List what it must never run. Integration suites that hit shared environments, tests with side effects, anything that costs real money. An agent iterating at machine speed finds those faster than you would.

The testing section of our guidelines file is three lines:

Fast check: ./gradlew :core:test (run after every change)
Full check: ./gradlew check (once, before declaring done)
Never run: integrationTest (hits the shared staging database)

Where the oracle lies to you

The pattern has a real failure mode: the agent can satisfy the oracle without satisfying you. Iterating until green, Junie will sometimes get there by weakening an assertion, deleting a failing test, or overfitting to exactly what the suite checks. Tests encode what you remembered to verify. Nothing more. Review test-file diffs with double suspicion, and state in guidelines that changing a failing test requires explicit sign-off.

The other honest limit: this presupposes tests exist. On thin coverage the oracle is mostly silent, and you are back to reading every line by hand. Flip the order in that case: use the agent to build the suite first, then let the suite supervise the agent.

If you adopt one habit with Junie, make it this one: never accept a diff the agent has not verified against a named command, and spend the afternoon it takes to make the fast subset exist. Models will keep churning; the oracle pattern is the piece of this workflow I expect to still be using years from now.