01Definitions
Two things people call AI refactoring
Some products combine both. Read what a tool does under the hood before you decide how much to trust its output.
| Deterministic recipes | LLM-based refactoring | |
|---|---|---|
| How it works | Parses code to a tree and applies explicit rules | A model reads code and generates new code |
| Repeatable | Yes, same input gives the same diff | No, output can differ between runs |
| Strength | Mechanical changes across many files | Cases with no rule: odd APIs, glue code, local conventions |
| Typical failure | Does nothing when no recipe matches | Plausible but wrong code, invented APIs, silent behavior change |
| Example | OpenRewrite | Tools such as AWS Transform and GitHub Copilot app modernization use models, among other techniques |
02Failure modes
Where recipes fail
The good news is that these failures are loud. The build breaks or nothing changes, and you see it.
- No recipe exists for your library, your in-house framework or your pattern.
- The recipe is correct for the common case and wrong for yours, for example a customized subclass.
- It changes code but not the things around it: CI images, container bases, runtime flags.
- A recipe stops partway, leaving code that does not compile.
03Failure modes
Where LLM refactoring fails
These failures are quiet. That is why model output needs stronger gates than a recipe diff does.
- Code that compiles and looks right but changes behavior, such as rounding, time zones or null handling.
- APIs that do not exist in the library version you use.
- Edits outside the scope you asked for, or tests changed to pass.
- Different answers on a second run, so a result cannot be reproduced for audit.
- Large codebases that do not fit the model's context, so it sees part of the picture.
04Decision rule
Which one should run first?
- 01
Is there a recipe for it?
If yes, run the recipe, for example
org.openrewrite.java.migrate.UpgradeToJava21. It is cheaper to review and repeatable. - 02
Is the change mechanical but unmatched?
Consider writing a custom recipe if it repeats across many files. Writing it once beats reviewing the same model edit many times.
- 03
Is the change judgment-heavy and local?
Use an LLM tool, in small slices, with the recipe output already committed.
- 04
Is the risk high?
Payment, security or regulated logic: keep an engineer in the loop at every step, whichever tool wrote the code.
05Review gates
The gates that apply to any AI-written change
These gates match the Ace8 method: test before you touch, and AI drafts while engineers decide.
- Tests first: characterization tests written before the change, so behavior is pinned.
- Small diffs: one slice per change, so a reviewer can read it.
- Compile and static analysis on every change.
- A named engineer approves every merge. The tool does not.
- Record what tool, version and prompt produced each change.
- Keep each step reversible, one commit per step.
06Evidence
What we are measuring
We are running OpenRewrite, AWS Transform and GitHub Copilot app modernization on the same open-source Java 8, Spring Boot 2 and javax application. For each we will log files changed, compile and test results before and after, manual fixes needed, and engineer review minutes. Results will be published on the AI modernization tools benchmark page. This page states no results yet.
07Next step
How Ace8 helps
Ace8 is the AI-assisted application modernization practice of AceMQ. We use recipes first, AI tools on the remainder, and engineers on every merge. See AI-assisted modernization for the full approach, or start with the assessment to find which applications fit.