Benchmark
One legacy Java repo, run through every tool, with the failures published.
The method, the scoring and the tool list are fixed before the run. Results are pending, and this page says so until they exist.
The short answer
This benchmark runs the same open-source Java 8, Spring Boot 2 and javax application through OpenRewrite, AWS Transform and GitHub Copilot app modernization, then scores each on changes made, build and test results, manual fixes and review time. The run is not complete. No results are published yet.
Status
Where does the benchmark stand today?
The method and scoring table below are fixed. The runs are not done, so this page contains no results and no ranking. When the runs finish, we will publish the raw log for every tool, including the runs that failed, plus a CSV of the numbers.
We publish the method first for a reason: you can check that we did not choose the scoring after seeing who won. Ace8 is the AI-assisted application modernization practice of AceMQ, and we use these tools on client work, so we have an interest in the answer. That is why the method is written down in advance.
Input
What codebase do we test on?
One open-source application, chosen for its licence and for being typical: Java 8, Spring Boot 2.x, javax namespace, a relational database, and an existing test suite. Every tool gets an identical clean checkout. The target is Java 21 and Spring Boot 3, the same target as our Java upgrade and Spring Boot 3 migration pages.
One repository is a narrow sample. A tool that does well on it may do badly on code with heavy reflection, a proprietary framework or no tests. The published results will say this next to the numbers.
Tools
Which tools are in the test?
Moderne and vFunction are not in the first run. Moderne is a commercial platform built around OpenRewrite, and vFunction analyzes architecture rather than rewriting code. See Moderne vs OpenRewrite and vFunction alternatives for where they fit. Ace8 is not affiliated with AWS.
| Tool | Type | How we run it | Page |
|---|---|---|---|
| OpenRewrite | Rule-based recipes, open source | Maven plugin, Java 21 and Spring Boot 3 recipes | OpenRewrite |
| AWS Transform | AI service from AWS | Against the same repo, default settings first | AWS Transform |
| GitHub Copilot app modernization | AI agent in the IDE | Upgrade assessment and plan, then apply | Copilot app modernization |
Scoring
How is each run scored?
Every run is measured the same way. We record the numbers, not an overall grade, because the right tool depends on the job.
| Measure | How it is taken | Why it matters |
|---|---|---|
| Files and lines changed | Diff stat against the clean checkout | Sizes the review |
| Build result | Compile on Java 21 with Spring Boot 3, errors counted | Shows what the tool left undone |
| Tests before and after | Same suite, same command, pass and fail counts | Shows whether behavior held |
| Manual fixes | Count and category of edits we had to make | The real effort after the tool finishes |
| Engineer review minutes | Timed review of the diff to merge quality | The cost that tool pricing hides |
| Repeatability | Run twice from a clean checkout, diff the diffs | Rule-based tools should match; AI tools may not |
| Tool cost | Metered charge or licence, as billed to us | Stated with the date and settings |
| Failures and wrong changes | Every change that broke behavior or was reverted | Published in full |
Rules
What rules keep the comparison fair?
- Same checkout, same target versions, same test command for every tool.
- Default settings first. Any tuning is logged and the run is repeated untuned for comparison.
- No hand edits are counted as tool output. A manual fix is recorded as a manual fix.
- Tool versions and run dates are recorded, since all of these tools change quickly.
- A tool that fails to finish is reported as failed, with the error, not dropped from the table.
Results
What are the results?
Pending. Nothing on this page is a result. When the runs complete this section will hold the table above filled in for each tool, a short reading of it, and a link to the CSV. Until then, treat any claim of a winner, from us or from anyone else, as unsupported by this test.
Use
How should you read it when it is live?
Read review minutes and manual fixes before change counts. A tool that changes many files and leaves a long review is not cheaper than one that changes fewer and passes tests. Then check which of the sample's traits match your codebase. If your estate has little test coverage, the results matter less than the coverage does; see modernization assessment.
If you want the same run on your own code, that is what the assessment includes for a sample of your applications.
No honest answer exists without a test on your code. This benchmark is built to produce evidence for one sample repo. Results are pending, so we do not name a best tool.
Not yet. The method and scoring are fixed, and the runs are in progress. We will publish the raw log and a CSV, including failed runs.
It keeps the comparison controlled and repeatable. It is also a limit, which the results page will state next to the numbers.
No. Ace8 uses these tools on client work and is not neutral about the topic. We publish the method in advance and the failures after so you can judge the result yourself.
Yes. A sample run on your code is part of the modernization assessment.
Know the cost before anything moves.
A fixed-price assessment gives you the inventory, the risk score and a phased plan. Scope and cost are fixed before the migration starts.
Book a Fixed-Price AssessmentGet in touch
Talk to a modernization engineer
Tell us what you run today. An engineer, not a sales script, replies within one business day.