The question worth asking after 42 experiments
42 experiments is the number that made me pause.
It matters because the story is easy to misread. The interesting part is not “AI can do research by itself now.” The sharper question is smaller and more useful: when an old research idea comes back with an AI agent attached to it, what should a working person actually check before calling it progress?
My answer is this: Autoresearch is less a research replacement than a disciplined retry machine. That may sound like a downgrade, but I think it is the more practical reading.
I followed the loop, not the label
The linked news.hada.io brief describes an Autoresearch setup where an LLM agent repeatedly edits `train.py`, runs training, checks the result, commits what improves the metric, and rolls back what does not. The loop is simple enough to explain to a non-developer manager: give a junior analyst one spreadsheet, one scoring rule, and permission to try many small changes. Keep the useful versions. Throw away the rest.
That is not magic. It is process.
The setup had clear fences. Experiments ran in a container sandbox. Network access and arbitrary code execution were blocked. The agent could modify only selected files such as `train.py` and internal source material. Each experiment was kept short, around a few minutes, with a practical goal: improve the score without letting the system wander too far.
I like this framing because it puts automation where it belongs. Not in vague “AI will innovate” territory, but in a bounded loop: try, measure, keep, revert.
Where I got stuck is the same place many AI research demos get slippery. The word “research” stretches too far. If the agent mostly tunes parameters, fixes a constraint, and runs more trials than a tired human would, is that research? Or is it a better lab assistant?
For today, I would call it the second.
The best result came from a boring fix, and that is the point
The experiment used the Ukiyo-eVG dataset instead of the original medical X-ray dataset, because the original data was not available. That matters. We are not looking at a clean reproduction of the old paper. We are looking at a practical adaptation: about 11,000 Japanese woodblock print images, phrase and bounding-box annotations, and a CLIP-based model trained to connect images and text.
The model stack was not tiny either. The brief mentions a ViT-Small backbone, DistilBERT, and a heatmap processor, roughly 90 million parameters in total. Training ran for 800 steps, about three minutes per experiment on an RTX 4090. Evaluation used 1,000 test images, with Mean Rank and Recall@K as the visible scorekeepers.
Here is the part I would underline for an office worker trying to understand the trend: the biggest gain did not come from a dazzling new idea. It came from relaxing a temperature clamp. That single change improved Mean Rank by 113 points. Later hyperparameter tuning added another 30-point improvement.
Across one day, the system ran 42 experiments. It kept 13 and reverted 29. Validation Mean Rank dropped from 344.68 to 157.43, a reported 54% reduction. Final test results reached Mean Rank 34.30, with R@5 around 53.0% for image-to-text and 51.4% for text-to-image.
That is useful. It is also humbling.
In normal workplace language, this is like discovering that a monthly report was bad not because the team lacked strategy, but because one locked setting was wrong and nobody had time to test the alternatives. A good automation system does not need to be visionary in that case. It needs to be patient, fenced, and honest about the scoreboard.
I would keep this comparison table:
| What to check | Why it matters | What this case showed |
|---|---|---|
| Is the task measurable? | Automation needs a score it can trust | Mean Rank and R@K gave the loop a target |
| Are experiments cheap? | Slow tests kill overnight automation | Each run was kept to a few minutes |
| Are permissions fenced? | Agents drift when the room is too open | Sandbox, no network access, limited editable files |
| Did improvement come from insight or tuning? | The story changes depending on the source of gain | Biggest win came from a temperature clamp fix |
| What happens after easy gains? | Many demos fade after the obvious fixes | Later architecture changes became unstable |
This is why I see Autoresearch as a retry machine first. Its strength is not that it thinks like a senior researcher. Its strength is that it can keep testing narrow changes while a person sleeps, then leave a trail of what worked and what failed.
That is already valuable. A lot of work is trapped because nobody has time to run the tenth careful attempt.
The weak spot is the scoreboard
I would be careful about taking this as proof of autonomous discovery.
The brief itself says the last 10% needed more human intervention. After the early phase, especially when the agent moved from parameter tuning into architecture changes, the success rate dropped. Some larger ideas failed. Some attempts became more random. The system occasionally forgot constraints or tried the wrong kind of command.
There is another problem: the metric can quietly become the boss. If Mean Rank is the main judge, the agent will chase Mean Rank. That is fine when the metric matches the real goal. It is dangerous when the metric is only a convenient proxy.
I have seen the office version of this many times. A team automates dashboard cleanup and starts optimizing for fewer red cells. The dashboard looks better. The business does not. The automation did exactly what it was told, not what people hoped it understood.
So the hard part is not “Can AI run more trials?” It can. The hard part is choosing a scoring rule that deserves that much obedience.
Try it on one small system before believing the big claim
If you want to use this idea today, do not start with “AI research.” Start with one repeatable workflow you already avoid because it is dull.
Try this:
① Pick one task with a visible score: response time, error count, conversion rate, reading time, support backlog, duplicate rows removed.
② Give the AI a narrow room: one file, one sheet, one prompt, one checklist, or one report format.
③ Make each attempt cheap: under five minutes if possible.
④ Keep a changelog: what changed, what improved, what got reverted.
⑤ Stop when the gains become random, not when the story still sounds exciting.
Copy-paste line to keep:
> “Before I automate this, what is the score, what is the fence, and what counts as a rollback?”
That question is the practical lesson. Old research ideas become interesting again when AI lowers the cost of trying them. But the future belongs less to people who believe every agent demo, and more to people who can build small systems that measure, retry, and stop.
For the next step, save the table above and use it on one automation idea this week. Next edition: I’ll look at when an AI agent should be allowed to make changes by itself, and when it should only prepare options for a human to approve.
Take-aways
- 42 experiments is the number that made me pause.
- It matters because the story is easy to misread
- My answer is this: Autoresearch is less a research replacement than a disciplined retry machine
→ 한국어 버전 →
