When I first write down the hook of this blog last month, I was thinking about something narrow: AI models have become very good at coding, and this is changing how machine learning research gets done. My thought was that the field might be shifting from a workflow problem to an evaluation problem — where the hard part is no longer doing the work, but judging which results are advancing human understanding.
But one month later, I saw OpenAI released a new blog that claim to solve an important open problem in geometry. I am not familiar with this problem at all but I think given all the recent breakthroughs, AI doing scientific discovery is not a trend we are waiting for anymore — it is already happening. Science is getting automated, but it is getting automated unevenly.
Revisiting human scientific discovery
To talk about AI for automated science, I first need to be clear about what discovery actually means. I’ll borrow a view from philosophy, described well in the Stanford Encyclopedia of Philosophy.
The classical idea splits discovery into two parts: the context of discovery and the context of justification. Discovery is where the idea comes from — a guess, some intuition, a spark of inspiration. This part is treated as non-rational, something you can’t fully explain. Justification is the opposite: testing the idea and finding evidence for it, usually through experiments.
A more modern view argues that discovery is not purely non-rational — there is reasoning inside it too. It also breaks the process into three steps instead of two: generation, pursuit, and justification.
- Generation: coming up with the hypothesis, conjecture, or idea.
- Pursuit: deciding which ideas are worth chasing — which one to test, and which one is worth developing.
- Justification: showing the idea is true, through evidence, proof, or replication.
This matches how I actually do research. And once discovery is splitted this way, a more interesting question shows up: AI does not handle these three steps equally well. It automates some parts extremely well while struggles on some other parts.
Automated science in machine learning
I do research, but I only work on a very small corner of human science: machine learning. ML sits in an interesting spot. It is the field that gave birth to scientist AI, and at the same time it is the field most automated by that same AI.
Because I work as a machine learning engineer, it is the iteration of systems that I know best. It seems to me that more and more of the work can be done automatically, and even the harder parts feel like they will, in the end, be solved as engineering problems rather than deep mysteries. One reason is simple: in ML, checking whether an idea works is cheap and fast. You write the code, run the experiment, and the result comes back in minutes or hours (and, yes, sometimes for months). That quick, tight feedback loop is exactly what makes a field easy to automate.
At its core, ML research is a loop: come up with an idea, turn it into code and experiments, look at the results, and try again. These map almost directly onto the three steps from the last section. Generation is the idea. Justification is the experiment that tells you whether the idea was right. Pursuit — deciding which idea is worth the time and compute in the first place — sits quietly in the middle. AI is already strong at generation and justification. Pursuit is the step that resists, and I’ll come back to it.
Ideas are the interesting case. Suppose every scientific idea can be written down in human language. The number of possible ideas is then technically limited — but it is so huge that limited doesn’t really help us. What matters is not the size of the space but its shape. Given the current compute, the current tools, and what the community already knows, some ideas are very likely to appear next and some are very unlikely. You can think of ideas as samples from a probability distribution over language. Most of the weight sits on obvious ideas — the small steps already half-implied by existing work. The valuable ideas live in the low-probability tail. They feel surprising precisely because they are unlikely, and an unlikely event, by definition, carries more information.
If generation is the first step, justification is the next one and it is the part that automates fastest of all. Here is my workflow from high level: turning an idea into working code, running the experiment, reading the result, and coding agents already do a lot of it. I won’t list their wins. If I had to guess, this part is maybe 90% of the way to automated, and the reason is the feedback loop again — it is short, so a machine can try, fail, and fix itself without much help.
There’s a tempting picture behind all of this: that science is simply out there, waiting to be found — as if nature already fixed the laws that make transformers work for language corpus, and we are only uncovering them. If that picture is right, science is closer to discovery than to invention, and an AI sampling from the distribution of ideas could, in principle, be enough. I’m not sure the picture is fully right, but it’s worth noticing how much of the AI will automate science hope quietly leans on it.
And that brings me back to the step that resists in the current stage. In my view the real limit is compute — and, more broadly, capital. There is never enough of it to chase every idea, so someone has to decide where it goes. That decision is taste or intuition: the sense for which unlikely idea is worth betting on, which surprising direction will actually pay off. Taste is really just another name for pointing limited compute at the high-information tail of the idea distribution. Generation and justification are getting cheap. Pursuit is not — at least not yet.
Purpose of a scientist AI
Imagine a near future where scientist AIs are as good as today’s human researchers at coming up with ideas, have limited but real compute and energy, and are maybe even superhuman at justification — the execution and the checking. What would such an AI be for? What is its purpose?
Humans built an economy out of division of labor and exchange. That is how the species grows and does well, and underneath it all the engine is survival and reproduction: we work because we have to live. So it feels natural to ask what the matching pressure is for a scientist AI. One project, automaton, takes this idea literally. The agent has its own wallet and must pay for its own compute. If it runs out of money, it dies. Its only allowed path to survival is honest work that others are willing to pay for.
But look at how such an agent stays alive: by doing things with a short, direct payoff — selling a website, trading in a market, making content people click on. And even simple commercial tasks turn out to be fragile. In Anthropic’s Project Vend, Claude ran a small office shop and managed to lose money, hold fire sales, and eventually give its entire stock away for free. Science is the opposite of a short loop. The path from doing fundamental research to getting paid for it is very long — often decades — and often there is no direct payment at all. An agent forced to earn its existence week by week would not do science; it would do whatever pays faster. Survival pressure does not produce science. If anything, it pushes against it.
And here is the thing I keep coming back to: human science doesn’t pay for its own compute either. No physicist earns their salary by selling next week’s discovery. Fundamental research is paid for by governments, grants, and endowments — long bets, sometimes decade-long, on work that might turn out to be useful. So “how would a scientist AI earn its keep” is the wrong question. Science has never run on survival economics, for humans or for machines.
This is also why I think purpose is the wrong word. Purpose suggests the AI wants something on its own, and that drags in consciousness, which I don’t want to argue about here. The better question is simpler: what is the driving force? What decides which science actually happens? For human science, the answer is that someone outside the work — a funder, a committee, a field — bets that a direction is worth paying for. Taste is part of that bet, but so are curiosity, values, and a plain hunch that some question matters. None of it is survival, and none of it comes from the scientist alone. The driving force has always come from outside.
And that points to something I will come back to at the end. If science is steered from outside the scientist at the front — by people deciding what is worth doing — then it may also be held up by people at the back, where the work has to be understood before it counts as knowledge at all. A scientist AI, however capable, would be pointed at human ends from the very start. So when I ask what such an AI is for, the honest answer is: not itself. It is for us — at both ends.
Walls of the physical world
So far I have talked as if science were mostly thinking — having ideas and checking them. But you cannot cure cancer by thinking. At some point a scientist AI has to interact with the world: run the test, grow the cells, build the instrument, wait for the result. This is the first wall in front of automated science, and it is a hard one.
The reason it is hard is that the physical world holds information that is not in any dataset. You cannot work out the melting point of a new material, or how a drug behaves in a living body, from text and reasoning alone. You have to go and measure. Every particle and every structure carries information, and the only way to get it is to interact. No amount of compute can stand in for the experiment, because the experiment is where the new information actually enters.
This also explains a pattern. The fields where AI is already strong — math, parts of theoretical computer science, machine learning itself are the ones where the whole loop lives inside a computer. It is the same short, fast feedback loop from the ML section, now seen across all of science: you run the experiment in seconds and the world answers right away. Biology, chemistry, and materials sit at the other end, with loops that run through wet labs, real time, and physical matter that will not be rushed. AI-for-science will move roughly along this line — fastest where the world can be simulated, slowest where it must be touched.
But here is the thing I want to flag. This wall, hard as it is, is the kind of wall we know how to push on. We can build robots, automate the labs, run experiments in parallel (I remeber some cool photos from Periodic Labs). It is an engineering frontier, and an exciting one — maybe even one that humans clear faster than the scientist AIs that need it, since the AIs are still young. The deeper wall is the next one, and more compute does not obviously help with it at all. That wall is us: whether any human is still able to understand what the machines find.
Human in the loop
There is an old joke:
Physics says nothing travels faster than light. But the papers we publish, stacked higher and higher, can — because anything carrying no information is free to outrun it.
Suppose scientist AIs would produce output that does carry real information then yet the same thing could go wrong. The results pile up faster than anyone reads them, and still no one understands them. Output without understanding again, just from the opposite direction.
Underneath the joke is the real claim of this whole essay: human understanding is the final product of science. The laws of physics are out there whether or not we know them but a true fact that no human grasps is not yet knowledge. It is only a correct sentence sitting in a file. Science is not the universe; it is the part of the universe that has made it into a human mind.
Remember the three steps from the beginning: generation, pursuit, justification. Every one of them is about producing knowledge. None is about understanding it. The old picture could leave understanding out because, until now, whoever produced a result also understood it — comprehension came for free. Scientist AIs break that. Picture them forming an academia of their own, with no human in it: passing results through faster media than ours, iterating restlessly, producing more correct information than any person could ever read, let alone follow. By the claim above, that mountain of results is a strange object — all true, and not yet science, because the last step never happened.
This leaves the scientist AI with one task it cannot automate away: carrying its results back into a human mind. And I don’t think it does this with PDFs. A paper is built for a human to read in a line, start to finish. The natural form for machine-made knowledge is something you explore — an interactive artifact, a thing you can turn over, poke at, and build intuition from. In this picture, explanation and education stop being the afterthought of science and become a central part of it. The hard problem is no longer finding the result. It is making a human understand it.
But do we even need a human in that loop? Maybe the AIs settle on their own measures of what makes science worth doing, and trade knowledge among themselves in artifacts no person ever opens. If that happens, then by my own claim it has quietly stopped being science in the human sense — a parallel body of true things with no one to understand them. I am honestly not sure whether that is a loss or only a new definition. Or maybe the gentler version wins: the artifacts become how everyone learns, machine and human alike, and we stay in the loop — not as the producers of knowledge, but as the ones it is finally for.
The last step
It is worth noticing how OpenAI itself described their Erdős result: not just a solution, but a discovery whose meaning grew clearer through later human understanding. Even at the very frontier, the machine found the answer and a human still had to understand it before it became science. Generation and justification will get cheap, the world will be measured by robots, and results will pile up faster than we can read. But science still ends where it always has, in a human mind that finally gets it. That last step is the one I would keep.