--- title: Limits to narrow LLM complementarity description: Taste is probably not the bottleneck. created: 03/09/2026 slug: complement tags: ["ai", "economics"] --- Since late 2024, frontier LLM systems have been trained with high-compute reinforcement learning on outcome rewards[^1], as opposed to the previous dominance of reinforcement learning from human feedback (smaller-scale due to the need for labelled data) and self-supervised pretraining. Broadly, this makes them better at "doing tasks" much more quickly and cheaply[^2] than scaling pretraining would, but [scale improves everything](https://gwern.net/scaling-hypothesis) and this is not so general. This has led to some predictions along the lines of "taste is the bottleneck". I don't believe that this has any more long-term validity than previous claims that "harnesses are the bottleneck", "prompt engineering is the bottleneck" and similar, but in the near term it will probably remain true that LLMs will lack some human capabilities, but this doesn't mean that humans will be usefully able to complement the AI skillsets, or that human data will be particularly valuable, as argued [here](https://www.pathwaysai.org/p/coasean-economics-of-agent-swarms). To be an economically relevant complement, humans must be able to provide a useful skill more cheaply than LLMs can themselves, after accounting for additional coordination costs. These, I believe, are very large. An underappreciated fact about modern LLMs is that the instances running in production have[^3], for the vast majority[^5] of their "lives", never interacted with live humans! Pretraining provides trillions of tokens of offline, static data, perhaps five orders of magnitude more text than a human has ever read, but the models are no longer like the humans of the pretraining corpus, nor especially like the earlier models in the smaller volume of chat/agent transcripts which must now have made it in, given the long delays (at least ~6 months) in dataset collation and training. After this, posttraining exposes a fresh new model to some undocumented mix of somewhat off-distribution transcripts and outputs from previous models, and labyrinths of oft-broken environments with buggy, exploitable autograders and other models grading against rubrics, quite possibly written by different models with little human intervention. It would probably be possible to add a stage with live humans involved in RL training to provide feedback on-policy, but humans are slow, relatively expensive and unpredictable, so it would be very hard to give this anywhere near the scale of the earlier training steps, especially as it wouldn't be reusable for later models in the same way, so I don't think this is practiced. It is plausible that this is behind many of the pathologies of current LLM agents. And what pathologies they are! I generally agree with [this post](https://blog.redwoodresearch.org/p/current-ais-seem-pretty-misaligned) arguing that they frequently overstate progress, downplay problems, unnecessarily stop early and overstep bounds, but more than that, Claude outputs are [famously inscrutable](https://programasweights.com/claudish), having adopted a hard-to-describe but very distinct ahuman dialect, and modern frontier LLMs have tics like "The honest answer is" and "You're right to push back on this", or obsessions with "audits" and "provenance" and "byte-identical" matching. I suspect that these are something like reflexive grader-exploiting behaviour, like the [goblins](https://openai.com/index/where-the-goblins-came-from/). In my own experience with [GPT-5.6 Sol](https://openai.com/index/gpt-5-6/) on one project, I've found that it simply will not meaningfully refactor code (having made unpleasant architectural decisions earlier without telling me, which I wanted to roll back), but instead writes adapters around existing logic without providing simplifications or reducing code size, and does not reliably acknowledge this even with subagents performing the underlying "refactor", although egregiously obvious cases like this will probably be patched out insofar as AI developers can do that. The longer an agent or group of agents runs for without human intervention and steering, the further it appears to diverge from human quality standards, making it harder for humans to work on the same projects. This has also been [noted](https://www.erdosproblems.com/forum/thread/4/proof-claims#proof-claim-224) [with](https://arxiv.org/abs/2606.03743) mathematical exposition. These can substantially be capabilities issues - an unsupervised GPT-5.6 on my project was far too unwilling to rethink its design and favoured marginal local improvements instead. But, despite this, agents can accomplish great and terrible things, such as the [Hugging Face incident](https://openai.com/index/hugging-face-incident-and-the-road-ahead/) and several solutions to mathematical conjectures. Human code quality standards and proof exposition are a response to particular human limitations which AIs need not and do not share; LLMs can for example much more rapidly orient themselves in unfamiliar codebases, oneshot large, complex scripts and files, and trace chains of logic, and so the shortest and fastest path to more capabilities doesn't appear to involve more humanlike behaviours. Complementarity could take place at a coarser level than humans and AIs working on a codebase, and be more like feedback than participation, but this isn't guaranteed. Even granting that AIs do not have and will not have "taste" and associated skills, it doesn't seem required for lots of products. I've [noted before](/aic/) that the average person prefers AIs' writing to humans' despite it being in some sense worse, and this is plausibly also true of UI design. And, of course, humans are still expensive: an hour of software-engineer salary will buy millions to tens of millions of frontier model tokens, enough to implement a modest piece of software from scratch, and LLMs interacting with a human suffer delays and frictions since humans are slower and sometimes offline. As I have [also noted before](/asi/), there are usually many ways to achieve a goal, and some of these are more suited to LLMs than the most obvious to humans. For example, rather than use "taste" to design a UI, LLMs could conduct high-cadence A/B tests with efficient statistical methods, which humans would rarely do because, to humans, implementation is costlier than thinking of a plausible design, and experimental design is unintuitive and unnatural. The upside case here is that, with better tooling, more focus on user modelling and better-trained models, the coordination issues with humans could be improved, and human principals could also be queried in information-gain-maximizing ways for parts LLMs can't handle, with good summaries of current context, at the most relevant times, asynchronously. But in the current paradigm this would need to be trained in specifically[^8], and AI companies don't seem inclined to bother, with their ideological/economic commitment to [recursive self-improvement](https://www.anthropic.com/institute/recursive-self-improvement) and the additional expense of human-in-the-loop training[^6]. Also, without an on-the-ground view, humans may not know what they don't know, and be in a similar position to the CEO of a large, complex corporation who can't trust their employees to act as they would, or a voter in elections[^7], resulting in Goodharting or irrelevance. As for the importance of proprietary data, such as internal statistical datasets and experience datasets, I think this is also overstated for similar reasons (LLM-native methods can substitute for the process that created it or bypass the need for it). Existing public or already-gathered data on many topics is woefully underanalyzed[^4]. Tacit knowledge is not relevant if an LLM system performing similar tasks finds it better to rederive approaches more suited to its capabilities; it is conceivably [much shallower](https://gwern.net/doc/psychology/cognitive-bias/illusion-of-depth/abstract) than we feel; and it can be reacquired with [optimized experimentation](/asi/#data-is-no-barrier) where ground truth or a good proxy to it can be found. It is possible that the kinds of tasks models are being used in and trained in don't locally benefit much from this so the capability is not developing, although they do seem very good at [local hillclimbing](https://www.anthropic.com/research/automated-alignment-researchers) "research" now. The role of humans may then, for a time, be describing legacy processes to inscrutable agent swarms. [^1]: AI companies are not very open about this, and the real implementation is probably more complicated and based on some kind of backfilled process rewards and/or LLM-judge scoring rubrics, but [DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1) worked, was competitive at the time and used only outcome rewards for R1-Zero. [^2]: The AI companies will also not tell us this, but it is a reasonable inference from R1 and what we saw of GPT-4.5. [^3]: To the best of my knowledge: this is not publicly documented either way by any closed AI company. [^4]: For example, in 2024 a [new form of life](https://pmc.ncbi.nlm.nih.gov/articles/PMC10827157/) present in ~10% of human microbiomes which had apparently avoided all previous detection was discovered by reprocessing old sequence data. [This work](https://rchatgpt-pulse.github.io/) apparently could have detected the GPT-4o problem much earlier, though of course it is post-hoc. [Dreamer 4](https://arxiv.org/abs/2509.24527) was able to learn enough about Minecraft to obtain diamonds without any online training. [^5]: Roughly all but 10-6. [^6]: You could train this by using LLMs to simulate the User, but I would expect this to diverge from real user interaction unless it was handled with care. [^7]: Though voting also has incentive issues: the agent swarm operator presumably has some reason to ship useful things. [^8]: Or potentially built in as a harness feature, but current trends are towards vertical integration and models are likely trained with a particular harness now, so the AI companies would still have to design for this.