--- title: Fixing Graphcore created: 01/08/2026 description: Throwing LLMs at ML engineering and reverse-engineering. slug: graphcorer tags: ["hardware", "ai"] series: graphcore series_index: 2 draft: yes --- ::: epigraph attribution="System Card: Claude Mythos Preview" link="https://www-cdn.anthropic.com/8b8380204f74670be75e81c820ca8dda846ab289.pdf#page=34" Previous system cards reported a suite of automated research tasks as rule-out evaluations on AI R&D capabilities: failure on these tasks demonstrated that a model lacked capabilities that are likely prerequisite to meaningful R&D acceleration. Claude Mythos Preview, like the models immediately before it, exceeds top human performance thresholds on all these tasks. The suite therefore no longer provides evidence that capabilities are short of the thresholds of interest. ::: In my previous post, I obtained a Graphcore C600 IPU, found that the software was worse than I had thought, and hubristically thought that LLMs could fix it for me. I was, as it turns out, about six months early: GPT-5.2 [TODO really?] and some mediocre Gemini model, the best non-Claude models available at the time[^1], struggled. The exchange protocol was more complex than I thought, and while GPT-[TODO] could understand some of it by reverse-engineering [IPUDOOM](https://github.com/jndean/IPUDOOM)'s device-side JIT, this was not sufficient to bypass the entire C++ runtime, which did substantial setup in the background, and the model didn't seem up to reversing that, though perhaps it could have with a good harness or more work by me. I didn't want to spend days learning to use Ghidra to reverse-engineer the host-side codegen myself, so the project languished. I was reminded of its existence recently, and decided to work on it again, using newer models (GPT-5.6 Sol and briefly GPT-5.5), which instead worked very well. I don't particularly like "vibe coding" extensively like this, as it pains me to not know any of the details of the code, the code quality is bad in some ways, and the models are annoying. It was challenging to get GPT-5.6 to stay on task reliably, and I had to repeatedly push it to do things properly and generalizably, avoid redundant code, label things properly and so on. It also repeatedly lobotomized itself with context compactions, downplayed problems[^2], failed to remove orphan codepaths and favoured small incremental changes too early. Also, OpenAI's infuriatingly worded cybersecurity prohibitions sometimes blocked reverse engineering. Regardless, it's a fearsomely fast and competent engineer, much more so than me, when well-motivated and "in its lane", particularly for compute kernel optimization[^3]. It reverse-engineered enough of the mostly-undocumented exchange control system through breakpointing C++ libraries, black-box-analyzing differentials, testing on hardware and sourcing register names from headers and bindings code to eventually operate device initialization, on-device point-to-point and multicast transfers and device-to-host/host-to-device communication with minimal interaction with the official SDK (its "codelet" compiler, a recorded blob of device initialization transfers and the kernel module are used). It took several days to have it resolve the problems, implement the debug interfaces and reliably generate all the exchange code, preambles, profiling logic and so on, and it left in lots of special-casing without mentioning it, but it was very rapidly able to optimize GEMMs to achieve "okay" MFU (~40% at 4096x4096 FP32), admittedly mostly by copying the Graphcore SDK's kernel design choices. It ended up having to more-or-less copy the Poplar architecture of a graph of operations serialized to programs on each tile assembled from codelets; there are some possible alternative approaches like some kind of per-tile megakernel, spatial unrolling, some kind of (ab)use of self-modifying code, or streaming weights off the host for large batches, but they would all be complicated and worse. I suspect that it is possible to drop the BSP model and statically schedule compute phases so that only one full-device sync is necessary per run - this would have been unjustifiable in a large-scale deployment, but I only have one IPU - but I don't care enough since I don't think there would be a huge gain from it, especially as you probably couldn't interleave communication and computation without constant bank conflicts. Watching Codex hill-climb the kernel performance did make me wonder whether I underestimated the complexity of programming for the architecture, since it would frequently complain about issues with memory allocation, bank conflicts, running things in the wrong context and forgetting to dual-issue float instructions, but on the other hand most people wouldn't be writing raw assembly and the model would not make these mistakes twice if it retained context. On the other other hand, the MFU was quite bad even after some optimization, and maybe the real problem is that it is hard to feed the AMP (accumulating matrix product) units. I came to respect the Graphcore SDK while watching Codex work, because its compiler was also quite slow and it too struggled to fit 817MiB of vision encoder weights into 897MiB of distributed SRAM which also had to fit scratch space and code, and it had to expensively allocate millions of commands, regions and transfers. Still, it is one language model, Graphcore had years to make this work, and the SDK still doesn't support FP8. It eventually implemented the full model despite the SRAM capacity challenges, the consequences of prior hacks/specialization in its code and the slow compiler. Apparently, however, it solved the capacity issues by switching all the weights to FP8 without telling me; I was expecting to do this later as an optimization step to increase batch size. By casually implementing several algorithms to improve the quality of the weight quantization, and getting lucky by finding that the SigLIP MAP pooler smoothed over lots of the error[^4], it was able to get sufficient accuracy as measured by cosine similarity, if very slowly. I let it loose to do more kernel/workload tuning, but it was stuck with extremely bad performance due to the weight of its slop and assumptions. I set it to xhigh overnight, but ran out of usage credits in a day, and while it was able to, apparently, fix something, it continued to bloat the code impractically for marginal improvements and did not appear to work well (<5% MFU). I decided to have another instance erase all the jankier code and to rewrite it, this time using slightly less "vibing" and reading some of the output and specifying how it should work, based on lessons learned from seeing GPT-5.6 in full auto mode complaining. This worked in that I retained some sense of what it was working on and the code was apparently less bloated and more general than with full "vibe" power, but did not work in that I did not follow much of it. I had it start with a multi-level graph compiler (high-level model representation to detailed layout choices for each operation based on search over a cost model to per-tile sequences to actual code) and work on optimizing and doing correctness for a single transformer MLP block characteristic of the SigLIP model. It found the upprojection easy to implement reasonably efficiently, as before, but struggled with occupancy and communications overhead for the downprojection, which took almost 10 times longer despite the same theoretical FLOP count. After a few days of increasingly esoteric-sounding optimizations and layout changes, cost model updates and some time spent removing/adding bugs from/to the exchange code generation (which it additionally found a [patent](https://patents.google.com/patent/US11321272B2/en) for, helpfully) this was improved to ~2x slower than upprojection, for 20% or so MFU overall. TODO flashattention TODO whole model TODO comprehend codex's quantizer?! TODO profiler graphics TODO upproj/downproj/etc This has taken TODO billion tokens, evaporating the majority of Lake Michigan. TODO clock frequency change? [^1]: I don't use Claude Code personally because of unpleasantly low rate limits. [^2]: At one point, it was analyzing device/host transfers, and managed to somehow lock the card into an invalid state which did not even yield to a [PCIe FLR](https://unix.stackexchange.com/questions/73908/how-to-reset-cycle-power-to-a-pcie-device) (perhaps because the device doesn't support them...); I had to power-cycle it to unstick it. It only mentioned this offhandedly as an explanation for its work not being fully "hardware-validated" yet. [^3]: I expect that OpenAI trained on this specifically. On the other hand, it has a bizarre unwillingness to read the profiles without prompting. [^4]: Possibly this is expected behaviour because the model is creating unreasonably large numbers as an "attention sink" but their exact magnitude isn't very important.