Fixing Graphcore

In my previous post, I obtained a Graphcore C600 IPU, found that the software was worse than I had thought, and hubristically thought that LLMs could fix it for me enough to run SigLIP, used in my meme technology. I was, as it turns out, about six months early: GPT-5.3-Codex and some mediocre Gemini model, the best non-Claude models available at the time^[1], struggled. The exchange protocol was more complex than I thought, and while GPT-5.3 could understand some of it by reverse-engineering IPUDOOM's device-side JIT, this was not sufficient to bypass the entire C++ runtime, which did substantial setup in the background, and the model didn't seem up to reversing that, though perhaps it could have with a good harness or more work by me. I didn't want to spend days learning to use Ghidra to reverse-engineer the host-side codegen myself, so the project languished.

I was reminded of its existence recently, and decided to work on it again, using newer models (GPT-6, GPT-5.6 Sol and briefly GPT-5.5), which instead worked quite well. I don't particularly like "vibe coding" extensively like this, as it pains me to not know any of the details of the code, the code quality is bad in some ways, and the models are annoying. It was challenging to get GPT-5.6 to stay on task reliably, and I had to repeatedly push it to do things properly and generalizably, avoid redundant code, label things properly and so on. It also repeatedly lobotomized itself with context compactions^[2], downplayed problems^[3], failed to remove orphan codepaths and favoured small incremental changes too early. Also, OpenAI's infuriatingly worded cybersecurity prohibitions sometimes blocked reverse engineering. Regardless, it's a fearsomely fast and competent engineer, much more so than me, when well-motivated and "in its lane", particularly for compute kernel optimization^[4].

It reverse-engineered enough of the mostly-undocumented exchange control system through breakpointing C++ libraries, black-box-analyzing differentials, testing on hardware and sourcing register names from headers and bindings code to eventually operate device initialization, on-device point-to-point and multicast transfers and device-to-host/host-to-device communication with minimal interaction with the official SDK (its "codelet" compiler, a recorded blob of device initialization transfers and the kernel module are used). It took several days to have it resolve the problems, implement the debug interfaces and reliably generate all the exchange code, preambles, profiling logic and so on, and it left in lots of special-casing without mentioning it, but it was very rapidly able to optimize GEMMs to achieve "okay" MFU (percent of theoretical peak) (~40% at 4096x4096 FP32), admittedly mostly by copying the Graphcore SDK's kernel design choices.

It ended up having to more-or-less-copy the Poplar architecture of a graph of operations serialized to programs on each tile assembled from codelets; there are some possible alternative approaches like some kind of per-tile megakernel, spatial unrolling, some kind of (ab)use of self-modifying code, or streaming weights off the host for large batches, but they would all be complicated and probably worse. I suspect that it is possible to drop the BSP model and statically schedule compute phases so that only one full-device sync is necessary per run – this would have been unjustifiable in a large-scale deployment, but I only have one IPU – but I don't care enough since I don't think there would be a huge gain from it, especially as you probably couldn't interleave communication and computation without constant bank conflicts.

Watching Codex hill-climb the kernel performance did make me wonder whether I underestimated the complexity of programming for the architecture, since it would frequently complain about issues with memory allocation, bank conflicts, running things in the wrong context and forgetting to dual-issue float instructions, but on the other hand most people wouldn't be writing raw assembly and the model would not make these mistakes twice if it retained context. On the other other hand, the MFU was quite bad even after initial optimization. I came to respect the Graphcore SDK while watching Codex work, because Codex's compiler was also quite slow and it too struggled to fit 400 million parameters of vision encoder weights into 897MiB of distributed SRAM which also had to fit scratch space and code, and it had to expensively allocate millions of commands, regions and transfers. Still, it is one language model, Graphcore had years to make this work, and the SDK's main interfaces (the Python ones) still don't support FP8.

It eventually implemented the full model despite the SRAM capacity challenges, the consequences of prior hacks/specialization in its code and the slow compiler. Apparently, however, it solved the capacity issues by switching all the weights to FP8 without telling me; I was expecting to do this later as an optimization step to increase batch size. By casually implementing several algorithms to improve the quality of the weight quantization^[5], and getting lucky by finding that the SigLIP MAP pooler smoothed over lots of the error^[6], it was able to get sufficient accuracy as measured by cosine similarity, if very slowly. I let it loose to do more kernel/workload tuning, but it was stuck with extremely bad performance due to the weight of its slop and assumptions. I set it to xhigh overnight, but ran out of usage credits in a day, and while it was able to, apparently, fix something, it continued to bloat the code impractically for marginal improvements and performance was very bad (~10 images/second).

I decided to have another instance erase all the jankier code and to rewrite it, because its changes seemed very bad (absurd hacks to shave off a few bytes of SRAM) and the code was growing enormously. This time, I used slightly less "vibing" and more reading the output and specifying how it should work, based on lessons learned from seeing GPT-5.6 in full-auto mode complaining. This worked in that I retained slightly more sense of what it was working on and the code was apparently less bloated and more general than with full "vibe" power, but did not work in that I did not follow much of it. I had it start with a multi-level graph compiler (high-level model representation to detailed layout choices for each operation based on search over a cost model to per-tile sequences to actual code) and work on optimizing and doing correctness for a single transformer MLP block characteristic of the SigLIP model.

It found the upprojection easy to implement reasonably efficiently, as before, but struggled with occupancy and communications overhead for the downprojection, which took almost 10 times longer despite the same theoretical FLOP count. After a few days of increasingly esoteric-sounding optimizations and layout changes, cost model updates and some time spent removing/adding bugs from/to the exchange code generation (which it additionally found a patent for, helpfully) this was improved to ~2x slower than upprojection, for 20% or so MFU overall. I had it implement a FlashAttention kernel, which seemed to work okay, and QKV projection into FlashAttention, which required more layout changes. Eyeballed performance was still quite poor, however, and GPT-5.6 struggled to improve it further with minor tweaks.

I dredged up the profile from when I had tried to use the official SDK/PopRT to run SigLIP and had the model compare it against its own code, then had it run minimal matmuls with the SDK. It noted that, while inefficient attention rearrangements dominated runtime, the SDK's GEMM planner, despite treating matrix multiplication as a degenerate form of convolution, was in fact highly optimized and about three times more efficient, particularly on the communications-heavy downprojection, with 60% higher throughput in the matmul compute and a more efficient exchange schedule. It put this down to differences in blocking behaviour, then more efficient context management to avoid reloading accumulators constantly, then differences in reduction and several other things. These did seem to be real problems, but not consequential ones responsible for more than a few percentage points: after lots more poking, a convoluted and bad attempt at introducing padding to allow more flexible layouts, and more marginal improvements, having it look at the SDK harder revealed that it had been using much more complicated and inflexible tensor layouts than necessary because of a strange choice of AMP coefficient load order made earlier.

In some places, the Graphcore hardware and software contain unusually elegant solutions which feel as if designed by aliens, or perhaps a superintelligence from the future, with only passing familiarity with Earth computers and the limitations of mortal engineers. One of these is that its Accumulating Matrix Product units work by loading a chunk of a matrix into a "CCCS"/"CWEI" buffer and computing dot products into an accumulator (roughly), and these instructions can be dual-issued with a flexible-stride simultaneous load/store instruction, which allows complex rearrangements during (some) weight loading and output. GPT-5.6 had apparently, aeons ago when writing a plausible GEMM kernel, picked some arbitrary permutation it liked and kept weight storage in one of several layouts more closely tracking the "natural" AMP inputs/outputs. This is in the ISA manual, but I only skimmed it and assumed that GPT-5.6 would do the right thing by default, only bringing up the load/store instructions when it wrote silly copy kernels. It took hours to have it untangle this and support more layouts, and ultimately it had to mimic the SDK design. The SDK also has a clever linearized activation layout which reduces the need to do cross-operator planning.

This does make me slightly more sympathetic to the "LLMs can't write novel things" people, though obviously it could do several novel things here and this is a matter of degree. Presumably it was trained to write kernels on less flexible GPUs. I'm sure there are similar features in other hardware, but I don't know the details of most platforms out there, and as far as I know (I don't program modern GPU kernels) Nvidia hardware uses specific hardwired swizzles and TMA rather than anything similar.

While trying to shave off the remaining gap with the SDK in MLP computation, through making changes to exchange scheduling (to benefit from some exchange time freed up by moving inefficiently placed rearrangement copies) which caused mysterious deadlocks, I learned that the earlier reverse-engineering and exchange schedule generation had been incomplete^[7]. GPT-5.6, having learned/been reminded of the relevant behaviour, wanted to add delays to hack around it until prompted to see what the SDK was doing with it. It then took enormous time and effort to have it implement the proper, general solution. Sometimes I wonder just how far it could go with a Codex plugin which says Don't add ugly hacks., Is that really the best way to do it? and such at regular intervals.

Eventually I grew annoyed by its code quality and persistent inability to refactor anything, so I procrastinated on the project until GPT-6 came out^[8]. It was able to refactor things, making me slightly happier with it, and generally felt more reliable and smarter. Of course, I can't reliably distinguish this from it being better at talking and not smarter, but it could improve the exchange scheduler algorithm usefully, whilst GPT-5.6 mostly flailed at this. After some time doing general cleanup, I returned it to performance, including MLP performance improvements (taking it past SDK performance, though with approximate GeLU rather than exact) through improved exchange scheduling, and then the trickier target of attention.

Attention is difficult because of the large softmax^[9] and Q/K/V rearrangement. The SDK had compiled it as an efficient fused QKV projection, sequential and very inefficient rearrangements into a very fast GEMM, and then an incredibly slow and unfused softmax/rearrange. At this time, "my" stack's attention was implemented^[10] as separate Q/K/V projections into also-inefficient rearrangements into FlashAttention, or alternatively into somewhat-inefficiently-computed fully-materialized attention. This was improved somewhat with further exchange improvements^[11] and various smaller optimizations and generalizations.

I then moved on to FP8^[12] support (which was less performant than I hoped because of slow casts, though this was eventually fixable) and the remaining pieces of the full model such as the input projection, LayerNorm and pooling, with a one-layer test. This was quite quick to implement but slow, but GPT-6 optimized the kernels for those eventually, parallelized better, parallelized better again, optimized more kernels, overhauled parts of the compiler for faster layout search and fused things. It then turned out that the exchange code generation made even more unjustified assumptions, which I removed.

The one-layer tests failed to scale beyond about 8 layers because of poor placement algorithms and inefficient memory use which various epicycles failed to fix, so I had GPT-6 throw away the high-to-mid graph planner, which couldn't reliably handle the tight-memory conditions, and switch to local optimization on a simpler, more memory-efficient baseline plan^[13]. This worked after more placement fixes, but achieved somewhat worse performance than the SDK, until after even more placement fixes it was able to switch from FlashAttention to fully materialized attention^[14] and slightly outperform the SDK (~100 images/second), making it still slightly slower but more power-efficient than my RTX 3090^[15]. Further minor but increasingly convoluted tweaks brought this up several percent, eventually, to 114 images/second including host comms time. It's probably not possible to do noticeably better^[16] without higher batch sizes than 1 (requiring a smaller model in progress; exchange schedules were being constructed wrong) or more IPUs so that less compressed layouts fit. I'm evaluating the Perception Encoder models, so while the SigLIP optimization is practically finished, I'm not deploying this yet.

Some hardware profiles

Clicking will bring you to an interactive profile viewer. These are very large files and will take some time to load. Do not attempt to load them on mobile.

An early, very poorly optimized implementation of a vaguely SigLIP-like-MLP-like MLP.

More sophisticated MLP with SDK-like performance, much more efficient exchanges and better work distribution and kernels.

Moderately optimized FlashAttention with Q/K/V projections. Note the poor occupancy in the rearrangement. The FlashAttention has high occupancy, but (though this profile doesn't show it) is still inefficient because of very small GEMMs.

Non-Flash attention using fused QKV projection. This still suffers lots from rearranging attention inputs. Softmax is slower than the attention score multiplication, though this is before some optimizations to the softmax kernels.

Single-layer ViT including input projection and pooler. The last part of the pooler is again low-occupancy and inefficient, but relatively small with all the layers running. LayerNorms and few-tile copies/padding lead to suffering. In this profile, the FP16 to FP8 cast kernels are fused with some rearrangement.

One layer of the final ViT model (profiling of repeated layers only measures the first instance), plus the input projection. It is sadly able to spend very little time in dense compute.

Indicative memory profile from the final ViT (only shows 3 of 1472 tiles). There's nontrivial free space, but apparently not enough to fit another batch element, partly from fragmentation.

This has taken 13 billion tokens (total, Codex CLI accounting), evaporating the majority of Lake Michigan, or 150 million output tokens, equivalent to ~0.11% of a Navier-Stokes Millennium problem. I stand by remarks elsewhere that coding agents are a terrible interface format for human augmentation, and thus for code quality or applications where you need to have any understanding of what is going on. However, for this project I don't especially need those, so they work, and there is currently no credible alternative. You can bask in the agony of the code here (realistically you will need an agent to use this). I'm not sure how much me having some idea of how the hardware worked and steering it thoroughly helped: I had some good ideas which GPT-6 didn't, but my ideas were also often worse (fixing some bad-looking copies did little for end-to-end performance; some odd behaviour in the exchange scheduler I wanted it to fix turned out to be correct; it wanted to overhaul low-level copy span generation and I wanted some minor performance improvements, and it turned out to be right^[17]; I sometimes didn't have any credible suggestions for what to do next and it did), and GPT-5.6's earlier low-intervention version did, technically, run.

GPT-6 didn't seem to make useful profiling tool changes/upgrades as often as I think would have been optimal unprompted, but maybe it was just used to me taking the lead on this and could have been elicited better. It couldn't easily read the HTML profiles, but had its own CLI profile query tool based on the same data. I never looked at exactly what it was reading from this, but it may have been better for reading per-kernel performance data than the general shape of the program. Sometimes GPT-5.6 and GPT-6 would, for some reason, extract and parse the JSON from the HTML profiles rather than using the original profile data. GPT-6 also likes writing complex Python scripts to edit files rather than using its harness's edit tool, for unclear reasons.

It would perhaps have been possible to achieve most of this with the normal Graphcore Poplar SDK, a better-optimized attention vertex and some kind of hack to get FP8 support in the dense matmuls, but there is no artistry in that, and it still wouldn't have allowed for cross-operation layout planning. I have also come to appreciate their architecture more: the IPU is a fearsome processor for small-batch dense matrix multiplication and probably convolutions, hampered by a few unreasonably hardware-unfriendly operations (layernorm, softmax^[18]) in small-batch transformers, model build times and lack of RAM^[19]. I'm sure some kind of superoptimizer or directly handwriting all the tile code could do better, but I don't have the practical ability to do that. A convolutional CLIP should also do well, but there are no modern ones. I think it should be possible to get ~20% or so more performance from hacking the firmware to allow higher clocks, as the C600 (1.5GHz) silicon is probably the same as the Bow-2000 (1.85GHz) silicon. If anyone has a lead on additional IPU hardware (other than C600 cards), please tell me.

I have some hardware suggestions for the next generation which may or may not exist:

• The rumoured LPDDR support is a good idea. There's too little memory for lots of things. The rack servers with DIMM slots might be better on this front.
• Collective operations such as reductions are annoying because of single-tile receive bandwidth limitations and exchange code size. I am sure there are some practical issues with this, but being able to run things like min/max/cross-tile sum directly in the exchange (by having ALUs take inputs from all tiles' (active) transmit lines simultaneously and output to the exchange) would be handy.
• The exchange interface should support strided send/receive (unless this is already there and my reverse-engineering AIs didn't find it). This is intuitively fairly cheap and would probably cut down on the amount of time spent copying things.

---

1. I don't use Claude Code personally because of unpleasantly low usage limits. ↩︎

2. GPT-6 has mostly fixed this. This manifested, for example, as GPT-5.6 dropping all but the immediate steps or most-recently-entered instructions for a long-running job across compactions. I was able to mitigate this somewhat by having it search its own session file. ↩︎

3. At one point, it was analyzing device/host transfers, and managed to somehow lock the card into an invalid state which did not even yield to a PCIe FLR (perhaps because the device doesn't support them...); I had to power-cycle it to unstick it. It only mentioned this offhandedly as an explanation for its work not being fully "hardware-validated" yet. ↩︎

4. I expect that OpenAI trained on this specifically. On the other hand, it has a bizarre unwillingness to read the profiles without prompting. ↩︎

5. I don't know how necessary these were, though. There were some accuracy issues when loading the real weights in the later GPT-6 version, but these were mostly resolved with tensor scaling fixes, rather than smarter quantization. ↩︎

6. Possibly this is expected behaviour because the model is creating unreasonably large numbers as an "attention sink" but their exact magnitude isn't very important. ↩︎

7. The deadlocks were triggered by bank conflicts triggered by incoming exchange data arriving later than expected, after the tile had set its incoming mux to neutral/no-incoming-data, when attempting to adjust scheduling to overlap better rather than separating all exchanges. This is because, presumably to keep wire area down, the exchange muxes are in the central area of the chip, with a significant transmission delay, not part of the tiles. Thus, to exit early from an exchange multicast, if the sender tile transmits onto the exchange to several receivers but some only need a fraction of the data, the receive mux has to be reprogrammed early while the useful data is still arriving. This had previously been conservatively blocked by scheduling. It also turned out that the hardware can do full-duplex traffic at each tile, at risk of even more banking problems and complicated stream scheduling. When I attempted to have GPT-5.6 implement this, it gave up and reverted it several times because it was making some tests fail. It's very OpenAI. ↩︎

8. I briefly tried dropping to GPT-5.5 for refactors, but this required me to pay too much attention, so I didn't continue with it. ↩︎

9. This is fixable but nobody bothers with this in public pretrains. ↩︎

10. I forgot to add an output projection as part of the attention benchmarks and by the time I remembered this it would have been slightly inconvenient to rerun everything. ↩︎

11. Specifically, loopback exchanges in place of some local copies. This is slower on its own, but as the sender can participate in its own multicast this added minimal extra time. ↩︎

12. Specifically E4M3, fixed per-tensor scales, tile-internal accumulation at FP32 exported and reduced at FP16. ↩︎

13. The idea was that starting with and locally optimizing a memory-efficient plan would produce a working result faster than aggressively searching for performance first, mostly because the earlier memory estimates were often a bit wrong and would fail at placement time if extremized against. ↩︎

14. FlashAttention is generally faster on GPU for bandwidth reasons, but on IPU it saves memory at the expense of performance. ↩︎

15. Running the same model at the same resolution, but at different batch sizes and precisions, subject to the constraint that cosine similarity of a few realistic inputs be >0.99 with the host reference. ↩︎

16. Unless there's an extremely clever algorithmic improvement not found in the SDK or known to GPT-6. ↩︎

17. This was previously done in an ugly way by GPT-5.6. I was having GPT-6 look for compiler performance improvements, and it identified this as a problem because of a sorting operation in it taking lots of CPU time. I suggested a simpler optimization of switching it to radix-sort, which worked slightly, but ultimately made it a major bottleneck when the rest of mid-to-low-level planning was optimized. ↩︎

18. We now know that these are avoidable (Sigmoid Self-Attention, various normless transformers/norm alternatives), but public models don't adopt this widely. ↩︎

19. I still don't believe it's generically "more difficult to program" than GPUs, considering Blackwell, but the tight memory constraints for this model turned lots of things into complex global optimization problems rather than local optimization. ↩︎