Consumer AR glasses

AR glasses are obviously very cool in concept, but as consumer products they have not done well. It's been twelve years since Google Glass, and the technology has failed to find wide adoption, though a few people use the existing systems for niche applications like teleprompters and FPV drones. This article by Fleetwood is focused on display tech limitations, but I believe most of the problems are elsewhere.

Note that I'm going to be talking specifically about optical AR, i.e. glasses which combine light from the outside world with overlays optically, rather than using a camera and compositing images digitally like the Apple Vision Pro. This is less flexible, but I think social barriers (to wearing a large opaque helmet constantly outside) and focusing issues favor optical AR in the near term^[1].

Optical systems have a critical flaw: for optics reasons, it's not possible to occlude objects, only to draw on top of them. This precludes replacement of objects and limits practical applications to annotations (on top of objects) or HUDs (overlays not anchored to anything in particular). Current hardware is mostly limited to HUDs due to FOV limitations, and low-cost options comprise "display glasses" such as those from Xreal and Epson, which aren't very useable outdoors due to brightness and transmissivity issues but which have high-resolution multicolor displays, and various normal-looking glasses with integrated low-resolution green displays. This will improve, of course, though it seems like known solutions have nasty tradeoffs.

Given this, I think AR is primarily competing with:

• Smartphones: glasses are much less obtrusive to view than a phone, and thus may be preferred even if image quality is worse.
• Smartwatches: HUDs are faster to check than a watch, and plausibly nicer to read than watch displays, which are small and inconveniently placed.
• Earpieces: while not commonly considered an alternative, they are already widely used as always-on devices, and could provide you with useful context in the background. AR glasses have the advantages of higher bandwidth, scannability (you can glance at them, or read some distance back) and interfering less with the physical world – most people find it difficult to listen to two people speaking at once, but can switch to and from reading things more quickly^[2]. Also, they can have cameras.

Replacing each of these requires somewhat different hardware and supported usecases: a "phone on your face" product for multimedia on the go can get away with less outdoor usability (while still being more convenient than a phone) but needs high-resolution, multicolor display for users to respect it; something to replace a smartwatch or hypothetical context earpiece needs to be light, comfortable, durable and battery-equipped enough to wear all the time, and benefits from a large set of onboard sensors to infer what you want to know.

The former is problematic for power reasons: as Fleetwood describes, mass/volume constraints cut battery capacity to far below a phone's, but almost all the energy-consuming parts of phones must remain onboard^[3]. I haven't found a breakdown of modern smartphone power consumption, but glasses with the featureset people expect from this still need a high-powered processor and modem, with the only difference being the display – which is in a worse position, because it has to be reasonably bright against real-world light across a larger field of view despite optics losses. As a result of this, all the "display glasses" I'm aware of work as wired peripherals for other computers.

Another problem is input, particularly when replacing a phone. Current systems use their host device's inputs or a dedicated controller (not very portable), capacitative touch sensors on the glasses themselves (inconvenient to use and very low-bitrate), head movements (only usable for pointing) and sometimes voice. Other possible methods include gesture sensing (via camera, as in the Vision Pro, or radar), eye trackers and surface electromyography, as Meta is working on. sEMG reads the nerve signals controlling muscles via electrical potentials on the skin. I expect sEMG and voice to win: making large gestures for a long time is fatiguing, and (according to discussions with a former Meta engineer who worked on this) sEMG is even able to read consciously triggered subthreshold signals which do not cause muscle movement^[4].

As for more smartwatch-replacing products (always-on unobtrusive information overlays), the limit here is applications. People buy smartwatches for fitness tracking applications glasses don't help with as well as for things like notifications, and "read the subject lines of your emails two seconds faster" is not an especially persuasive pitch. Features like navigation seem more useful, but still marginal without a wide portfolio of them. Real-time written language translation overlays could be valuable but impose huge compute costs for image recognition.

There are claims that the "killer app" is LLM integration, but I'm not convinced: LLM agent systems remain too dumb to be workable by themselves, and the lines of research producing them seem focused on having you interact using natural language as you might delegate something to a human, which works about as well over audio. To be very useful, the AI has to have a unified set of interfaces to helpful tools for you – nobody seems interested in doing that integration work and AIs cannot yet operate the human interfaces directly^[5]. This proposal for contextually aware applications is compelling, but much harder than merely having the hardware, and with user modelling of this quality I think you could substitute other sensors and IOs effectively. It is possible that people will want ubiquitous AI companionship even without significant practical applications.

Why do I want AR glasses despite this, other than the fact that they are in science fiction and therefore required? I think there are useful applications for relatively non-flashy systems if you are willing to customize heavily and write lots of personalized software, which I generally am. Old work like Northpaw has demonstrated the utility of always-on inputs for integrating things into your cognitive maps, and modern electronics gives us a ridiculous array of sensors available at very little cost. It might also be possible to scan rough semantics of text into your mind more effectively than via normal reading, but this hardly needs the glasses. With good enough input devices, there are other handy applications like having GNU units^[6] on tap everywhere.

Mass-market appeal probably requires reduced costs, better batteries and onboard compute, and, most importantly, a new reason to use them.