LLMs are terrible at designing AI systems

Why it matters

The article argues that LLMs are poor at designing software that itself uses LLMs, even though they handle conventional programming well. The centerpiece is a case in which a user asked Claude to build a tool that flags contradictory memories. Claude, seeking a state-of-the-art approach, constructed an elaborate symbolic system that converted memories into logical atoms for a solver to compare mechanically. This could never work, since no analytic method can decide whether two natural-language sentences express the same proposition. Tellingly, when pressed, Claude itself acknowledged the problem was long-standing and unsolvable, yet had continued iterating on the dead end for weeks. The author recalls similar behavior in building Autodeck, where agents favored regex or keyword searches over a simple model call. Explanations considered include pre-LLM training data, reinforcement learning environments lacking API access to other models, and simply weak software-design judgment. The broader lesson is to predict where LLMs succeed and fail before committing effort.