A Shared Subcircuit Lets LLMs Count Down Across Tasks

Researchers identified a "countdown subcircuit" in Llama-3.1-70B-Instruct that compares current position to a goal length and estimates remaining tokens. The subcircuit was isolated in a controlled fixed-length sentence task, then found to use an identical motif previously identified in a separate frontier LLM, suggesting cross-model sharing. Unsupervised probing on natural language data revealed the subcircuit operates across diverse tasks, including cases where target length is inferred from context rather than explicitly stated.