Taalas takes on the GPU: hardwired logic over flexibility for 17,000 tokens per second

For the past decade, the entire AI industry has rested on one unspoken axiom: silicon has to be flexible. Models change every week, architectures evolve every quarter, and only programmable GPUs can keep up with that pace. Toronto-based startup Taalas thinks this exact logic has led the industry into a dead end. Its radical alternative: hardwired chips that can do nothing except inference, but do it at 17,000 tokens per second.

To grasp the scale of that claim, it helps to recall the context. Modern GPUs, from the NVIDIA H100 to the latest Blackwell parts, are essentially supercomputers on a chip, capable of running arbitrary computations. Their architecture inherits decades of graphics processor development: thousands of programmable cores, complex memory hierarchies, flexible data buses. That versatility lets the same hardware handle training giant models, inference, and scientific simulations. But versatility comes at a price — in power consumption, latency, and cost. Every clock cycle spent decoding instructions and managing data flows is energy and time that doesn't go into the actual matrix multiplication.

This is exactly where Taalas aims. The company designs chips with compute paths wired directly into the silicon — so-called hardwired logic. The chip doesn't interpret a program on the fly; it physically embodies specific operations of the transformer architecture: matrix multiplications, attention functions, normalization. In effect, instead of a general-purpose processor, you get a circuit that does exactly one thing, but does it with minimal overhead.

The approach isn't new in principle. ASICs (application-specific integrated circuits) have long been used in cryptocurrency mining, telecommunications, and video processing. Back in 2016, Google introduced the TPU, a tensor processor also specialized for neural network workloads, though it retains a degree of programmability. Taalas, however, appears to go further, pushing specialization as far as possible in pursuit of maximum per-token performance.

The 17,000 tokens per second figure deserves a closer look. For comparison, typical inference of a large language model on a single H100-class GPU produces anywhere from a few hundred to a few thousand tokens per second, depending on model and batch size. If Taalas really hits the claimed speed at comparable model quality and size, it could mean a multifold drop in inference costs — the biggest expense for companies running AI services in production. Today it's inference, not training, that drives the economics of most AI products: every ChatGPT query, every Copilot call, every image generation is money spent on GPU time.

The approach does carry an obvious and serious risk. Hardwired logic means a hard dependency on a specific model architecture. If the industry moves away from transformers tomorrow toward something fundamentally different — say, state-space model architectures or hybrid approaches — Taalas chips risk becoming expensive paperweights. It's the classic specialization dilemma: you gain efficiency but lose adaptability. Google can update the software on its TPUs, and NVIDIA ships new drivers and CUDA libraries, while Taalas would have to design a new chip.

Still, the startup has a strong counterargument. The transformer architecture has dominated for eight years and shows no signs of going away soon. The core operations — matrix multiplication and the attention mechanism — have stayed fundamentally the same from GPT-2 to the latest models. On top of that, the push toward "ubiquitous inference," which Taalas puts in its slogan, assumes AI compute should become as cheap and accessible as electricity. Getting there requires specialized, energy-efficient chips rather than expensive general-purpose GPUs.

There's also market context. GPU shortages and NVIDIA's monopoly position have created strong demand for alternatives. Major cloud providers — Amazon, Google, Microsoft — are already building their own chips. Startups such as Groq, Cerebras, and SambaNova offer unconventional architectures. Taalas fits this trend, but takes the most radical position on the flexibility-versus-specialization spectrum.

The main question Taalas has to answer isn't technical but economic. Can it manufacture and update its chips fast enough to keep pace with model evolution? Can it convince customers that betting on hardwired logic is justified? If so, we may see the start of a new era in which AI inference stops being a luxury and becomes standard infrastructure. If not, it will be one more lesson in why the industry clings so tightly to flexibility.