AI code needs catching up

The Machines Are Evolving Faster Than the AI Code: Inside the Strange New World of AI Chips

Walk into any modern laptop, smartphone, data center, or server room today and you’ll find a quiet revolution hiding under the casing. It’s not the CPU. That old workhorse is still there, doing what it has always done: running the operating system, responding to user input, making decisions, jumping between tasks, and keeping the whole machine from descending into chaos.

The revolution is happening beside it.

A new class of silicon has appeared: GPUs, NPUs, tensor processors, neural engines, and other accelerators built for a kind of computation traditional machines were never designed to handle. These chips can tear through enormous mathematical workloads at speeds that would make a CPU blush. Give them the right problem and they can perform trillions of operations every second.

And yet, in many systems, they’re barely awake. This is one of the defining paradoxes of the AI era: the hardware has already leapt into the future, but the software is still negotiating with the past.

A Different Kind of Machine

The reason is architectural. CPUs were built for logic. They excel at branching, decision-making, and unpredictable workflows. They can run an operating system, calculate a spreadsheet, process a network request, and switch to another task almost instantly. They’re designed to deal with whatever happens next, even when “whatever happens next” is impossible to predict.
A CPU is like a virtuoso soloist: flexible, precise, and capable of moving rapidly between complicated instructions.

AI chips are different. They are more like orchestras. Instead of relying on a small number of sophisticated cores, they contain vast collections of simpler compute units designed to work in parallel. Each unit may perform a relatively basic operation, but thousands of them can perform that operation at the same time, in massive synchronized waves.

They don’t want clever logic. They want repetition.

They don’t want a long sequence of unrelated instructions. They want the same mathematical pattern applied across enormous blocks of data. Most of all, they want numbers. Preferably arranged in neat, rectangular tensors. That difference sounds abstract, but it sits at the heart of the problem. A CPU is comfortable working with scalar values: individual numbers processed through a sequence of instructions. An AI accelerator is built to work with tensors: large collections of numbers organized into multidimensional grids.

Neural networks live almost entirely in that world.

The image entering a vision model is a tensor. The words entering a language model become tensors. The weights that store what the model has learned are tensors. The activations moving between layers are tensors. Even the model’s final output begins as another collection of numbers. Every layer, every weight, every attention calculation, and every prediction depends on mathematical operations performed across these structures.

AI chips are built to devour them.

The Chip Cannot Feed Itself

But there’s a catch: the hardware cannot prepare its own meal. Software has to organize the data before the accelerator can process it. It must create tensors, arrange their dimensions, place them in memory, convert them into suitable numerical formats, combine compatible operations, and transfer everything through a runtime that understands the hardware. If any part of that chain is missing, the accelerator may do very little. It’s like installing a Formula 1 engine in a family car. The engine may be extraordinary, but the gearbox, cooling system, fuel delivery, tires, and chassis all have to be redesigned around it. Without those changes, the engine’s theoretical power is irrelevant. This is why organizations can spend heavily on AI hardware and still see disappointing results.

The chip is fine. The software is not ready.

Sometimes the model cannot run on the accelerator at all. Sometimes only a few operations are supported, forcing the rest back onto the CPU. Sometimes the accelerator completes its work quickly but then waits for data to arrive. Sometimes the cost of moving information between memory and the chip exceeds the time saved by using the chip.

In each case, the hardware is technically working. It just isn’t being used well. Peak performance looks impressive on a specification sheet. Useful performance depends on whether the entire system can keep the chip busy.

The Translation Layer

Developers do not usually communicate with AI hardware directly. They rely on layers of software that translate neural-network operations into instructions the accelerator understands. For NVIDIA, that world is closely associated with CUDA. AMD has ROCm. Apple provides Metal and its machine-learning frameworks. Intel offers tools such as OpenVINO. Microsoft supports hardware acceleration through technologies including DirectML. Other manufacturers have their own compilers, runtimes, libraries, and deployment systems.

These ecosystems are not optional extras. They are the bridges leading to the hardware. A neural network may be written in a high-level framework, but the accelerator does not understand the model in the way a developer sees it. It sees kernels, memory addresses, matrix operations, synchronization points, and streams of numerical data.

Something has to translate between those worlds.

That translation is far more complicated than it appears. The software must inspect the model, construct a computational graph, determine which operations the hardware supports, choose optimized kernels, schedule them efficiently, allocate memory, and minimize unnecessary movement of data. If the translation layer fails, the operating system will not somehow discover the accelerator’s potential. It will often route unsupported work back to the CPU, because that is the safe and familiar path software has followed for decades.

The application may still run. It may simply run far more slowly than expected.

That creates a dangerous illusion: the presence of an AI chip is mistaken for the use of an AI chip.

They are not the same thing.

The Arithmetic Has Changed

Even when the right framework is present, the software still has to change its habits. Traditional software often treats numerical precision as something to preserve. If 32-bit floating-point values are available, why use anything less? AI accelerators ask a different question: how much precision does this calculation actually need?

Neural networks can often tolerate lower-precision formats such as FP16, BF16, INT8, or even smaller representations in carefully controlled situations. These formats use fewer bits per number, which means more values can fit into memory and more calculations can be performed in parallel.

The result can be faster execution, lower energy use, and reduced memory consumption.

But reduced precision is not free.

Convert a model carelessly and its predictions may deteriorate. Sensitive layers may lose important information. Values may overflow, underflow, or become too coarsely represented. A model that worked perfectly during development may become subtly less reliable after optimization. This means quantization is not simply a switch that makes a model faster. It is an engineering process. Developers may need to calibrate the model using representative data, measure accuracy before and after conversion, keep certain operations at higher precision, and adjust the architecture to make it more tolerant of numerical compression.

AI hardware rewards approximation, but software must decide where approximation is safe.

That is a very different relationship with mathematics from the one most conventional applications were built around.

Speed Is Often a Memory Problem

The arithmetic is only part of the story. AI workloads move enormous amounts of data. A large model may contain billions of parameters, and those parameters must be loaded from memory before the chip can use them. Intermediate results must be stored, retrieved, and passed from one layer to the next.

This makes memory bandwidth one of the most important constraints in modern AI computing.

An accelerator may be capable of performing an astonishing number of operations per second, but that ability means little if the data arrives too slowly. The compute units sit idle, waiting for memory. It is a little like building a factory with thousands of workers and supplying it through a single narrow doorway. The factory’s theoretical output may be enormous. Its actual output will be determined by the doorway.

This is why optimization increasingly revolves around data movement.

Developers try to keep frequently used values close to the compute units. They reuse data already loaded into fast memory. They avoid creating unnecessary intermediate tensors. They combine several operations into a single kernel so that results do not have to be written out and read back again.

This process is known as operator or kernel fusion, but the idea is simple: if several steps can happen during one trip through memory, do them together. A model may contain a sequence of operations that looks clean and logical in source code. On the hardware, however, each separate operation can introduce another launch, another synchronization point, and another movement of data.

The mathematical work might be cheap. Moving the numbers can be expensive.

In AI computing, distance matters—even when that distance is measured in millimeters across a piece of silicon.

Small Workloads on Giant Machines

AI chips also dislike being underfed. A CPU can handle a small request efficiently because it is designed to react quickly to individual tasks. An accelerator is usually most effective when many operations can be launched together.

That is why batching matters.

Instead of asking the chip to process one image, one sentence, or one request at a time, software can collect several inputs and process them as a group. The work becomes larger, more regular, and easier to spread across the available compute units.

But batching introduces another trade-off.

Waiting for more requests improves throughput, yet it may increase latency for the first request in the queue. That is acceptable when processing millions of images overnight. It may be unacceptable for an interactive assistant expected to respond immediately. Training systems often prioritize throughput: keep the expensive hardware busy and complete as much work as possible. Interactive inference systems care more about latency: begin responding quickly, even when the workload is small or unpredictable.

The same chip can behave very differently under those conditions.

This is why “How fast is the accelerator?” is rarely the right question.

A better question is: fast at what, with what batch size, using which precision, under what memory constraints, and with how much of the model actually running on the device?

Without that context, performance numbers are closer to marketing than measurement.

The Hidden Cost of Falling Back

One of the strangest performance failures happens when most of a model runs on the accelerator but a few unsupported operations fall back to the CPU. On paper, this sounds harmless. If the accelerator cannot perform one step, let the CPU handle it.

In practice, the handover can be painfully expensive.

Data may have to be copied out of accelerator memory, converted into another format, processed by the CPU, and then copied back. The two processors may need to synchronize, meaning one waits while the other finishes. A single unsupported operation can interrupt the entire pipeline. The accelerator might spend only a few milliseconds doing its part, while the system spends far longer moving information around it. This creates situations where adding hardware acceleration makes an application slower. Not because the accelerator lacks power, but because the path between processors was never designed as part of the application.

The lesson is uncomfortable: acceleration is not about sending the expensive parts of a program to a faster chip. It is about designing an uninterrupted flow of work that suits the entire machine.

Compilers Are Becoming Part of the Hardware

In the old computing world, the boundary between hardware and software felt relatively stable. A processor executed instructions, an operating system managed the machine, and a compiler translated code.

AI has blurred those layers.

Modern machine-learning compilers do much more than translate one instruction set into another. They reshape computational graphs, fuse operators, select kernels, rewrite memory layouts, remove redundant calculations, specialize execution for known tensor dimensions, and decide which parts of a model should run where.

In effect, the compiler helps determine what the hardware becomes capable of doing.

Two systems with identical chips can produce dramatically different results because one uses a more mature compiler, a better kernel library, or a runtime with stronger support for the model’s operations. This is one reason AI performance improves even when the hardware does not change. A software update can suddenly unlock parts of the chip that were always present but poorly used.

The silicon was waiting. The compiler finally learned how to speak to it.

Portability Is the New Compatibility Problem

There is another complication: AI hardware does not speak one universal language. A model optimized for one accelerator may perform poorly on another. An operation supported by one runtime may be missing from the next. A numerical format that runs efficiently on a data-center GPU may behave differently on a laptop NPU. Memory limits, preferred tensor layouts, and optimal batch sizes vary between devices.

Developers are therefore facing a new kind of compatibility problem. The application may be portable. The performance often is not.

A model can move from one platform to another and still produce the correct answer, yet take three times longer, consume more memory, or fall back partially to the CPU. From a functional perspective, the migration succeeded. From an engineering perspective, it may have failed completely.

This is where hardware vendors gain power.

The strongest platforms are not necessarily those with the most impressive chips. They are often the ones with the best development tools, documentation, libraries, debuggers, profilers, and community support. Developers need to understand why an operation is slow, where memory is being consumed, which part of a graph is falling back, and how much of the accelerator is actually occupied.

A chip without a usable software ecosystem is not really a platform. It is an isolated piece of potential.

Training and Inference Are Different Worlds

The phrase “AI workload” also hides an important distinction. Training a model and running a trained model are not the same computational problem.

Training involves repeatedly processing data, calculating errors, and updating enormous numbers of weights. It demands high throughput, substantial memory, and efficient communication between many accelerators. Large training runs can spread across entire clusters, where the network connecting the chips becomes almost as important as the chips themselves. Inference is the act of using the finished model to make a prediction or generate an output. It can happen in a data center, on a laptop, inside a phone, or at the edge of a network. Inference may prioritize low latency, low power consumption, privacy, or the ability to operate without an internet connection.

A chip designed for one of these worlds may be poorly suited to the other.

The enormous GPU cluster training a model has different responsibilities from the small neural processor performing speech recognition on a phone. Both are called AI accelerators, but the similarity can be misleading. One is built to sustain a computational storm. The other may be designed to wake briefly, finish a narrow task, and return to an energy-saving state. Software has to understand those differences. A model that works beautifully in a data center may need to be compressed, quantized, partitioned, or partially redesigned before it makes sense on a personal device.

Moving AI closer to the user is not just a deployment decision. It often requires a new model and a new way of thinking about the workload.

The Energy Equation

Performance is not the only reason these chips matter. General-purpose processors can perform AI calculations, but they may use far more energy to do so. Specialized accelerators improve efficiency by dedicating more of their silicon to the mathematical patterns neural networks use most often. This is especially important in data centers, where power consumption affects operating costs, cooling systems, infrastructure planning, and the number of models that can run inside a fixed energy budget.

It matters just as much on personal devices.

An AI feature that drains a laptop battery or overheats a phone is not a useful feature, no matter how impressive the model appears in a demonstration. Local AI has to coexist with every other responsibility of the device. It must share memory, operate within thermal limits, and avoid turning occasional intelligence into permanent power consumption.

The most important metric may therefore not be raw speed, but useful work per watt.

A slower accelerator that completes the task with a fraction of the energy may be the better machine. A smaller model that runs locally may be more valuable than a larger model that requires a constant connection to a distant server. The AI era is not simply increasing the demand for computation. It is forcing the industry to decide where that computation should happen and how much energy each answer is worth.

The Operating System Is Still Learning

For AI accelerators to become truly ordinary, operating systems must learn how to manage them as naturally as they manage CPUs. That transition is still underway. A CPU is a shared resource with decades of scheduling logic behind it. The operating system understands how to divide its time, prioritize applications, isolate processes, and respond when demand changes.

AI accelerators introduce new questions.

What happens when several applications want the NPU at once? Which workload receives priority? How should models share limited accelerator memory? Can one application’s data leak through a shared runtime? Should background AI tasks be paused to preserve battery life? When should work run locally, and when should it move to the cloud?

These are not merely hardware questions. They are policy questions embedded in software.

Until operating systems manage AI resources more consistently, many applications will continue to bring their own runtimes, libraries, model formats, and scheduling assumptions. That duplication makes deployment larger, maintenance harder, and performance less predictable.

The hardware may be standardized inside the device long before the software agrees on how to use it.

The Benchmark Illusion

The growing complexity of AI systems also makes them unusually easy to benchmark badly. A vendor can advertise the theoretical number of operations a chip can perform, but theoretical throughput says little about whether a real model can reach it. The model may be limited by memory bandwidth. Its tensor dimensions may not align with the hardware. Some operations may not use the fastest numerical format. The workload may be too small to occupy the device. Even two benchmarks that claim to test the same model may use different precision levels, batch sizes, input lengths, optimization settings, or quality targets.

Both results can be technically accurate while describing completely different realities. For users and organizations, the meaningful benchmark is the actual application.

How quickly does it respond? How many requests can it serve? How much power does it consume? How much memory does it require? Does optimization change the quality of the output? Does performance remain stable under real workloads rather than carefully prepared demonstrations? AI hardware has made simple comparisons difficult. A larger number does not automatically mean a better experience.

The chip matters. The workload matters more.

For Decades, Hardware Adapted to Software

This is where the story becomes historically interesting. For decades, hardware bent itself around software. Processor makers added instruction sets to support existing applications. They preserved compatibility with old binaries. They built prediction systems and caches to hide inefficient code. They tolerated decades of accumulated assumptions because breaking old software was considered unacceptable. Modern CPUs became extraordinary machines partly because they learned to disguise the cost of software’s habits.

AI chips are less forgiving. … They demand that software evolve.

They force developers to rethink how data moves, how models are structured, how memory is allocated, how operations are grouped, and how precision is handled. They reward regularity and punish unpredictability. They excel when an application exposes large amounts of parallel work and become strangely unimpressive when it does not.

This changes the balance of power between code and silicon.

The developer can no longer assume the hardware will make inefficient software fast. The software must reveal the shape of the problem clearly enough for the accelerator to solve it. In return, the hardware offers something remarkable: enormous performance and efficiency, but only on its own terms.

An Expensive Chip Waiting for Work

The result is a strange moment in computing history. We have machines capable of extraordinary performance, but only a fraction of software knows how to use them well. Manufacturers stamp “AI-powered” onto laptops, phones, cameras, and appliances, inviting users to imagine that intelligence is flowing constantly through the device.

In reality, the accelerator may be sitting idle.

Perhaps the applications have not been updated. Perhaps the operating system lacks the right scheduling support. Perhaps the model uses unsupported operations. Perhaps the software vendor decided that maintaining several hardware-specific paths was too expensive. Perhaps the CPU is already fast enough for the tiny workload, making acceleration pointless.

This does not necessarily mean the chip is useless. New hardware often arrives before its software ecosystem matures. Graphics processors followed a similar path: specialized devices gradually became programmable platforms, then essential engines for games, scientific computing, media production, and eventually AI itself.

But it does mean consumers should be skeptical of hardware labels. Owning an AI accelerator is not the same as benefiting from one. The value appears only when applications are designed to use it, runtimes support it, models fit within its limits, and the resulting improvement is meaningful to the person using the device.

Until then, the chip is a promise.

The Companies Pulling Ahead

Meanwhile, the companies that understand the new relationship between software and AI hardware are pulling ahead. Their advantage does not always come from owning better silicon. Often, they are using the same accelerators available to everyone else.

The difference is in the system surrounding them.

They profile real workloads. They measure where time and memory are being spent. They choose numerical precision deliberately. They keep data on the accelerator instead of moving it unnecessarily. They batch requests when throughput matters and reduce batching when latency matters. They redesign unsupported operations. They treat the compiler, runtime, model, and hardware as parts of one machine. Their models run faster. Their infrastructure costs fall. Their applications support more users. Their devices last longer on a charge. Their systems feel modern, even when the underlying hardware is identical to everyone else’s.

The difference is not simply the chip. It is the code—and the willingness to change it.

Software Has to Grow Up

This is the quiet revolution happening under the hood. It is not the flashy demonstrations or the endless wave of products described as intelligent. It is the deeper architectural shift forcing software to grow up.

AI chips have changed the rules of computation. They have introduced new forms of parallelism, new numerical compromises, new memory constraints, and new dependencies between hardware and compilers. They have turned data movement into a central performance problem and made software ecosystems nearly as important as transistor counts. They have also exposed something the computing industry could previously afford to ignore: general-purpose software carries an enormous amount of inefficiency. For years, faster CPUs concealed much of that waste. AI accelerators expose it. They make the difference between code that merely runs and code that understands the machine beneath it impossible to ignore.

The future will not belong automatically to the company with the largest model or the chip with the highest theoretical performance. It will belong to the systems that connect every layer successfully: model, compiler, runtime, memory, accelerator, application, and user. Because AI chips are not magical devices that make ordinary software intelligent.

They are specialized machines waiting for software capable of meeting them where they are.

The hardware has already arrived.

Now the code has to catch up.