Computer Fundamentals: Hardware Architecture, Memory Systems & Accelerators

At its most fundamental level, every computing device follows the classic Input → Processing → Memory → Storage → Output paradigm. An input device captures data (keystrokes, audio, or network packets), the processing units perform mathematical operations, high-speed memory buffers active instructions, non-volatile storage preserves state across power cycles, and output interfaces deliver results.

However, modern AI engineering requires looking past this simplistic abstraction. An AI engineer does not interact directly with raw silicon; software runs through an orchestrated hardware-software stack:

Key Physical Hardware Roles for AI Engineers

A common beginner misconception is: "CPUs are for normal software, GPUs are for 3D video games, and NPUs are for AI." This is inaccurate. Real computing architectures are differentiated by their execution philosophies and memory bandwidth profiles:

Why AI Workloads Benefit Massively from Accelerator Cores

Consider a standard neural network layer: computing activations requires a massive matrix multiplication:

Y = Activation(W · X + B)

Where:
- W (Weight Matrix): 4096 × 4096 floats (~16.7 million values)
- X (Input Vector):  4096 floats
- Total Multiply-Accumulate Operations: ~16,777,216 operations for a single layer!

CPU Execution (8 Cores):
- Executes sequentially or in small SIMD vectors (AVX-512).
- Must loop through millions of steps. High clock, but bounded parallelism.

GPU Tensor Core Execution (Thousands of Cores):
- Dispatches matrix tiles across hundreds of Streaming Multiprocessors (SMs).
- Fused Multiply-Add (FMA) instructions compute entire 16×16 matrix blocks in a single clock cycle!

Computer memory is governed by a fundamental physical constraint: you cannot have memory that is simultaneously ultra-fast, massive in capacity, and inexpensive. Silicon physics dictates that proximity to the compute cores determines speed. Consequently, computers use a hierarchical pyramid:

The Golden Rule of AI Hardware: "Capacity is a Hard Wall, Bandwidth is a Speed Dial"

In AI Engineering, understanding the difference between memory capacity and memory bandwidth is essential:

Dedicated VRAM vs. Unified Memory Architectures

Why can't you just add 16 GB System RAM and 8 GB GPU VRAM together to get 24 GB?

In standard PC architecture (Windows/Linux with discrete GPUs), the GPU and CPU have physically isolated memory pools connected only by the PCIe expansion bus. The GPU cannot access DDR5 System RAM at native VRAM speeds—it must request transfers across the PCIe bus, which is 30x to 60x slower.

In contrast, Unified Memory Architectures (such as Apple Silicon M-series chips or high-end server APUs like NVIDIA GH200) place the CPU and GPU on the same substrate, sharing a single high-bandwidth LPDDR5X memory pool (up to 128GB–192GB at 400–800 GB/s). While peak bandwidth is lower than dedicated HBM3e, it eliminates the PCIe bus barrier and allows running massive 70B+ models without multi-GPU clusters.

Computers operate exclusively on binary digits (bits: 0 or 1), physically represented by microscopic transistor voltage levels. Eight bits constitute a Byte (256 distinct values). From bytes, we construct hexadecimal notations, ASCII/Unicode text encodings, and numerical data types.

In AI Engineering, every neural network—whether processing natural language, images, or audio—converts inputs into tensors (multidimensional arrays of numerical values). The amount of memory a model requires is directly determined by its numerical precision:

The Motherboard is the physical highway system connecting CPU, memory, accelerators, and storage via etched copper traces and bus protocols:

When you open your terminal and execute python run_inference.py, an orchestrated sequence of events occurs across the operating system and physical hardware:

Hardware Bottleneck Quick Diagnostic Matrix

What to Learn Next

Now that you understand how physical silicon, memory hierarchies, and accelerators function, the next logical question is: who manages all this hardware and shares it fairly among running applications?

That is the job of the Operating System. In the next topic, you will explore how the OS kernel manages virtual memory, schedules threads across CPU cores, communicates with GPUs via device drivers, and coordinates processes.