Operating Systems: Kernel Architecture, Process Scheduling & Memory Management

If software applications were allowed to talk directly to physical hardware, computing would instantly descend into chaos. If Chrome, your Python script, and a video game all tried to write voltage directly to the exact same RAM transistors or storage sectors simultaneously, data would be permanently corrupted within milliseconds.

The Operating System (OS) exists to solve two grand challenges:

Operating System vs. Kernel vs. User Applications

It is critical not to confuse the Kernel with the entire Operating System:

Modern microprocessors physically implement hardware privilege levels known as Protection Rings. On x86 and ARM64 architectures:

The System Call (Syscall) Lifecycle

Whenever your Python code needs to read a dataset file, allocate RAM, or send a network packet, it must transition from User Mode to Kernel Mode via a System Call:

1. Python User Code:
   data = file.read(4096)

2. Standard C Library Wrapper (libc):
   Places the syscall number (e.g., SYS_read = 0 on x86-64) into register RAX,
   places arguments (file descriptor, buffer address, count) into RDI, RSI, RDX.

3. CPU Hardware Trap:
   Executes the specialized hardware instruction: `syscall` (x86-64) or `svc` (ARM64).
   The CPU switches privilege ring from Ring 3 to Ring 0 and jumps to the kernel syscall table.

4. Kernel Execution:
   The kernel verifies permissions, reads the data from the NVMe storage driver or page cache,
   and copies bytes into the process's virtual buffer.

5. Return to User Mode:
   Kernel executes `sysret` or `eret`. CPU switches back to Ring 3. Python resumes execution!

Let's establish precise distinctions between three terms that are frequently confused:

What Actually Happens When You Run: python app.py?

Multi-Process vs. Multi-Threaded: The Architectural Trade-Off

Why does Python AI engineering frequently use multiprocessing instead of multithreading?

CPython has a mutex known as the Global Interpreter Lock (GIL) that prevents multiple threads from executing Python bytecode simultaneously on separate CPU cores. Therefore, to achieve true multi-core parallel CPU preprocessing, PyTorch DataLoader spawns separate Processes (each with its own Python interpreter and GIL). The trade-off: processes cannot share memory directly without IPC or shared memory (/dev/shm).

If you open Task Manager or htop right now, you will notice 200+ processes and 3,000+ threads active on your machine. Yet your computer only has 8 to 16 physical CPU cores! How does this work without the computer freezing?

The answer is the Preemptive CPU Scheduler. The scheduler assigns each runnable thread a tiny slice of CPU execution time (called a Time Quantum, typically 2ms to 20ms). When the quantum expires, a hardware timer interrupt fires, returning control to the kernel. The kernel pauses the current thread, executes a Context Switch, and dispatches the next runnable thread.

Virtual Memory is one of the most brilliant software-hardware engineering co-designs in history. It completely decouples the memory addresses a program uses from the actual physical DRAM chips on the motherboard.

The Nightmare Scenario for AI Engineers: Swap Thrashing

What happens when your Python script tries to allocate 12 GB of tensor embeddings on a laptop with only 8 GB of RAM?

The OS cannot allocate physical RAM that does not exist. To prevent an immediate crash, the kernel invokes Paging / Swapping: it picks inactive 4KB pages belonging to background apps and writes them to a hidden file on your SSD (the swap space / pagefile).

A storage drive (NVMe SSD) is just a massive array of physical flash sectors (blocks). The File System (such as Linux ext4, Windows NTFS, or Apple APFS) provides the organizational tree of files, directories, access permissions, and metadata (inodes).

The OS Page Cache: Why Spare RAM Is Never Wasted

New developers often panic when they inspect Linux memory and see: "15.8 GB / 16 GB RAM used!"

In reality, modern operating systems follow an ironclad design rule: Unused RAM is wasted RAM. When physical memory is not needed by running processes, the OS kernel automatically uses all remaining RAM as a high-speed Page Cache for recently accessed disk files.

When your PyTorch DataLoader reads 50,000 image files during Training Epoch 1, the files are streamed from the SSD into the Page Cache. When Epoch 2 begins, the kernel serves those exact same files directly from RAM memory buffers at 60+ GB/s without issuing a single physical read request to the SSD! If a running process suddenly needs more RAM, the kernel instantaneously discards clean cached pages in microseconds.

Operating system security is built around the principle of Least Privilege and hardware-enforced boundaries:

The AI Engineer's Operating System Rulebook

What to Learn Next

Now that you understand physical hardware (Computer Fundamentals) and how the Operating System manages processes, CPU scheduling, and memory, the next step is connecting machines together: Internet & Networking Basics.

In modern AI engineering, no serious model is trained on a single machine. High-throughput distributed training (FSDP, DeepSpeed) and cloud LLM APIs require a solid grasp of IP addressing, TCP/UDP sockets, DNS resolution, and HTTP/HTTPS protocol headers.