Neural Networks: Foundations & Architectures

At its core, a neural network is not a mystical thinking machine; it is a parameterized computation graph composed of stacked affine transformations and non-linear mappings that converts an input vector x into a prediction or latent embedding y.

Consider a tabular customer churn predictor:

Before assembling large networks, understand the atomic building block: the artificial neuron (often historically termed a perceptron). An artificial neuron takes multiple numerical inputs x₁, x₂, ..., xₙ, multiplies each by a corresponding learned weight w₁, w₂, ..., wₙ, sums the results, and adds a scalar bias b:

z = (w₁ · x₁) + (w₂ · x₂) + (w₃ · x₃) + ... + (wₙ · xₙ) + b = ∑ (wᵢ · xᵢ) + b = wᵀx + b

Why weights matter:

• Large positive weight: Strong positive correlation; when this feature increases, the neuron fires higher.
• Negative weight: Inhibitory effect; when this feature increases, the neuron activation decreases.
• Near-zero weight: Feature is largely ignored by this neuron during the forward pass.

Every neural network operates with two distinct classes of numbers:

Individual neurons are limited: a single linear neuron can only compute a flat hyperplane. By grouping neurons intolayers, an architecture computes multiple simultaneous projections:

• Input Layer: Holds raw features. Size is governed strictly by the dataset (e.g. 5 features).
• Hidden Layers: Perform learned affine transformations. You configure their count (depth) and width (neuron count).
• Output Layer: Produces the task prediction. Size is governed strictly by target format (1 for regression, K for multi-class).

A layer is described as fully connected (or dense) because every input feature connects to every neuron in the layer. In PyTorch, this is instantiated as torch.nn.Linear(in_features, out_features, bias=True).

# Official PyTorch affine transformation formula: y = x · Aᵀ + b # Tensor Dimensions: # x: Shape (*, in_features) # A (weight): Shape (out_features, in_features)  <-- Transposed during multiplication! # b (bias): Shape (out_features) # y (output): Shape (*, out_features)

layer = nn.Linear(in_features=4, out_features=3, bias=True)
layer.weight.shape -> torch.Size([3, 4])  # (3 rows, 4 cols)
layer.bias.shape   -> torch.Size([3])
Total parameters   -> 15 (12 weights + 3 biases)

In production deep learning, 90% of architectural bugs are tensor shape mismatches. PyTorch models process mini-batches of inputs. The tensor flow through stacked linear layers must strictly adhere to the inner-dimension matching rule of matrix multiplication:

In deep learning, forward propagation (or the forward pass) is the calculation that moves data forward through each layer of the network to produce a model prediction:

x (input tensor) ↓ Layer 1: z₁ = x · W₁ᵀ + b₁ ↓ [Activation Placeholder: a₁ = f(z₁)]  <-- Covered in next topic! ↓ Layer 2: z₂ = a₁ · W₂ᵀ + b₂ ↓ Output: ŷ (logits / prediction tensor)

Every linear layer connecting n_in inputs to n_out neurons has:

Weights = n_in × n_out Biases  = n_out Total Parameters = (n_in + 1) × n_out

In PyTorch, custom neural networks subclass torch.nn.Module. Building a model requires two essential steps:

• __init__(self): Define your layers (e.g. self.fc1 = nn.Linear(...)). Always call super().__init__() first!
• forward(self, x): Define how data moves through the layers. Never call model.forward(x) directly in your code; always invoke the instance as model(x), which triggers PyTorch hooks and autograd mechanisms.

import torch
import torch.nn as nn

class CustomClassifier(nn.Module):
    def __init__(self, in_features=4, hidden_dim=8, num_classes=2):
        super().__init__()
        # Define layer transformations
        self.fc1 = nn.Linear(in_features, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Input tensor shape: (batch_size, 4)
        x = self.fc1(x)  # -> (batch_size, 8)
        # Activation placeholder: e.g., torch.relu(x)
        x = self.fc2(x)  # -> (batch_size, 2)
        return x

# Instantiate model
model = CustomClassifier()
print(model)

# Test forward pass with synthetic batch
sample_batch = torch.randn(16, 4)
logits = model(sample_batch)
print("Output shape:", logits.shape)  # torch.Size([16, 2])

When your neural network is a straight, unbranching line of transformations where the output of Layer N flows directly into Layer N+1, torch.nn.Sequential eliminates boilerplate code:

import torch.nn as nn

# Linear sequential feed-forward container
model = nn.Sequential(
    nn.Linear(4, 8),
    nn.ReLU(),
    nn.Linear(8, 2),
)

print(model)

PyTorch automatically registers learnable parameters when layers are assigned to self inside __init__(). You can inspect them via two primary methods:

Use this interactive sandbox to configure network depth, width, and input features. When you click Run Forward Pass, the workbench calculates the exact affine transformations across every layer using genuine vector-matrix operations:

Select a broken deployment or model implementation below, inspect the traceback, diagnose the root architectural bug, and apply the correct PyTorch fix:

model = nn.Linear(in_features=10, out_features=4)
x = torch.randn(32, 8)
y = model(x)  # Throws RuntimeError!

RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x8 and 10x4)

In this practical project, you configure an enterprise tabular neural network that evaluates customer retention. The input comprises 5 normalized numerical features:[Age, Monthly Spend, Support Tickets, Months Active, Usage Frequency]. The output computes two classification logits: [score_retained, score_churn].

import torch.nn as nn

# Churn prediction architecture
churn_model = nn.Sequential(
    nn.Linear(in_features=5, out_features=8),
    nn.Linear(in_features=8, out_features=2)  # [stay_logit, churn_logit]
)

No single neural architecture is universally "best." Architecture selection involves balancing representational capacity against latency budgets and memory constraints:

Every modern AI architecture—from Vision Transformers (ViT) and CNNs to Large Language Models (GPT-4, Llama 3) and diffusion image generators—is built upon the exact same principles you learned in this module: