Neural Network Architectures

What Are Neural Networks?

Feedforward Neural Network (FNN)

The Feedforward Neural Network is the most fundamental neural network architecture. Data flows in one direction from input layer through one or more hidden layers to the output layer, with no cyclic connections. Each layer's neurons are fully connected to the next layer, with activation functions (e.g., ReLU, Sigmoid) introducing non-linearity.

FNNs are suitable for simple classification and regression tasks on structured data, such as tabular data prediction and credit scoring. When data has spatial structure (images) or temporal characteristics, CNN or RNN should be used instead.

Key Pipeline

PyTorch

import torch
import torch.nn as nn

class FNN(nn.Module):
    def __init__(self, in_dim, hidden, out_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_dim, hidden),
            nn.ReLU(),
            nn.Linear(hidden, hidden),
            nn.ReLU(),
            nn.Linear(hidden, out_dim),
        )
    def forward(self, x):
        return self.net(x)

model = FNN(784, 256, 10)  # e.g. MNIST digits

Convolutional Neural Network (CNN)

CNNs use convolutional filters (kernels) that slide across input data to automatically extract local features like edges, textures, and shapes. Convolutional layers use weight sharing to drastically reduce parameter count; pooling layers reduce spatial dimensions and enhance translation invariance. By stacking multiple convolutional layers, the network progressively abstracts from low-level features (edges) to high-level semantic features (object parts, full objects).

CNNs are the foundational architecture for computer vision, widely used in image classification, object detection, and semantic segmentation. While Vision Transformers (ViT) have surpassed CNNs on some tasks in recent years, CNNs remain important due to their efficiency and mature tooling.

Core Component Pipeline

Famous CNN Models

PyTorch

import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 32, 3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(32, 64, 3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(64 * 8 * 8, 256),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(256, num_classes),
        )
    def forward(self, x):
        return self.classifier(self.features(x))

RNN / LSTM / GRU

Recurrent Neural Networks (RNN) are designed for sequential data. The hidden state h(t) at each time step depends on both the current input x(t) and the previous hidden state h(t-1), capturing temporal dependencies in sequences. However, standard RNNs suffer from severe vanishing/exploding gradient problems on long sequences.

LSTM (Long Short-Term Memory) introduces three gating mechanisms (forget gate, input gate, output gate) and an independent cell state, effectively solving the long-range dependency problem. GRU (Gated Recurrent Unit) is a simplified variant that merges the forget and input gates into a single "update gate," requiring fewer parameters and training faster, while achieving comparable performance on many tasks.

RNN vs LSTM vs GRU Comparison

PyTorch

import torch.nn as nn

class LSTMClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden, num_classes):
        super().__init__()
        self.emb = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden, num_layers=2,
                            batch_first=True, dropout=0.3)
        self.fc = nn.Linear(hidden, num_classes)

    def forward(self, x):
        x = self.emb(x)                    # (B, T, E)
        _, (h_n, _) = self.lstm(x)          # h_n: (2, B, H)
        out = self.fc(h_n[-1])              # last layer hidden
        return out

Transformer

The Transformer was proposed by Google in the 2017 paper "Attention Is All You Need." It completely eliminates recurrence, relying solely on the self-attention mechanism to model dependencies between any positions in a sequence. This enables highly parallel training, solving the sequential bottleneck of RNNs.

At its core is the Multi-Head Attention mechanism: input is projected into Query, Key, and Value matrices, and scaled dot-product attention computes how much each position should attend to every other position. Positional Encoding injects sequence order information, since self-attention itself is permutation-invariant.

Core Formula

Architecture Components

Transformer Variants

PyTorch — Self-Attention

import torch
import torch.nn as nn
import math

class SelfAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.n_heads = n_heads
        self.d_k = d_model // n_heads
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)

    def forward(self, x):
        B, T, C = x.shape
        q = self.W_q(x).view(B, T, self.n_heads, self.d_k).transpose(1,2)
        k = self.W_k(x).view(B, T, self.n_heads, self.d_k).transpose(1,2)
        v = self.W_v(x).view(B, T, self.n_heads, self.d_k).transpose(1,2)

        # Scaled dot-product attention
        attn = (q @ k.transpose(-2,-1)) / math.sqrt(self.d_k)
        attn = torch.softmax(attn, dim=-1)
        out = (attn @ v).transpose(1,2).contiguous().view(B, T, C)
        return self.W_o(out)

# Usage: attn = SelfAttention(d_model=512, n_heads=8)

GAN (Generative Adversarial Network)

Proposed by Ian Goodfellow in 2014, GANs consist of two competing networks: a Generator that attempts to produce realistic data from random noise, and a Discriminator that tries to distinguish real data from generated data. They are trained through a minimax game, until the generator produces samples indistinguishable from real data.

Training instability is the main challenge of GANs, with issues like mode collapse and oscillation. Techniques such as WGAN and Spectral Normalization have partially addressed these problems.

GAN Variants

PyTorch — Simple GAN

import torch.nn as nn

class Generator(nn.Module):
    def __init__(self, z_dim=100, img_dim=784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(z_dim, 256), nn.LeakyReLU(0.2),
            nn.Linear(256, 512), nn.LeakyReLU(0.2),
            nn.Linear(512, img_dim), nn.Tanh(),
        )
    def forward(self, z):
        return self.net(z)

class Discriminator(nn.Module):
    def __init__(self, img_dim=784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(img_dim, 512), nn.LeakyReLU(0.2),
            nn.Linear(512, 256), nn.LeakyReLU(0.2),
            nn.Linear(256, 1), nn.Sigmoid(),
        )
    def forward(self, x):
        return self.net(x)

Diffusion Models

Diffusion models are a class of probabilistic generative models based on two processes: the forward process gradually adds Gaussian noise to data until it becomes pure noise; the reverse process learns to progressively denoise to recover the original data. By parameterizing a denoising network (typically a U-Net), the model learns to predict and remove noise at each step.

Compared to GANs, diffusion models offer more stable training, higher generation quality, and better diversity, but slower inference (requiring multiple denoising steps). Sampling strategies like DDPM and DDIM can accelerate inference.

Representative Models

Architecture Comparison

Architecture Selection Guide

Choose the most suitable neural network architecture based on your task:

Related Resources

Dive deeper into machine learning and deep learning frameworks:

Frequently Asked Questions