Building Neural Networks from Scratch: From Perceptron to XOR

This article builds upon the concepts introduced in The Perceptron, extending single-layer learning to multi-layer neural networks.

The perceptron we explored previously can solve linearly separable problems - cases where we can draw a straight line to separate two categories. But what happens when the data isn't linearly separable? This is where multi-layer neural networks shine, and the classic example that demonstrates their power is the XOR problem.

The XOR Problem: Why Single Layers Aren't Enough

The XOR (exclusive OR) function represents a fundamental limitation of single-layer perceptrons. XOR outputs 1 when exactly one input is 1, but outputs 0 when both inputs are the same:

A	B	XOR
0	0	0
0	1	1
1	0	1
1	1	0

If you plot these points on a 2D plane, you'll discover something important: no single straight line can separate the 1s from the 0s. This non-linear separation requires a more sophisticated approach - multiple layers working together.

---

From Single Neurons to Neural Networks

The Multi-Layer Approach

A multi-layer neural network solves the XOR problem by combining multiple simple decision boundaries. Think of it this way:

1. First layer: Creates multiple linear decision boundaries
2. Second layer: Combines these boundaries to form complex, non-linear regions
3. Output layer: Makes the final classification

For XOR, we need at least one hidden layer with two neurons to create the necessary decision boundaries.

Mathematical Foundation: Matrix Operations

While a single perceptron computes:

$$y = f\left(\sum_{i=1}^{n} w_i x_i + b\right)$$

A neural network layer processes multiple neurons simultaneously using matrix operations:

$$\mathbf{z}^{[l]} = \mathbf{W}^{[l]} \cdot \mathbf{a}^{[l-1]} + \mathbf{b}^{[l]}$$$$\mathbf{a}^{[l]} = f^{[l]}(\mathbf{z}^{[l]})$$

Where:

• $\mathbf{a}^{[l]}$ is the output of layer $l$
• $\mathbf{W}^{[l]}$ is the weight matrix connecting layers
• $\mathbf{b}^{[l]}$ is the bias vector
• $f^{[l]}$ is the activation function

This matrix formulation allows us to compute all neurons in a layer simultaneously, making the implementation both efficient and elegant.

---

Designing Our Neural Network Framework

Before diving into implementation details, let's design how we want to use our neural network library:

This design provides flexibility in specifying layer architectures while keeping the interface clean and intuitive.

---

Building the Network Infrastructure

Layer Specifications

Our framework supports multiple ways to specify layers, providing flexibility while maintaining type safety:

The Network Class

The Network class serves as the coordinator, managing layers and orchestrating the training process:

---

Forward Propagation: From Input to Output

Forward propagation is the process of passing input data through the network to generate predictions. In our Network class, this is handled by the __call__ method:

This simple implementation demonstrates the power of our design: each layer transforms the input and passes it to the next layer, creating a computational pipeline.

---

The Layer Implementation

Automatic Differentiation with SymPy

One of the most elegant features of our implementation is automatic differentiation using SymPy. This eliminates the need to manually derive and implement derivatives:

This function takes a symbolic activation function (like lambda x: 1/(1 + exp(-x)) for sigmoid) and automatically generates both the function and its derivative in multiple computational backends.

The Layer Class

Key Implementation Features

Weight Initialization: The layer uses He initialization (np.sqrt(2 / fan_in)) which works well with ReLU-like activation functions and helps prevent vanishing/exploding gradients.

Forward Pass: The __call__ method implements the mathematical formula we discussed:

1. Adds bias term if specified
2. Computes pre-activation values: z = W @ x + b
3. Applies activation function: a = f(z)
4. Stores intermediate values for backpropagation

Bias Handling: Instead of maintaining separate bias vectors, the implementation appends a 1 to the input and includes bias weights in the main weight matrix. This simplifies the matrix operations.

---

Backpropagation: Learning from Mistakes

Backpropagation is how neural networks learn - by propagating errors backward through the network and adjusting weights to minimize those errors.

The Mathematics

The backpropagation algorithm uses the chain rule to compute gradients:

1. Output gradient: $\frac{\partial L}{\partial \mathbf{z}^{[l]}} = \frac{\partial L}{\partial \mathbf{a}^{[l]}} \odot f'^{[l]}(\mathbf{z}^{[l]})$

2. Weight gradient: $\frac{\partial L}{\partial \mathbf{W}^{[l]}} = \frac{\partial L}{\partial \mathbf{z}^{[l]}} \cdot (\mathbf{a}^{[l-1]})^T$

3. Bias gradient: $\frac{\partial L}{\partial \mathbf{b}^{[l]}} = \frac{\partial L}{\partial \mathbf{z}^{[l]}}$

4. Input gradient: $\frac{\partial L}{\partial \mathbf{a}^{[l-1]}} = (\mathbf{W}^{[l]})^T \cdot \frac{\partial L}{\partial \mathbf{z}^{[l]}}$

Layer-Level Backpropagation

The back method in our Layer class implements these equations:

Network-Level Backpropagation

The Network class coordinates backpropagation across all layers:

This elegant implementation demonstrates how the chain rule propagates through the network:

1. Start with the error at the output
2. Each layer computes its gradients and updates its parameters
3. Each layer passes the gradient to the previous layer
4. The process continues until we reach the input

---

Solving XOR: Putting It All Together

Now let's use our neural network to solve the XOR problem:

Understanding the Results

When you run this code, you should see the network successfully learn to classify XOR inputs. The decision boundary visualization will show how the network creates non-linear regions to separate the different classes - something impossible with a single-layer perceptron.

The key insight is that the hidden layers learn to create intermediate representations that make the final classification possible. Each hidden neuron learns a different linear boundary, and the output layer combines these to create the complex XOR decision boundary.

---

Key Insights and Next Steps

What We've Accomplished

1. Built a complete neural network from scratch using only NumPy and SymPy
2. Implemented automatic differentiation eliminating manual derivative calculation
3. Demonstrated multi-layer learning solving the classic XOR problem
4. Created a flexible framework supporting different architectures and activation functions

The Power of Multi-Layer Networks

Our XOR solution demonstrates several crucial concepts:

• Non-linear decision boundaries: Multiple layers can solve problems that single layers cannot
• Feature learning: Hidden layers automatically learn useful intermediate representations
• Gradient-based optimization: Backpropagation efficiently trains complex networks
• Composability: Simple components (neurons) combine to solve complex problems

Looking Forward

This foundation opens the door to understanding modern deep learning:

• Deeper networks: More layers can learn more complex patterns
• Different architectures: CNNs, RNNs, Transformers all build on these principles
• Advanced optimizers: Adam, RMSprop improve on basic gradient descent
• Regularization: Dropout, batch normalization improve training

The neural network you've built here contains the essential DNA of every modern deep learning system. While production frameworks like PyTorch and TensorFlow add many optimizations and conveniences, the core principles remain the same: forward propagation, backpropagation, and gradient descent working together to learn from data.

Understanding these fundamentals gives you the insight needed to debug training issues, design new architectures, and push the boundaries of what's possible with neural networks.

---

Bonus

As a "showcase", here is a small javascript neural network implementation and it's decision boundary.

---