Automatic differentiation.

BEFORE YOU BEGIN

Make the chain rule concrete.

Every differentiable operation computes a result and records how to send a gradient back to its inputs. Follow the forward graph first, then walk the same operations backward.

Observe accumulation, then reset it

Run this complete example after installation. Rebuilding the same scalar expression creates a fresh graph, while the leaf tensor keeps its accumulated gradient until you clear it.

import everyo as eo

x = eo.tensor([2.0], requires_grad=True)
(x * x).backward()
print(x.grad)  # [4.]

(x * x).backward()
print(x.grad)  # [8.] -- contributions accumulate

x.zero_grad()
(x * x).backward()
print(x.grad)  # [4.]

The computation graph

Every differentiable operation does three things:

1. computes the forward result,
2. captures whatever it needs for the backward pass in a closure,
3. attaches a Node recording the inputs and that closure.

def multiply(a, b):
    left, right = _binary_inputs(a, b, "multiply")
    data = _dispatch("multiply", device, left.data, right.data)

    def backward_fn(gradient):
        return (
            unbroadcast(gradient * right.data, left.shape),
            unbroadcast(gradient * left.data, right.shape),
        )

    return _make(data, (left, right), "multiply", backward_fn)

The closure implements the product rule: d(ab)/da = b and d(ab)/db = a, each multiplied by the incoming gradient (the chain rule).

Nodes are created only when gradient mode is enabled and at least one input requires gradients. Inference therefore builds no graph at all.

The backward pass

Tensor.backward() seeds the output gradient (ones, for a scalar), walks the graph in reverse topological order and accumulates gradients into each tensor's .grad.

for node in topological_order(root):
    incoming = gradients.get(id(node))
    if node.requires_grad and (node.is_leaf or node.retains_grad):
        node.accumulate_grad(incoming)
    for parent, parent_grad in zip(node.grad_node.parents, node.grad_node.backward_fn(incoming)):
        gradients[id(parent)] = gradients.get(id(parent), 0) + parent_grad

Two details matter:

• The traversal is iterative. A recursive walk would hit Python's recursion limit on deep graphs; tests/test_autograd.py checks a 2000-operation chain.
• Gradients are summed, not replaced. When a tensor feeds several operations, every path contributes. That is why x + x gives dx = 2, and why you must call zero_grad() between steps unless you want accumulation.

Broadcasting

NumPy broadcasting means a (3,) bias can be added to a (32, 3) activation. The gradient then arrives with the broadcast shape (32, 3) and must be summed back down to (3,) — summing over the expanded axes is exactly the derivative of broadcasting. unbroadcast() does this, and every broadcasting operation routes its gradients through it.

Only leaves keep gradients

Intermediate tensors do not retain .grad, which keeps memory bounded. Call .retain_grad() on one if you want to inspect it:

hidden = (x * 3).retain_grad()
(hidden * 2).backward()
print(hidden.grad)  # [2.]

Turning it off

with eo.no_grad():
    predictions = model(features)  # no graph is built

no_grad is thread-local, works as a decorator, and restores the previous mode on exit. enable_grad() re-enables tracking inside a no_grad block.

Supported operations

Gradients are implemented and finite-difference tested for: add, subtract, multiply, divide, negate, power, exp, log, sqrt, abs, clip, matmul (including the vector cases), dot, sum, mean, max, min, variance, reshape, transpose, flatten, concatenate, stack, indexing, ReLU, sigmoid, tanh, softmax and log-softmax.

How correctness is verified

For a scalar function f, the central difference

(f(x + h) - f(x - h)) / 2h

approximates df/dx with error O(h²). tests/test_autograd.py compares every analytic gradient against this approximation in float64. If a backward pass were wrong, the comparison would fail — which is exactly what happened, and got caught, while this engine was being written.

What is not implemented in v0.1.0

• Second-order derivatives (the graph is not differentiable itself).
• In-place operations that would invalidate saved tensors.
• Gradients through integer or boolean tensors (these cannot require grad).