Legible by design
You can follow one number from loss.backward() through the graph walk,
into a matrix multiply, and out to a hand-written CUDA kernel — reading real
code the entire way.
10 / 10
roadmap shipped · MIT · no API keys, no telemetry
A neural network framework you can actually read.
EveryO implements tensors, reverse-mode automatic differentiation, layers, optimizers and the training loop from scratch in Python on NumPy. It is not a wrapper around PyTorch, TensorFlow or JAX, and it does not call out to one at runtime.
791 tests passing Python 3.9 – 3.12 · Linux, macOS, Windows
import everyo as eo
x = eo.tensor([2.0], requires_grad=True)
y = x * x
y.backward()
print(x.grad) # [4.]
# ^ a gradient engine you can read,
# not a binding to someone else's261
lines in the gradient engine — the whole thing
~12,800
lines across 70 focused modules
791
tests, running in about 35 seconds on CPU
0.000e+00
difference vs TensorFlow on matmul and conv2d
What it is
Every deep-learning tutorial ends at . Every production framework starts a million lines below it.
There is almost nothing in between that you can read. EveryO is that middle: small enough to follow end to end in an afternoon, and verified carefully enough to trust the numbers it produces.
Verified, not asserted
Every differentiable operation is checked against central finite differences, and the numerical results against TensorFlow. Where the two disagree, the project states by how much.
Nothing to sign up for
No accounts, no API keys, no credentials, no telemetry and no network calls from
the core. Model loading is pickle-free by design — an .evo
archive cannot execute code.
Who it is for
- People learning how deep learning works — read the derivative of every operation you use, rather than only calling it.
- Engineers verifying a result — a small, deterministic, dependency-light reference you can step through in a debugger.
- Teachers and students —
pip install -e .and everything runs, offline, on a laptop.
What it is not
EveryO is not a production training runtime. The roadmap is finished, but “implemented” is not “battle-tested”: it will not out-perform a tuned framework on a large model, its CUDA paths need a GPU this project’s CI does not have, and multi-node training has been exercised across processes rather than across machines.
- Those limits are stated where they apply, not left to be discovered after adoption.
- The verification matrix says what was checked, and what was not.
Capabilities
What ships today
Everything in this table is implemented and covered by the test suite. Items that are experimental or planned are labelled as such rather than quietly listed alongside.
| Area | What it covers | Status |
|---|---|---|
| Tensors | dtypes, devices, broadcasting, indexing, reductions, NumPy interop | Available |
| Autograd | reverse-mode gradients for 25+ operations, iterative graph walk, no_grad | Available |
| Layers | Linear, Conv2D, MaxPool2D, AvgPool2D, Embedding, Flatten, Dropout, Sequential | Available |
| Normalization | BatchNorm1D, BatchNorm2D, LayerNorm — running statistics survive save/load | Available |
| Recurrent | RNN, LSTM (unit forget bias), GRU — BPTT on the same autograd engine | Available |
| Attention | MultiHeadAttention, PositionalEncoding, TransformerEncoder, causal and padding masks | Available |
| Losses | MSE, MAE, BCE, BCE-with-logits (numerically stable), cross-entropy | Available |
| Optimizers | SGD, momentum, Nesterov, weight decay, Adam, AMSGrad | Available |
| Training | validation, metrics, early stopping, checkpointing, LR scheduling, gradient clipping | Available |
| Mixed precision | autocast and GradScaler, fp32 master weights, dynamic loss scaling | Available |
| ONNX export | export_onnx layer by layer — verified against ONNX Runtime, not only the schema checker | Available |
| Tracing ONNX export | export_onnx_traced — runs the model, exports the graph it leaves behind, so recurrent and attention models export too | Available |
| Quantization | quantize_dynamic — post-training int8 weights, per-output-channel scales, no calibration set | Available |
| Profiling | profile() — nested module timings with shapes, and Chrome trace export | Available |
| Distributed | data parallelism on one machine, or across machines with init_tcp_process_group | Available |
| Serialization | .evo archives that cannot execute code on load | Available |
| CUDA | 5 hand-written kernels with pybind11 bindings and automatic CPU fallback | Needs a GPU |
| GPU-resident tensors | eo.cuda.to_device — a chain of operations stays on the device instead of copying back per call | Needs a GPU |
On the “needs a GPU” label: the CUDA paths require the native
extension built against a real NVIDIA toolchain. Their tests exist and they
skip in this project’s CI, which has no GPU — so those two rows are the
only ones on this page not exercised by every run. Everything else here runs on CPU.
The verification matrix below says which is which.
Correctness
Correctness is the feature
Anyone can write something that looks like a framework. These are the numbers the test suite produces, on CPU, with no GPU and no credentials required.
gradients vs finite differences ..... every differentiable op, float64
matmul vs TensorFlow ............. 0.000e+00
conv2d vs TensorFlow ............. 0.000e+00 (valid/same, strided)
max/avg pool vs TensorFlow ............. 0.000e+00
relu vs TensorFlow ............. 0.000e+00
softmax vs TensorFlow ............. 2.980e-08
cross-entropy vs TensorFlow ............. exact to 6 decimals
its GRADIENT vs TensorFlow ............. 3.725e-09
CUDA tests skip themselves without a GPU. TensorFlow tests skip themselves without
TensorFlow. The core suite needs no network, no GPU and no credentials.
Finite differences as the bar
Every gradient is compared against a central finite-difference approximation in float64. A layer is not considered done because it trains — it is done when its derivative matches the numerical one.
TensorFlow as the referee
Tensors use the NHWC layout TensorFlow uses, so convolution results
compare against tf.nn.conv2d directly — no transposing, no
wiggle room.
Results
Produced by the code, not by a designer
Every image below came out of an actual run in the repository, rendered with EveryO's own visualization module. Reproduce them with the bundled example scripts.
examples/neural_network.py.
TransformerEncoder, each having learned its own routing pattern.
The LSTM's advantage is measured, not claimed. Gradient reaching the
first of 40 timesteps: RNN
1.38e-11, LSTM
7.71e-06 — roughly 560,000× larger.
Scaling
Mixed precision, export, quantization and distributed training
The things a framework is supposed to grow into. All of them are implemented, and all of them are measured rather than claimed — including where they do not help.
autocast + GradScaler
matmul and conv2d run in float16; everything else stays in
float32 and the parameters never leave it. The backward pass genuinely runs in
float16, so gradients genuinely underflow — which is what loss scaling exists
to fix.
- Without a scaler: 192 of 193 gradient elements flush to zero.
- With one: 1 of 193.
- Final MSE 6.2983 unscaled vs 0.0474 scaled — matching float32's 0.0474.
ONNX export, layer by layer
EveryO is NHWC; ONNX Conv is NCHW. Rather than
paper over that, every convolution is wrapped in a real pair of Transpose
nodes and "same" padding is written as explicit pads.
- A trained digit CNN re-run through ONNX Runtime agrees to 1.144e-05.
- 100% of predictions match.
- Emits a real
Convnode, not the primitives it decomposes into.
Tracing ONNX export
An LSTM is a program, not a formula — there is no ONNX node that means
one. So export_onnx_traced runs the model and exports the graph the run
leaves behind, and never needs to know what an LSTM is.
- Every recurrent and attention model exports. The table is below.
- 26 operations translated; the rest raise rather than guess.
dynamic_batchchecks its own rewrite before writing the file.
int8 quantization
quantize_dynamic stores Linear weights as signed int8 with
one symmetric scale per output channel. Activations stay in floating point, so no
calibration dataset is needed and the change is inference-only.
- Largest output change on a 64→128→10 network: 0.0222.
- Agreement with float32 on argmax: 100%.
- A size and fidelity trade — not a faster kernel.
Profiling
profile() is opt-in and nests with your modules, reporting calls and
total/mean/max milliseconds per layer. export_chrome_trace() writes a
file you open in chrome://tracing.
- Per-module timings with input shapes recorded.
record_function()labels your own spans, such as data loading.- No profiler active means no overhead: the hooks are no-ops.
Data-parallel training
Processes, not threads, with a shared-memory gradient all-reduce. Averaging over disjoint shards is the same arithmetic as one large batch — so the result should match single-process training, and the example checks that.
- 1 / 2 / 4 workers: 0.26 s, 0.16 s, 0.14 s.
- Every rank ends bit-identical.
- Largest drift from single-process training: 2.09e-07.
Across machines
init_tcp_process_group swaps the shared-memory group for a TCP one: rank
zero runs a rendezvous server, every rank connects, and the collective is the same
arithmetic over a socket instead of shared memory.
- Exercised between two separate OS processes over real sockets.
- Different interpreters, nothing shared but the wire.
- No NCCL and no encryption — it assumes a trusted network.
Mixed precision on a CPU saves memory, not time. NumPy has no native
float16 arithmetic — it upcasts to compute — so the float16 path is usually
slower there. The speedup mixed precision is known for comes from GPU tensor
cores. The numerics are honest; the marketing is not borrowed.
Multi-node is proven across processes, not across machines. Two
interpreters over real TCP is the part that has to hold for ranks on different hosts,
but a second physical machine is not something this project’s CI has. And set
OMP_NUM_THREADS=1 first: on the same 4-core box, four workers took 1.8 s
with it set and 11.0 s without — slower than not parallelising at all.
Interoperability
Taking a model out of EveryO
A framework you cannot leave is a trap. Two exporters cover the whole library between them, and every export on this page was re-run through ONNX Runtime and compared against EveryO — not trusted because the schema checker accepted it.
| Model | Traced ops | Max |diff| |
|---|---|---|
RNN | 45 | 3.02e-07 |
LSTM | 141 | 1.19e-07 |
GRU | 165 | 1.19e-07 |
MultiHeadAttention | 21 | 4.77e-07 |
TransformerEncoderBlock | 46 | 1.19e-06 |
TransformerEncoder ×2 | 101 | 8.34e-07 |
import numpy as np
import everyo as eo
model = eo.LSTM(16, 32, seed=0).eval()
rng = np.random.default_rng(0)
x = rng.standard_normal((4, 8, 16)).astype("float32")
eo.export_onnx_traced(
model, "lstm.onnx", example_input=x
)
# and check it, rather than assume:
np.abs(eo.run_onnx("lstm.onnx", x)
- model(eo.tensor(x)).data).max()
1.1920929e-07
Why a tracing exporter exists at all. The layer-by-layer exporter can
only export layers it has been taught, and an LSTM cannot be one of them: there is no
ONNX node meaning “EveryO LSTM”. Its meaning is roughly 140 primitive
operations in an order that only exists once the layer has run. So the tracer
runs it, then walks the graph left behind. It only ever sees adds, matmuls, sigmoids
and slices.
What tracing costs, stated plainly. A trace records one path through
the model, so control flow is flattened: the same LSTM traced at 2, 4 and 8 timesteps
produces 39, 73 and 141 operations. The export is correct for the length you traced and
is not a general-length model.
dynamic_batch recovers the batch dimension
where it can and verifies that against a second batch size before writing the
file, falling back to a fixed batch when the rewrite does not hold.
Get started
Running in about sixty seconds
One dependency for the core: NumPy. TensorFlow and CUDA are optional and the suite skips their tests cleanly when they are absent.
Install
Clone and install in editable mode.
git clone https://github.com/krishanth7/EveryO.git
cd EveryO && pip install -e .Check the installation
everyo doctor reports what is present and what is optional.
everyo doctor
everyo demo # trains a digit classifier end to endBuild and train
The API is deliberately familiar, so the interesting part is the source beneath it.
import everyo as eo
model = eo.Sequential(
eo.Conv2D(1, 8, 3, padding="same"),
eo.ReLU(),
eo.MaxPool2D(2),
eo.Flatten(),
eo.Linear(4 * 4 * 8, 10),
)
trainer = eo.Trainer(
model,
eo.Adam(model.parameters(), lr=0.005),
eo.CrossEntropyLoss(),
metrics=["accuracy"],
)
history = trainer.fit(train_loader, epochs=25)
# Export it for any other runtime
model.eval()
eo.export_onnx(model, "model.onnx",
input_shape=(1, 8, 8, 1))
everyo doctor and everyo demo, running end to end on a machine with no GPU.
Roadmap
Complete — and here is what that word covers
The roadmap began as ten things EveryO could not do. All ten now ship. A finished list is only worth anything if it says what was actually checked, so the second table does exactly that — including the two rows nobody here has been able to run.
- ShippedConvolution and pooling —
Conv2D,MaxPool2D,AvgPool2D - ShippedBatch and layer normalization —
BatchNorm1D,BatchNorm2D,LayerNorm - ShippedRecurrent layers, attention, transformers —
RNN,LSTM,GRU,TransformerEncoder - ShippedMixed-precision training —
autocast,GradScaler - ShippedONNX interoperability —
export_onnx, verified against ONNX Runtime - ShippedDistributed training — single-machine data parallelism
- ShippedGPU-resident tensors, removing per-call transfers —
eo.cuda.to_device - ShippedMulti-node distributed training —
init_tcp_process_group - ShippedModel quantization and profiling tools —
quantize_dynamic,profile - ShippedA tracing ONNX exporter, so recurrent and attention layers export too —
export_onnx_traced
10 / 10
roadmap items shipped
What “implemented” means, item by item
| Item | Verified how | Not verified |
|---|---|---|
| Tracing ONNX exporter | Six recurrent and attention models exported and re-run through ONNX Runtime; agreement with EveryO to 1.2e-06 or better | Nothing outstanding |
| Quantization | int8 weights round-trip; 100% argmax agreement with float32, largest output drift 0.0222 | Speed. int8 here is a size and fidelity change, not a faster kernel |
| Profiling | Nested module timings, recorded shapes and Chrome trace export, all asserted | Nothing outstanding |
| Multi-node training | All-reduce across two separate OS processes over real TCP sockets — different interpreters, nothing shared but the wire | Two physically separate machines. No second host is available in CI |
| GPU-resident tensors | Transfer counting: a ten-operation chain crosses the host–device boundary 3 times, not 30 | The CUDA kernels themselves. No GPU, no CUDA toolkit and no driver in this project’s CI |
The GPU row is the one to read carefully. What is proven on CPU is the
design claim — that residency removes per-call transfers — using a
stand-in for the native extension that counts every crossing. What is not
proven anywhere in CI is that the kernels compute the right answers on real hardware.
Those tests exist and they skip. Nobody here has run them.
Sponsors
Support
Sponsor EveryO
Fund the roadmap directly — GPU-resident tensors, multi-node training and a tracing ONNX exporter are the next three items.
Become a sponsor → ProjectStar the repository
The cheapest way to help. It costs nothing and it is how the next person learning autograd finds this.
Open on GitHub →
Available
Your placement here
One slot, self-hosted: an image and a link, no tracking script. Enquiries through the repository discussions.
Enquire →
Governance
How the project is run
EveryO is independently maintained, and how it is run is written down rather than implied. If you are deciding whether to depend on it, contribute to it or teach with it, these documents say who decides what and what is expected of participants.
| Document | Read it when you want to know |
|---|---|
| Governance | Who maintains EveryO, how technical decisions are made, and who approves a release |
| Project policy | Terms for using, contributing to, forking and referring to the project |
| Repository rules | The standard an issue, discussion or pull request is held to |
| Code of Conduct | Expected behaviour, and how to report a problem |
| Contributing guide | Development setup, running the suite, and getting a change reviewed |
| Support | Where to ask a question, and what response to expect |
| Security policy | How to report a vulnerability privately, and the threat model |
| License | MIT — the legal terms the project policy supplements but never replaces |
In short: the maintainer holds final technical decisions and release
approval; anyone may contribute through issues, discussions, review or pull requests;
and contributing does not by itself grant commit or release access. Security reports go
through the security policy and are not discussed publicly before coordinated disclosure.
Read the source. That is the whole idea.
The gradient engine is 261 lines. Start there, and follow a number until it stops surprising you.