# CUDA acceleration

CUDA is **optional**. EveryO runs fully on the CPU and every GPU feature
degrades gracefully when the hardware or toolchain is absent.

## Status

| Component | State |
| --- | --- |
| Kernel sources (`cuda/src`) | Available |
| pybind11 bindings | Available |
| CMake build | Available |
| Automatic CPU fallback | Available |
| Kernels used by tensors on a `cuda` device | Available (add, multiply, matmul, relu, sum) |
| GPU-resident inference tensors (no per-call transfer) | Available |
| Fused kernels, mixed precision | Planned |

## Checking availability

```python
import everyo as eo

eo.cuda.is_available()  # True only with a built extension AND a device
eo.cuda.device_count()
eo.cuda.unavailable_reason()  # a sentence explaining why, or None
eo.cuda.runtime_info()  # full details, including device properties
```

None of these raise. On a CPU-only machine `everyo doctor` prints the reason.

## Building the extension

Requirements: an NVIDIA GPU, the CUDA Toolkit (nvcc), CMake 3.18+ and
pybind11. CUDA is supported only on NVIDIA's own supported platforms and
toolchains — there is no CUDA path on machines without NVIDIA hardware, and
EveryO does not pretend otherwise.

```bash
pip install -e ".[cuda]"     # installs pybind11
./scripts/build_cuda.sh
```

Or with CMake directly:

```bash
cmake -S cuda -B build/cuda -DCMAKE_BUILD_TYPE=Release
cmake --build build/cuda --parallel
```

The build copies `_everyo_cuda*.so` next to the Python package so
`import everyo._everyo_cuda` resolves.

A build host without a GPU is fine: the kernels compile and the extension
imports, EveryO simply reports CUDA as unavailable until it runs on a machine
with a device. `scripts/build_cuda.sh` says so explicitly rather than treating
it as a failure. To target a specific compute capability:

```bash
./scripts/build_cuda.sh --arch=86
```

Clean up with `./scripts/build_cuda.sh --clean`.

## Using it

```python
import everyo as eo

x = eo.tensor([[1.0, 2.0], [3.0, 4.0]], device="cuda")
y = eo.tensor([[5.0, 6.0], [7.0, 8.0]], device="cuda")

z = eo.matmul(x, y)  # runs the tiled CUDA kernel when available
```

`eo.device("auto")` picks CUDA when it is usable and CPU otherwise.

## Fallback rules

Core operations select an `OperatorBackend` through
`everyo.backends.backend_for()`. A dispatch returns the requested backend, the
backend that actually executed, and a reason whenever CUDA falls back. This
makes unsupported dtypes, ranks, and broadcast shapes testable instead of
silently hiding the decision. Host/device conversion is represented by the
backend's `from_host()` and `to_host()` boundary; the current native extension
still performs a transfer for each kernel call.

| Situation | Behaviour |
| --- | --- |
| `eo.device("cuda")`, no CUDA | Returns the CPU device, logs a warning |
| `eo.device("cuda", strict=True)`, no CUDA | Raises `EveryOCudaError` |
| `eo.tensor(..., device="cuda")`, no CUDA | Tensor is created on the CPU |
| `eo.cuda.relu(x)`, no CUDA | Computes with NumPy |
| `eo.cuda.relu(x, strict=True)`, no CUDA | Raises `EveryOCudaError` |
| `EVERYO_DISABLE_CUDA=1` | CUDA is treated as unavailable even when built |

Nothing crashes because CUDA is missing. The only way to get an error is to ask
for one with `strict=True`.

## The kernels

| Source | Kernel | Notes |
| --- | --- | --- |
| `vector_add.cu` | `VectorAdd`, `VectorMultiply` | One thread per element, bounds-checked |
| `relu.cu` | `Relu` | One thread per element |
| `matmul.cu` | `MatMul` | 16x16 shared-memory tiling |
| `reduction.cu` | `Sum` | Block-level tree reduction, host-side final sum |
| `device.cu` | Device queries | Never raises when no driver is present |

All of them use `EVERYO_CUDA_CHECK`, which turns a CUDA error code into a C++
exception carrying the file, line and failing call. `EVERYO_CUDA_CHECK_KERNEL`
also synchronises, so an asynchronous kernel fault surfaces at the launch site
instead of at some unrelated call later. Device memory is owned by an RAII
`DeviceBuffer`, so it is released even when a later call throws.

## Performance

**Do not assume the GPU is faster.** Each kernel call copies its inputs to the
device and the result back; for small problems that transfer dominates and the
CPU wins. Measure on your own hardware:

```bash
python benchmarks/benchmark_matmul.py --sizes 128 512 1024 2048
```

Removing per-call transfers by keeping tensors resident on the device is the
main optimisation for chained supported operations. Use `eo.cuda.to_device()`
to upload float32 data once; addition, multiplication, ReLU, 2-D matrix
multiplication and sum then use device-owned buffers. Call `.numpy()` for an
explicit download. The resident API is inference-only and does not yet attach
EveryO autograd nodes.

## Verifying correctness

Kernel output is compared against the NumPy backend in
`tests/test_devices.py::TestCudaKernels`. Those tests skip automatically
without a GPU, and run for real with one:

```bash
pytest tests/test_devices.py -v
```