CUDA & devices.

BEFORE YOU BEGIN

Know where computation runs.

Start with a working CPU installation. Add the native extension only when compatible NVIDIA hardware and the build toolchain are available.

Status

Checking availability

import everyo as eo

eo.cuda.is_available()  # True only with a built extension AND a device
eo.cuda.device_count()
eo.cuda.unavailable_reason()  # a sentence explaining why, or None
eo.cuda.runtime_info()  # full details, including device properties

None of these raise. On a CPU-only machine everyo doctor prints the reason.

Building the extension

Requirements: an NVIDIA GPU, the CUDA Toolkit (nvcc), CMake 3.18+ and pybind11. CUDA is supported only on NVIDIA's own supported platforms and toolchains — there is no CUDA path on machines without NVIDIA hardware, and EveryO does not pretend otherwise.

pip install -e ".[cuda]"     # installs pybind11
./scripts/build_cuda.sh

Or with CMake directly:

cmake -S cuda -B build/cuda -DCMAKE_BUILD_TYPE=Release
cmake --build build/cuda --parallel

The build copies _everyo_cuda*.so next to the Python package so import everyo._everyo_cuda resolves.

A build host without a GPU is fine: the kernels compile and the extension imports, EveryO simply reports CUDA as unavailable until it runs on a machine with a device. scripts/build_cuda.sh says so explicitly rather than treating it as a failure. To target a specific compute capability:

./scripts/build_cuda.sh --arch=86

Clean up with ./scripts/build_cuda.sh --clean.

Using it

import everyo as eo

x = eo.tensor([[1.0, 2.0], [3.0, 4.0]], device="cuda")
y = eo.tensor([[5.0, 6.0], [7.0, 8.0]], device="cuda")

z = eo.matmul(x, y)  # runs the tiled CUDA kernel when available

eo.device("auto") picks CUDA when it is usable and CPU otherwise.

Fallback rules

Core operations select an OperatorBackend through everyo.backends.backend_for(). A dispatch returns the requested backend, the backend that actually executed, and a reason whenever CUDA falls back. This makes unsupported dtypes, ranks, and broadcast shapes testable instead of silently hiding the decision. Host/device conversion is represented by the backend's from_host() and to_host() boundary; the current native extension still performs a transfer for each kernel call.

Nothing crashes because CUDA is missing. The only way to get an error is to ask for one with strict=True.

The kernels

All of them use EVERYO_CUDA_CHECK, which turns a CUDA error code into a C++ exception carrying the file, line and failing call. EVERYO_CUDA_CHECK_KERNEL also synchronises, so an asynchronous kernel fault surfaces at the launch site instead of at some unrelated call later. Device memory is owned by an RAII DeviceBuffer, so it is released even when a later call throws.

Performance

Do not assume the GPU is faster. Each kernel call copies its inputs to the device and the result back; for small problems that transfer dominates and the CPU wins. Measure on your own hardware:

python benchmarks/benchmark_matmul.py --sizes 128 512 1024 2048

Removing per-call transfers by keeping tensors resident on the device is the main optimisation for chained supported operations. Use eo.cuda.to_device() to upload float32 data once; addition, multiplication, ReLU, 2-D matrix multiplication and sum then use device-owned buffers. Call .numpy() for an explicit download. The resident API is inference-only and does not yet attach EveryO autograd nodes.

Verifying correctness

Kernel output is compared against the NumPy backend in tests/test_devices.py::TestCudaKernels. Those tests skip automatically without a GPU, and run for real with one:

pytest tests/test_devices.py -v