Tensor devices and performance

Aner executes rank two Tensor operations on CPU or an explicitly selected, available accelerator. CPU supports Float64 and Float32. The Metal/MPS and CUDA backends use Float32 device storage, native matrix multiplication, elementwise and reduction kernels, reverse mode gradients, and resident parameter updates. Device placement is independent of Dev/Prod observation policy.

The complete Tensor device example, bounded matrix stress example, classifier comparison, VS Code notebook, and Jupyter notebook exercise CPU and whichever accelerator is available. They do not silently select another backend when an explicitly requested device is unavailable.

Runtime prerequisites

An ordinary CPU only Aner binary cannot use a GPU solely because the hardware or driver is installed. The native backend must be included in that binary. Creating the native device context, selecting a queue/stream, owning device buffers, dispatching kernels, and checking completion are responsibilities of the Aner runtime. No external connection or server login is needed from an Aner program to use a local GPU. nn.require_device validates initialization before the numerical workload.

Apple documents MPS matrix multiplication as a native Metal operation. The NVIDIA cuBLAS documentation describes GPU memory matrix operations, while the CUDA installation guide defines the supported host toolchain and driver requirements. Match the toolkit to the installed GPU architecture; support in a newer toolkit must not be assumed for an older GPU.

Native builds configure accelerator integration explicitly: Metal/MPS is enabled by the macOS build option, while CUDA is opt in and must target a compute capability supported by the selected toolkit. The build includes the native framework/runtime libraries; the program then selects an available device. Deployment packages must match both the host platform and their documented accelerator runtime requirements.

Explicit precision and placement

import aner.nn

nn.require_device(device: "mps")
let input = nn.random(rows: 32, cols: 64, seed: 2026, scale: 0.25)
    .astype(dtype: "Float32").to(device: "mps")
var weights = nn.parameter(nn.random(rows: 64, cols: 16, seed: 42, scale: 0.1)
    .astype(dtype: "Float32").to(device: "mps"))
let output = nn.matmul(input, weights)
print(output.dtype())
print(output.device())
print(output.backend())
nn.synchronize(device: "mps")

For a CUDA build, replace each "mps" with "cuda". For a matched Float32 CPU run, use "cpu". Construction remains CPU Float64 by default; .astype("Float32") is an explicit precision change, and .to is a separate memory placement operation. Convert detached values before nn.parameter. Changing precision on a differentiable value is rejected; use an explicit detach when discarding its graph is intended.

Operands of a numerical operation must have matching precision and device. Device transfer preserves differentiation; same device transfer shares immutable storage. Float64 GPU requests fail rather than changing precision. A host scalar read, value inspection, or explicit .to("cpu") observes completed work and may cause a synchronization or readback. Keep these observations outside a measured resident loop when measuring throughput.

The zero argument nn.backend() describes the compiled CPU provider. Use the Tensor overload when reporting accelerator execution. A report of physical GPU presence alone is insufficient evidence that a Tensor operation executed there.

Classifier placement

import aner.nn
import aner.ml
import aner.dataset

let split = dataset.iris().split(test: 0.2, seed: 2026)
let network = nn.sequential(seed: 42)
    .dense(inputs: 4, outputs: 16, initialization: "he").relu()
    .dense(inputs: 16, outputs: 3, initialization: "xavier")
let plan = ml.classifier(network, standardize: true,
    loss: "cross_entropy", optimizer: "adam").on(device: "cuda")
let fitted = plan.fit(split.train(), epochs: 20, rate: 0.02, progress: true)
print(fitted.weights(0).device())
print(fitted.weights(0).dtype())
print(fitted.evaluate(split.test()).accuracy())

CPU preprocessing fits and applies normalization in Float64. Accelerator fitting converts the preprocessed features, one hot targets, parameters, and optimizer moments to Float32 and places them once before the retained training loop. Forward, backward, SGD, and Adam updates execute on the selected device. Adam moments remain resident between epochs; checkpoint state is read back after fitting. Evaluation reuses the fitted scaler and selects the same device and precision for its numerical inputs. Metrics and requested progress observations may read results back to the host.

Float32 checkpoints use version 3 and retain precision plus source device provenance in the portable scalar format. Loading reconstructs CPU Float32 parameters even when the source accelerator is unavailable. Apply the desired classifier plan with .on(device).warm_start(restored) to continue on that device. Version 1 and 2 remain readable; CPU Float64 saves retain version 2. Exact continuation is scoped to the same backend and precision; changing device or precision can change reduction order and rounding.

Repeatable stress and performance measurements

Build the native aner_gpu_benchmark target alongside Aner, then run:

aner_gpu_benchmark --device all --repetitions 3
aner_gpu_benchmark --device cpu --size 512 --repetitions 5
aner_gpu_benchmark --device mps --size 768 --repetitions 5
aner_gpu_benchmark --device cuda --size 768 --repetitions 5

Use the benchmark executable supplied with the matching native Aner build. Run the first command on each machine to retain a matched CPU Float32 baseline. A specifically requested unavailable accelerator fails. all records unavailable backends as skipped. Output is JSON Lines, suitable for retaining alongside the Aner source, version, compiler configuration, host and GPU inventory, and thermal/power conditions. The executable supports sizes 128, 256, 512, and 768 and 1 through 20 repetitions. Default execution is bounded; repeated stress should be selected deliberately.

Each matrix measurement separates dtype conversion, upload/placement, warmed synchronized resident operation, and readback. Resident timing includes output allocation and finite result validation; it is a runtime operation measurement rather than an isolated vendor library kernel time. Warm up uses the same shape and precision and is excluded. GPU measurements compare every output cell with matched CPU Float32 results. An independent scalar double precision reference checks all 128 × 128 result cells, or 64 deterministic cells for larger products. Extra CPU Float64 measurements use sizes 128 and 256 within the existing Float64 work limit.

The runner also measures a 32 row matrix forward pass, reverse differentiation of its mean squared output, and SGD update, checking resulting parameters and loss against CPU Float32. Incorrect placement, nonfinite results, excessive numerical discrepancy, or native errors fail the benchmark rather than producing a speedup claim.

For operation duration t, a square n × n matrix product reports approximately 2n³ / t floating point operations per second. Speedup is the same machine's matched CPU Float32 operation duration divided by its accelerator duration. Precision, shape, workload, synchronization, and warm up must match. Small workloads can be slower on GPU because context dispatch, allocation, validation, and synchronization dominate useful work. Classifier training_seconds() includes preprocessing, observers, and final metrics; it must not be treated as isolated matrix throughput.

Dev adds bounded statistics, values, gradient captures, and progress observations. Run Prod for performance measurements, then use Dev to inspect placement and training behavior. Notebook observations are available after the cell returns; the current notebook protocol does not stream an in progress fit.

Verified measurements: October 11, 2026

These correctness checked measurements use five warmed repetitions, Release optimized native code, and Float32 operands on both devices. They time complete resident Tensor matrix operations, including output allocation, finite result checks, and synchronization. Conversion, upload, readback, and independent reference validation are measured separately.

The Mac uses an Apple M2 Max and the Accelerate CPU provider; its GPU is Metal/MPS. The Ubuntu test host uses a GTX 1060 with 3 GiB VRAM and a virtualized Xeon E5 to 2697 v2 with two available vCPUs. Its current CPU backend is the portable single thread cache blocked implementation, without an installed optimized BLAS provider. The large CUDA ratios describe this particular native CPU baseline and virtual machine; they do not predict a comparison against optimized CPU BLAS or another host.

Smaller workloads do not always benefit. On this Mac, 128 × 128 multiplication took about 0.047 ms on CPU and 0.248 ms on MPS; 256 × 256 took 0.155 ms and 0.226 ms respectively. The separate 32 row forward/backward/SGD workload at width 512 took about 0.949 ms per update on Mac CPU and 2.529 ms on MPS. On Ubuntu, the same width took 17.877 ms on its portable CPU backend and 0.659 ms on CUDA. These training step timings include graph admission and multiple synchronized tensor operations. They are not full classifier training or large model benchmarks.

All measured GPU outputs were compared against CPU Float32 results, and the independent double precision checks passed. Timings are observations of the stated workload and configuration, without confidence intervals or a guarantee of the same performance under different load, power, thermal, memory, or compiler conditions.

Complexity and current limits

Let n be stored elements, G reachable graph nodes, m × k and k × p be matrix operands, and P trainable parameters. GPU parallelism changes constants and achievable throughput, not these total work bounds.

Differentiable publication also validates graph admission in O(G) time and temporary metadata. Each Tensor is limited to one million elements. Float64 matrix products retain the 50 million multiply add bound; Float32 products allow one billion. Graph, architecture, and classifier admission bounds otherwise remain in force. These checks are deterministic workload limits, not a total RAM/VRAM quota. A device allocation failure remains an explicit error.

NDArray remains CPU typed storage, and GPU placement is currently a Tensor feature. This implementation targets rank two Dense workflows and explicit Tensor experiments. General rank autograd, CNN/RNN layers, mixed precision, automatic device selection, multiple local GPUs, device memory accounting, distributed execution, and large model scheduling require separate implementations and validation. See Tensor, neural workflows, and development mode.