Tensor

Tensor is Aner's immutable, rank two numerical data structure with native reverse mode differentiation. Elements have explicit Float64 or Float32 precision. CPU supports both; an available Metal/MPS or CUDA backend executes resident Float32 values. Construction defaults to CPU Float64. See Tensor devices and performance for build requirements, placement, and synchronized benchmarks.

Construction

Import aner.tensor for neutral construction and inspection. Import aner.nn for differentiable numerical operations. Both namespaces operate on the same Tensor type.

import aner.tensor

let samples: Tensor = [[1.0, 2.0], [3.0, 4.0]]
let zeros = tensor.zeros(rows: 2, cols: 3)
let seeded = tensor.random(rows: 2, cols: 3, seed: 42, scale: 0.5)
print(tensor.rows(samples))
print(tensor.cols(samples))
print(tensor.value(samples, row: 1, column: 0))

The output is 2, 2, 3.0. Literal rows must be nonempty and rectangular. Literal cells can contain expressions but must be Float64; integer values are not implicitly converted. Row and column counts must be positive and their product must not exceed 1,000,000.

Every Tensor has two dimensions. A scalar uses shape 1 × 1; a row vector uses 1 × n; a column vector uses n × 1. General rank, empty dimensions, and implicit precision promotion are unavailable. .astype("Float32") explicitly converts detached values; GPU placement requires Float32.

tensor.random and nn.random use deterministic SplitMix64 generation from the supplied Int64 seed. Scale must be finite and nonnegative. Values lie in [-scale, scale); zero scale yields zeros. A signed seed maps modulo 2^64. There is no hidden global generator affecting these calls.

Storage and ownership

Values use contiguous row major native storage. CPU Float32 values have explicitly rounded precision; GPU values occupy native Float32 device buffers. Copying a binding shares immutable values and its differentiation graph. nn.detach returns an independent graph free value; GPU immutable storage can be shared without a host readback. nn.parameter creates a new trainable leaf, sharing immutable resident storage when possible. Numerical functions produce new values; writable element indexing is unavailable.

Arithmetic operators such as +, *, and @ do not operate on Tensor. Use the explicit numerical functions. Tensor itself cannot be printed directly; use rows, cols, value, or item to inspect scalars.

Numerical operands must have identical device and precision. Elementwise operations also require identical shapes. Matrix multiplication requires matching inner dimensions. General broadcasting is unavailable; nn.add_bias explicitly repeats a 1 × columns bias across rows.

Gradient state is shared graph metadata. Handles referencing the same graph observe the same gradient state. Concurrent differentiation or gradient access on a shared graph is unsupported, regardless of whether a compiled matrix provider internally uses worker threads.

A Float64 Array, DynamicArray, or rank two NDArray can produce a detached Tensor through its documented conversion method. The result does not share array storage or inherit a gradient graph.

Methods and complexity

Let n be the number of input elements, G the number of nodes reachable in a differentiation graph, and S the number of elements across that graph. A matrix product uses shapes m × k and k × p. Numerical operation rows report kernel time and newly allocated numerical results. A differentiable result additionally validates its reachable graph in O(G) time and O(G) temporary traversal space. Retained graph values belong to the existing graph, rather than being recopied by each operation.

Construction and inspection

item requires 1 × 1. Host reads synchronize resident device work; the first observation of a resident Tensor can read and cache all n values in O(n) time and space, while later immutable observations reuse that cache. value uses nonnegative, zero based coordinates inside the tensor bounds. argmax returns one zero based class index per row, represented as Float64; ties select the first index. It is nondifferentiable.

The nn namespace also exposes zeros, random, rows, cols, item, and value with the same value/shape contracts. nn.argmax(value) has the same detached argmax contract.

Numerical operations

Softmax normalizes each row stably. Cross entropy accepts raw logits and constant exact one hot targets, averaging loss over rows. It combines normalization and logarithms without taking the logarithm of rounded probabilities. Differentiable targets and soft label distributions are unsupported. The logit derivative is (probabilities − targets) / rows.

MSE averages squared error over all cells. Accuracy returns the correctly classified fraction; labels must have shape rows × 1 and contain finite integral class IDs in range. Scores may be logits or probabilities. Accuracy and argmax do not record gradient edges.

Matrix products use cache aware native traversal. Eligible macOS builds can use the compiled CPU provider. Native backward products use transposed access without requiring separate transpose copies. CPU Float64 small or extreme magnitude cases retain checked native arithmetic. Available GPU products use native MPS or cuBLAS Float32 kernels on the selected device. Provider managed packing/workspace and threads are additional implementation resources, separate from the numerical result space bounds.

Differentiation and updates

nn.parameter marks a trainable leaf. Operations involving a trainable value retain the graph needed for the reverse pass; constants otherwise retain only their result values. When a parameter contributes through several graph paths, one backward pass sums those contributions.

nn.backward requires a differentiable 1 × 1 loss. It replaces gradients on the reachable graph rather than accumulating across separate calls or batches. A failed backward attempt invalidates gradients on that graph. Nodes outside it are unaffected. nn.grad requires a valid gradient from a completed reverse pass involving that value.

nn.sgd requires a trainable leaf with a valid gradient and a finite positive rate. It returns parameter − rate × gradient as a new independent trainable leaf. It does not mutate the old parameter. Update every parameter from gradients of the same loss, then recompute the forward pass. Retaining all iteration losses or intermediate values retains their graphs and increases memory usage.

ReLU uses derivative zero at zero. Higher order differentiation, persistent gradient accumulation, generic optimizer objects, and low level Adam updates are unavailable. Fitted model optimizer state is described separately in neural workflows.

Device placement

Float64 remains CPU only. Convert detached values explicitly to Float32 before GPU transfer or parameter creation. Same device requests preserve immutable storage and the graph without copying numerical values. Different device transfers preserve differentiation through a transfer node. Numerical operations reject mismatched dtype or device rather than promoting or moving an operand automatically.

The compiled CPU provider is "apple_accelerate" or "native_blocked"; available accelerators report "metal_mps" or "cuda_cublas". Known unavailable "rocm" and "cluster" requests fail without fallback, and unknown names are errors. Device placement is separate from development recording policy. Tensor devices and performance documents prerequisites, classifier conversion policy, portable checkpoints, and matched benchmark methodology.

Bounds and errors

All input values, operation results, and gradients must remain finite. Invalid shapes, bounds, targets, gradient state, rates, placement requests, nonrepresentable values, and resource limit failures report source located runtime errors. The nn namespace uses R2001 for Tensor errors; aner.tensor uses R2501. Statically known argument type errors are detected before execution. Host allocation failure uses the CLI's E9003 handler.

Dimensions and size arithmetic are validated before allocation. Limits apply to individual values, operations, and reachable graphs; they are not a total process memory quota. Interpreter source size, nesting, call depth, and execution step limits also apply. Raising --max-steps does not raise Tensor limits. Persistent session runtime errors clear session state.

Reproducibility is scoped to the same backend, build, and environment. A provider can change floating reduction order. Record source, seeds, Aner version, settings, and backend when comparing experiments; bitwise equivalence across devices or builds is not guaranteed.

Example

This example performs one differentiable parameter update without a model abstraction:

import aner.nn

let input: Tensor = [[1.0, 2.0]]
let target: Tensor = [[1.0]]
var weights = nn.parameter(nn.random(rows: 2, cols: 1, seed: 42, scale: 0.1))
let prediction = nn.matmul(input, weights)
let loss = nn.mse(prediction, target)
nn.backward(loss)
print(nn.item(loss))
print(nn.value(nn.grad(weights), row: 0, column: 0))
weights = nn.sgd(weights, rate: 0.1)
print(weights.device())

Development inspection and related references

Development observations record bounded shape, actual precision/device/backend, numerical statistics, and available gradient summaries. Resident observations can synchronize and read back values, adding measurable recording cost. Individual values require the explicit bounded value option. Recording does not change Tensor values, graph edges, or optimizer updates.

• Neural networks: model construction, training, and inference.
• Neural workflows: initialization, optimizer state, progress, and device policy.
• Development mode: observation limits and notebook inspection.
• NDArray: multidimensional typed storage and detached Tensor conversion.