Neural learning, research, and devices

Neural network configuration provides two interfaces: immutable Sequential descriptions for Dense classifiers, and user defined classes with explicit Tensor operations for custom forward passes and objectives. Both interfaces use the same native differentiation and selected CPU or accelerator kernels. Classes organize model state; inheritance is not required to configure a Sequential network.

This guide describes implemented behavior in the current interpreter, with VS Code extension 0.1.29 or newer. Runnable downloads are the research classifier, Adam checkpoint continuation, a custom model class, the VS Code notebook, and the Jupyter notebook. VS Code notebooks use .anernb; Jupyter uses .ipynb with the Aner kernel.

Sequential model configuration

import aner.dataset
import aner.nn
import aner.ml

print(nn.devices())
let split = dataset.iris().split(test: 0.2, seed: 2026)
let network = nn.sequential(seed: 42)
    .dense(inputs: 4, outputs: 16, initialization: "he").relu()
    .dense(inputs: 16, outputs: 3, initialization: "xavier")
let plan = ml.classifier(network, standardize: true,
                         loss: "cross_entropy", optimizer: "adam")
    .on(device: "cpu")
let fitted = plan.fit(split.train(), epochs: 400, rate: 0.02, progress: true)
print(fitted.evaluate(split.test()).summary())
print(fitted.predict(split.test()).device())

Iris is Fisher (1936), distributed by UCI under CC BY 4.0; Aner preserves its source identity and attribution. Split before fitting normalization. Standardization learns only from training rows and is reused for held out evaluation and prediction. The final Dense layer produces raw logits; the classifier applies softmax as needed. Add Dense, ReLU, Tanh, or Sigmoid stages with visible dimensions; every adjacent width is checked.

plan is an immutable specification. Each fit returns an independent fitted model; it does not silently mutate a model used for inference. A fitted model owns detached parameters and fitted preprocessing. predict and evaluate reuse fitted preprocessing and parameters on the selected device, so training, testing, and inference share one explicit placement contract.

Initialization and loss

Each Dense layer accepts one of these initialization policies, with zero biases and a reproducible seed equal to the network seed plus its zero based Dense index:

Initialization policies do not establish a model accuracy guarantee. Explicit weights and biases take precedence through the separate custom overload:

let custom = nn.sequential(seed: 42)
    .dense(inputs: 4, outputs: 3,
           weights: nn.random(4, 3, seed: 7, scale: 0.1),
           bias: nn.zeros(1, 3))

Weights must be finite Tensor values with shape inputs × outputs; bias must be 1 × outputs. The builder detaches immutable values, so a separate parameter update cannot mutate the specification. CPU plans use Float64; accelerator plans explicitly convert initialization to Float32 before fitting.

Classifier loss is "cross_entropy" or "mse". Cross entropy averages over rows using raw logits and one hot targets. Classifier MSE averages squared softmax probability error over rows and classes. A custom Tensor loop can instead define an objective from supported differentiable operations; a user function or class cannot currently be passed directly as a classifier loss callback.

Optimization and checkpoint continuation

"sgd" remains the default and updates each parameter by parameter - rate × gradient. "adam" retains first and second gradient moments, uses bias correction with the cumulative update count, and updates by rate × corrected_mean / (sqrt(corrected_second_moment) + epsilon). Adam uses fixed beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8, with no weight decay or AMSGrad. Hyperparameter objects, momentum SGD, scheduling, clipping, and minibatches remain future extensions.

Each epoch is one full batch forward pass, backward pass, and parameter update. rate must be finite and positive; epochs is an explicit positive fixed horizon. An optimizer is separate from the model's layer description and selected loss. The same configuration is used in Dev and Prod.

let partial = plan.fit(split.train(), epochs: 100, rate: 0.02)
partial.save("research.anermodel")
let restored = ml.load_classifier("research.anermodel")
let final = plan.warm_start(restored)
    .fit(split.train(), epochs: 100, rate: 0.02)

Saving creates a new file and refuses to overwrite an existing path. Version 2 checkpoints retain topology, initialization policies, detached parameters, normalization, schema, provenance, metrics, optimizer choice, Adam moments and update count, and measured duration. They do not contain executable source, raw training feature/target rows, or an autograd graph. Version 1 checkpoints remain readable as the original SGD model; their missing duration is reported as zero. Float32 accelerator models use a version 3 extension retaining dtype and source device provenance. They load as CPU Float32 without requiring the source GPU; select .on(device).warm_start(restored) to continue on an available accelerator. CPU Float64 saves remain version 2. New checkpoints require this updated interpreter.

Warm starting requires matching topology, loss, scaling policy, and data schema. It retains the original normalization even when fitting compatible new rows. The same Adam optimizer restores its moments and update count, so two consecutive fits with identical rows, rate, precision, and native backend reproduce one uninterrupted fit. Changing device or precision can change rounding and reduction order. Deliberately choosing a different optimizer keeps the weights and scaler but resets optimizer state; the summary identifies that reset. Continuation restores only the state retained by the checkpoint format. Seeds and exact split settings still belong in your source.

Progress, measured timing, and Dev mode

progress: true prints bounded progress in the CLI and flushes each selected observation. The maximum is 64 lines per fit. Each line identifies completed epoch count, measured loss phase, elapsed seconds, approximate remaining seconds, actual placement, precision, and native kernel provider.

Intermediate losses are the objective immediately before the latest completed update, avoiding an extra forward pass just to report progress. The final loss is measured after the last update. Dev metadata also records loss_epoch, so charts and tables can distinguish these phases. Missing epochs are not fabricated.

After at least five completed updates, estimated remaining time is measured_loop_seconds / completed_epochs × remaining_epochs. Before that, progress says warming up; completion reports zero remaining. The estimate includes observed update and recording cost, but excludes preprocessing and future final evaluation, serialization, or report writing. It can change with thermal throttling, CPU contention, caching, or different observer cost. It is an estimate for a fixed epoch horizon, never a prediction of convergence or accuracy. Aner currently has no unbounded fit API; a user controlled indefinite loop receives no fabricated completion estimate.

fitted.training_seconds() measures this fit using a monotonic clock from setup and preprocessing through training, final metrics, observation callbacks, and parameter detachment. It is separate from pure kernel throughput and from saving/report writing. Repeating a fit can give different timings while returning the same model values on the same backend.

aner neural_research.aner --dev research-report
aner neural_research.aner --prod

Use a new report directory for each Dev run. Open research-report/inspection.html for bounded variables, actual Tensor dtype/device/backend metadata, loss curves, measured progress, and named captures; research-report/index.html adds topology and available gradient statistics. --debug-values opts into bounded individual values. Dev records real observations without changing the optimization algorithm. Prod disables automatic recording and the extra final backward needed for gradient inspection; optional progress is an independent choice. Remove progress: true when its output is not wanted.

In VS Code choose Dev and Inspector; the supplied .anernb opens in development mode. In Jupyter the supplied .ipynb uses %aner dev. Variables, loss curves, and measured progress appear after completed cells. The current notebook protocol buffers output until a cell returns, so it does not yet stream a running fit. %aner prod disables notebook recording. Pausing, stepping inside a native operation, editing live variables, and streaming notebook progress remain planned. Development mode explains the recording limits.

Dev/Prod describe observation policy, not neural training/inference semantics. If future Dropout or BatchNorm layers require different forward behavior, they need a separate explicit training/evaluation setting. Production also remains an interpreter running native C++ kernels; compiling Aner programs is a separate future feature.

Custom model classes

class DenseBlock {
    var weights: Tensor
    var bias: Tensor
    fn forward(input: Tensor) {
        return nn.tanh(nn.add_bias(nn.matmul(input, weights), bias))
    }
    fn update(rate: Float64) {
        weights = nn.sgd(weights, rate)
        bias = nn.sgd(bias, rate)
    }
}
let block = DenseBlock(
    weights: nn.parameter(nn.random(4, 8, seed: 42, scale: 0.25)),
    bias: nn.parameter(nn.zeros(1, 8)))

Import aner.nn before this snippet. Compose class methods and Tensor operations to build explicit forward logic, then call nn.backward(loss) on a scalar objective and update each parameter leaf. nn.grad(parameter) exposes its most recent gradient. Replace each parameter with the result of nn.sgd; intermediate graphs are released when references disappear. Aner does not automatically discover class fields as trainable parameters, serialize arbitrary user classes, or provide automatic parameter registration. The complete custom XOR class shows all steps and captures actual training states.

The architecture separates model description, parameter state, objective evaluation, and optimization. A future common Layer/Model interface requires parameter enumeration, shape inference, checkpoint state, and explicit training/evaluation behavior.

Devices and the actual backend

CPU preprocessing fits and applies normalization in Float64. Accelerator fitting converts features, targets, initialized parameters, and Adam state to Float32 once before training; numerical forward, backward, and parameter updates remain on the selected device. Adam moments stay resident between epochs. Host metrics, requested progress, and Dev inspection are explicit observations that can synchronize and read back device data.

GPU execution requires the native backend to be compiled into Aner and a compatible local device/driver. The runtime initializes its own context and queue or stream; source programs do not open a separate external connection. Unavailable requests fail without fallback, and no numerical operation implicitly moves or promotes an operand. Float64 GPU transfer is rejected. NDArray remains CPU storage; its Float64 rank two to_tensor() conversion makes a detached CPU Tensor.

Tensor devices and performance provides build requirements, CPU/MPS/CUDA examples, portable Float32 checkpoints, notebook inspection, and synchronized benchmark methodology. AMD and distributed execution require separate backend implementations. CNN and RNN operations require their own native forward and gradient coverage.

Native execution and complexity

Tensor values use row major native storage with explicit Float64 or Float32 precision. Forward and transposed backward matrix products share a cache blocked native kernel, avoiding materialized transpose copies. macOS builds can use Accelerate for eligible products; small or extreme magnitude cases retain checked native arithmetic. nn.backend() identifies the compiled provider, not a promise that every operation uses BLAS. Vendor libraries may manage their own threads, packing buffers, and caches. The portable native CPU path is single threaded; the reported logical CPU count is not a promise that every core is saturated. GPU products use resident Float32 MPS or cuBLAS operations, with native elementwise and gradient kernels.

Let B be batch rows, layer widths be d0 through dL, P be the total weights and biases, A be retained activation and loss graph elements, G be reachable graph nodes, E be epochs, and S be stage count. For a matrix product use dimensions m × k and k × n.

Each differentiable operation also validates its reachable graph before publication. This adds O(G) traversal time and temporary metadata to that operation; the training bound includes graph validation across stages. Detached inference avoids differentiation graph construction.

Dev graph/statistic captures add bounded traversal and optional value copying at selected epochs, plus a final backward pass when gradient inspection is requested. The CPU provider can change floating point reduction order; repeatability is scoped to the same settings and native backend, not bitwise identity across devices or vendors. No speedup factor is claimed without a matching benchmark.

Current limits remain 32 stages, one million parameter elements, one million elements per Tensor, four million retained graph elements, 100,000 epochs, and 500 million conservatively estimated work units per fit or prediction/evaluation. Float64 Tensor products allow 50 million multiply add terms; Float32 Tensor products allow one billion. The classifier retains its separate 50 million term Dense stage admission bound. Adam admission accounts for its extra update work. These limits are deterministic safeguards, not RAM quotas or automatic resource tuning. They make this runtime suitable for small measured experiments, not unrestricted large model training.

Model extension requirements

Keep architecture, objective, optimizer state, fitted preprocessing, observation policy, device placement, and distributed strategy as separate concerns. CNN and RNN layers should enter the same visible layer composition and custom forward path once their native forward and gradients exist. General rank differentiation and native convolution kernels are still required for CNNs; CPU NDArray reshaping alone does not add a differentiable convolution backend. BatchNorm/Dropout need their own train/evaluate semantics. Distributed execution additionally needs coordination and sharding, through a distributed strategy distinct from selecting a local device.

The next performance phase should use matched datasets, topology, initialization, batch size, loss, optimizer, dtype, convergence targets, and device. Report end to end time, warmed throughput, peak memory, and synchronization cost separately for Dev and Prod. Performance evaluation should begin with a larger Dense baseline. CNN/RNN benchmarks require implemented and validated operations; results from the Iris demonstration do not establish large model throughput. Machine learning architecture describes the longer term backend and model interface plan.