MNIST Dense classification benchmark

This benchmark measures a reproducible Dense classifier on canonical MNIST. It retains the official training and test sources, fixes validation membership, and compares source programming styles under the same native numerical configuration.

The APIs on this page require the current development interpreter with audited Dataset partitions, lazy batches, and mini batch training, together with editor extension 0.1.31. Previously published packages labelled 0.1.0-dev may predate these APIs; that version label alone does not confirm availability.

Verified data and held out roles

Place these decompressed IDX files in a local directory. dataset.mnist(directory, partition) validates file SHA256, exact payload lengths, digit labels, 28 × 28 shape, and the official 60,000/10,000 image counts. It retains compact unsigned byte pixels and scales them by 1/255 when batches materialize. The data are loaded locally; the teaching catalogue remains Iris and Wine.

The checksum verified preparation script downloads the four compressed canonical files from the attributed CVDF mirror, verifies fixed compressed SHA256, decompresses them, and verifies the raw hashes below. Run it from the project directory with bash fetch_mnist.sh; its default destination is build/benchmarks/mnist/data, matching the supplied complete programs. The concise fragment below uses a shorter illustrative mnist path; select the directory actually prepared. No dataset bytes are embedded in the source downloads.

Retain the canonical MNIST source attribution and its dataset terms. Dataset terms remain separate from Aner software licensing. The distribution includes no inferred writer identifiers.

import aner.dataset

let original = dataset.mnist(directory: "mnist", partition: "train")
let parts = original.partition(validation: 0.1, test: 0.0, seed: 2026, stratify: true, group_aware: false)
let training = parts.train()
let validation = parts.validation()
let testing = dataset.mnist(directory: "mnist", partition: "test")
print(parts.audit())
dataset.require_disjoint(training, testing)
dataset.require_disjoint(validation, testing)

The exact partition is 54,000 training rows and 6,000 validation rows. The official 10,000 test rows support final evaluation. Test data do not select hyperparameters, stopping conditions, or a best epoch. Folds, when used for a different experiment, draw only from the official training source.

Fixed model and training configuration

The measured model has widths 784 → 300 → 100 → 10, ReLU after both hidden Dense layers, and raw output logits. It contains 266,610 trainable scalars. The two hidden widths follow the 300 + 100 fully connected baseline listed on the canonical MNIST page. This experiment uses modern ReLU, cross entropy, and Adam; it does not replicate that historical model or claim its reported error rate. Xavier uniform initialization uses bounds sqrt(6/(inputs + outputs)), Dense seed run seed + layer, and zero biases.

Use five epochs, mini batches of 128, Adam at 0.001, beta1 0.9, beta2 0.999, epsilon 1e-8, and mean one hot cross entropy from logits. Each 54,000 row epoch contains 422 updates; the final batch contains 112 rows. Run seeds are 42, 43, and 44. Epoch shuffle seed is run seed + completedEpoch.

import aner.nn
import aner.ml
import aner.dataset

nn.configure(device: "auto", precision: "Float64", policy: "preserve")
let original = dataset.mnist(directory: "mnist", partition: "train")
let parts = original.partition(validation: 0.1, test: 0.0, seed: 2026, stratify: true, group_aware: false)
let network = nn.mlp(inputs: 784, hidden: List(300, 100), outputs: 10, seed: 42)
let model = ml.classifier(network, standardize: false, optimizer: "adam")
let config = ml.training(epochs: 5, batch: 128, rate: 0.001, seed: 42)
    .validation(parts.validation()).select_best(false)
let fitted = model.fit(parts.train(), config: config, progress: true)
let testing = dataset.mnist(directory: "mnist", partition: "test")
print(fitted.evaluate(testing).summary())

The preserve profile retains Float64 on CPU or CUDA. Metal/MPS resolves a Float64 preserve request to CPU with a warning. Select adapt to permit MPS Float32, or explicitly select Float32 for matched precision comparisons. Command line device and precision policy options can change the run configuration while keeping the source program the same. Tensor devices documents this contract.

Concise, functional, and class based source

A concise program delegates fixed epoch mini batches to ml.training. The supplied functional program exposes each Dense initialization and activation in build_network, then composes a train_network function around the immutable training contract and classifier fit. The class based program exposes parameter tensors, forward and detached inference methods, a replaceable objective function, explicit batch loops, and keyed optimizer updates while preserving the same numerical operation sequence. Each style uses the same batch schedule and immutable source identity.

Download the complete concise program, functional program, and class based program. Corresponding Jupyter concise, functional, and class based notebooks, and VS Code concise, functional, and class based notebooks preserve the source workflow. Source programs expect the repository root as their working directory. Notebook data cells use a local mnist directory: run the preparation script as bash fetch_mnist.sh mnist from the notebook directory. Downloaded notebooks contain no saved results.

The custom programs retain Dataset views and explicitly check training roles and disjoint membership before materializing Tensors. Those checks are essential because a materialized Tensor does not retain its Dataset role. The configured mini batch classifier enforces its retained fitting and selection history automatically.

nn.optimizer("adam", rate) retains resident moments and a separate update counter for each stable parameter key. Retain each returned replacement parameter. Distinct trainable tensors require distinct keys; changing shape, device, or dtype under an existing key raises an error. reset() deliberately discards that optimizer history.

Custom objectives belong in an explicit Tensor forward/training loop, as exposed by the class based example. The classifier's built in objective remains cross entropy or its documented probability MSE.

Measured loss and progress

Online training loss is the row weighted mean of batch losses measured before their updates. It combines observations from changing parameter states across the epoch. Validation loss and accuracy are detached passes over the held out validation rows after the epoch. The final training metric is a separate pass over the returned parameter snapshot. These phases must be labelled separately.

Development mode records bounded epoch observations and separate training/validation curves. Disable automatic variable capture with dev.automatic(false) for larger experiments and keep explicit captures or progress observations that answer a research question. Production mode removes those automatic observations; numerical precision and the optimization algorithm remain independently configured. The supplied pipeline captures its first batch at selected epochs and performs an additional final backward pass for gradient inspection; the explicit class loop captures its last batch each epoch. Compare Dev against Prod within the same source style. These observations differ between styles, so their timing difference does not isolate a common instrumentation cost.

Notebook cells default to a 120 second execution timeout. Before a larger Jupyter training cell, run %aner timeout 600 in a separate cell. In VS Code set aner.notebook.timeoutMs to 600000. These changes affect subsequent cells and retain current variables; timeout or cancellation terminates the session and clears its variables. Increase the limit to match the planned run rather than treating it as an estimate of training completion.

Reproducibility and resource reporting

The native manifest exporter retains verified source file hashes, exact ordered split row IDs, all epoch permutations, class counts, and initial parameter bytes. Index artifacts use UInt32 little endian; initial parameters use Float64 little endian with named shapes and file SHA256. Both precisions start from those same canonical parameters. Pixel normalization occurs in CPU Float64, followed by the declared placement/precision conversion.

Report each trial's actual device, precision, provider, compiler/build, CPU threading policy, fixed seeds, epoch losses, validation/test results, training duration, and measured resource usage. Device timing requires completion synchronization. Keep initialization, source loading, training, validation, final test evaluation, and report generation distinguishable.

Native Tensor counters measure owned host and accelerator buffers, including observed peaks. They omit dataset storage, graph metadata, vendor workspaces, and unrelated process allocations. Whole process resident memory and accelerator allocator observations require separate measurements. Float32 datatype width does not imply that total process memory halves; the current CPU implementation also keeps numerical host caches.

Repeat the fixed seeds and report individual results plus aggregate variation. A single small demonstration or a single best run does not establish general throughput or accuracy. Matched precision comparisons and explicit convergence checks provide different evidence from comparisons that change precision.

Measured Mac results

Measurements were collected on October 11, 2026, on an Apple M2 Max with 12 logical CPU cores, 38 integrated GPU cores, and 64 GiB RAM, running macOS 26.7.1 (build 25G241). The interpreter used Apple Clang 21.0.0, C++20 Release optimization (-O3 -DNDEBUG) with -fno-fast-math -ffp-contract=off. CPU products used Apple Accelerate; VECLIB_MAXIMUM_THREADS=2 requested a provider thread cap, while the actual provider thread count was unavailable. See the verified hardware and build record.

The following results use the concise program in production mode, the fixed fifth epoch, and paired seeds 42, 43, and 44. Each table entry is the median of three runs. Dataset loading is excluded from fit_seconds; fitting includes model setup, training, epoch validation, final training metrics, and parameter detachment. Test evaluation is measured separately. These observations characterize this bounded Dense workload on the measured machine.

For this workload, Metal/MPS Float32 did not improve elapsed fit time over CPU Float32. Kernel dispatch, synchronization, transfer, and workload dimensions affect accelerator utilization. This result is not a conclusion about larger models or other devices. The observed accuracy variation across three seeds does not establish one precision as generally more accurate.

The current CPU Float32 implementation owns both Float32 matrix caches and a Double payload for checked native operations. Its measured memory therefore exceeds CPU Float64 in this experiment. Native Tensor buffer counters are narrower than whole process RSS. On Apple unified memory, host/device buffer categories do not represent physically independent memory pools; their individual peaks can occur at different times and must not be summed to infer a process peak.

Download the individual public trials and three seed production aggregates. The public trial record includes source/binary fingerprints, epoch metrics, timing, native owned buffer peaks, and process RSS. Reproduce each configuration before extending the conclusion to a different workload.

The large MNIST Ubuntu/CUDA benchmark has not yet been measured. Earlier native CUDA correctness and matrix tests establish different evidence and are not substitutes for this end to end training experiment.

Complexity

Let n be training rows, v validation rows, b batch size, e epochs, p parameter count, and a the per row activation width summed across layers including the input width. Dense numerical work is O(e(n + v)p + e ceil(n/b)p). Inference over t test rows adds O(tp). Working numerical storage is O(ba + p), with two additional O(p) Adam moment buffers. The compact source retains O((n + v + t) × 784) bytes; schedules and partition membership retain O(n + v + t) indices. A fresh fit also sorts provenance membership in O((n + v) log(n + v)); epoch schedule construction and hashing add O(en). If q Tensor operations per training batch retain a graph with G nodes, admission adds expected O(e ceil(n/b) qG) traversal work. The fitted result retains O(e) epoch metrics and schedule fingerprints; the 64 event Dev capture limit does not bound that history. Provider workspaces and report observations add separate costs.

Dataset partitions defines exact membership, group/fold constraints, workload bounds, and preprocessing complexity. Neural workflows defines fitted state and checkpoint continuation.