{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "aner", "display_name": "Aner", "language": "aner" }, "language_info": { "name": "aner", "file_extension": ".aner" }, "aner": { "execution_mode": "persistent-session", "runtime_mode": "development" } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": "# MNIST: oop neural training\n\nThe same MLP, immutable partitions, Xavier initialization, Adam optimizer and mini batch schedule are used at every level of control. Download the [verified preparation script](https://aner.albarena.ai/reference/tools/benchmarks/fetch_mnist.sh), then run `bash fetch_mnist.sh mnist` from this notebook's directory before execution. The data cells use that local `mnist` folder. The official test set is evaluated only after the fixed fifth epoch.\n\nSelect Dev in the Aner editor to see variables, training progress and gradient samples. Select Prod and restart/rerun every cell to measure execution without observations. Native Float64 is supported on CPU and CUDA; MPS requires Float32 or an explicit precision policy. Set the profile in the configuration cell before creating parameters.\n\nUse `%aner prod` in a separate cell and rerun all cells after changing modes.\n" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%aner timeout 600" ] }, { "cell_type": "code", "metadata": {}, "source": [ "%aner dev\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": "## Execution profile\n\nUse `cuda` on an available accelerator, or `mps` with `Float32` on a supported Mac. Placement is explicit and inspectable.\n" }, { "cell_type": "code", "metadata": {}, "source": [ "import aner.nn\n", "nn.configure(device: \"cpu\", precision: \"Float64\", policy: \"preserve\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": "## Dataset protocol\n\n54,000 training rows, 6,000 stratified validation rows, and 10,000 untouched official test rows. No writer IDs are supplied by IDX; the validation split is row stratified and does not claim writer grouping.\n" }, { "cell_type": "code", "metadata": {}, "source": [ "// MNIST MLP: 784 -> 300 -> 100 -> 10. Fixed five epochs; test used once at the end.\n", "// First run tools/benchmarks/fetch_mnist.sh from the repository root.\n", "import aner.dataset\n", "import aner.ml\n", "import aner.dev\n", "import aner.viz\n", "import aner.metrics\n", "\n", "let seed = 42\n", "let epochs = 5\n", "let batch_size = 128\n", "let official = dataset.mnist(directory: \"mnist\", partition: \"train\")\n", "let testing = dataset.mnist(directory: \"mnist\", partition: \"test\")\n", "let partitions = official.partition(validation: 0.1, test: 0.0, seed: 2026, stratify: true, group_aware: false)\n", "let training = partitions.train()\n", "let validation = partitions.validation()\n", "print(partitions.audit())\n", "dataset.require_disjoint(training, testing)\n", "dataset.require_disjoint(validation, testing)\n", "training.require_training()\n", "// Placement follows CLI options; preserve is the default precision policy.\n", "print(nn.execution())\n", "print(nn.backend_details())\n", "dev.automatic(false)\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": "## Model and fixed training run\n\nKeep this phase in a single cell when measuring its elapsed time. The loss is the row weighted mean of each mini batch objective before its update; validation uses detached parameters after each epoch.\n" }, { "cell_type": "code", "metadata": {}, "source": [ "nn.reset_memory_peaks()\n", "let started = nn.clock()\n", "// Research control: parameters, forward, objective, batches and Adam updates.\n", "class MnistNetwork {\n", " var w1: Tensor\n", " var b1: Tensor\n", " var w2: Tensor\n", " var b2: Tensor\n", " var w3: Tensor\n", " var b3: Tensor\n", "\n", " fn forward(input: Tensor) {\n", " let hidden1 = nn.relu(nn.add_bias(nn.matmul(input, w1), b1))\n", " let hidden2 = nn.relu(nn.add_bias(nn.matmul(hidden1, w2), b2))\n", " return nn.add_bias(nn.matmul(hidden2, w3), b3)\n", " }\n", " // Detached parameter leaves avoid constructing a training graph during evaluation.\n", " fn infer(input: Tensor) {\n", " let hidden1 = nn.relu(nn.add_bias(nn.matmul(input, nn.detach(w1)), nn.detach(b1)))\n", " let hidden2 = nn.relu(nn.add_bias(nn.matmul(hidden1, nn.detach(w2)), nn.detach(b2)))\n", " return nn.add_bias(nn.matmul(hidden2, nn.detach(w3)), nn.detach(b3))\n", " }\n", " fn update(state: Optimizer) {\n", " w1 = state.update(w1, key: \"w1\")\n", " b1 = state.update(b1, key: \"b1\")\n", " w2 = state.update(w2, key: \"w2\")\n", " b2 = state.update(b2, key: \"b2\")\n", " w3 = state.update(w3, key: \"w3\")\n", " b3 = state.update(b3, key: \"b3\")\n", " }\n", "}\n", "fn objective(logits: Tensor, target: Tensor) {\n", " return nn.cross_entropy(logits, target)\n", "}\n", "class Scores {\n", " public let loss: Float64\n", " public let accuracy: Float64\n", "}\n", "fn evaluate_network(network: MnistNetwork, samples: Dataset, size: Int64) {\n", " var loss_sum = 0.0\n", " var correct_sum = 0.0\n", " for batch in samples.batches(size: size, seed: 0, shuffle: false) {\n", " let input = nn.place(batch.features())\n", " let target = nn.place(batch.one_hot())\n", " let output = network.infer(input)\n", " let count = Float64(batch.rows())\n", " loss_sum = loss_sum + nn.item(nn.cross_entropy(output, target)) * count\n", " correct_sum = correct_sum + metrics.accuracy(nn.argmax(output), batch.targets()) * count\n", " }\n", " return Scores(loss: loss_sum / Float64(samples.rows()), accuracy: correct_sum / Float64(samples.rows()))\n", "}\n", "let network = MnistNetwork(\n", " w1: nn.parameter(nn.place(nn.xavier(rows: 784, cols: 300, seed: seed))),\n", " b1: nn.parameter(nn.place(nn.zeros(1, 300))),\n", " w2: nn.parameter(nn.place(nn.xavier(rows: 300, cols: 100, seed: seed + 1))),\n", " b2: nn.parameter(nn.place(nn.zeros(1, 100))),\n", " w3: nn.parameter(nn.place(nn.xavier(rows: 100, cols: 10, seed: seed + 2))),\n", " b3: nn.parameter(nn.place(nn.zeros(1, 10))))\n", "let state = nn.optimizer(name: \"adam\", rate: 0.001)\n", "fn train_network(network: MnistNetwork, state: Optimizer, training: Dataset, validation: Dataset,\n", " epochs: Int64, batch_size: Int64, seed: Int64, started: Float64) {\n", " let initial = evaluate_network(network, training, batch_size)\n", " print(\"BENCHMARK_INITIAL_LOSS\")\n", " print(initial.loss)\n", " var epoch = 0\n", " while epoch < epochs {\n", " var weighted_loss = 0.0\n", " var batch_index = 0\n", " let schedule = training.batches(size: batch_size, seed: seed + epoch, shuffle: true)\n", " for batch in schedule {\n", " let input = nn.place(batch.features())\n", " let target = nn.place(batch.one_hot())\n", " let loss = objective(network.forward(input), target)\n", " weighted_loss = weighted_loss + nn.item(loss) * Float64(batch.rows())\n", " nn.backward(loss)\n", " // One gradient/topology sample per epoch; no per-batch inspection overhead.\n", " if batch_index == schedule.count() - 1 { viz.capture(epoch + 1, loss) }\n", " network.update(state)\n", " batch_index = batch_index + 1\n", " }\n", " epoch = epoch + 1\n", " let held_out = evaluate_network(network, validation, batch_size)\n", " let online_loss = weighted_loss / Float64(training.rows())\n", " dev.progress(epoch: epoch, epochs: epochs, loss: online_loss, accuracy: held_out.accuracy, seconds: nn.clock() - started, validation_loss: held_out.loss)\n", " print(\"BENCHMARK_EPOCH\")\n", " print(epoch)\n", " print(online_loss)\n", " print(held_out.loss)\n", " print(held_out.accuracy)\n", " }\n", " return evaluate_network(network, training, batch_size)\n", "}\n", "let final_training = train_network(network, state, training, validation, epochs, batch_size, seed, started)\n", "print(\"BENCHMARK_FIT_SECONDS\")\n", "print(nn.clock() - started)\n", "print(\"BENCHMARK_TRAIN_LOSS\")\n", "print(final_training.loss)\n", "print(\"BENCHMARK_TRAIN_ACCURACY\")\n", "print(final_training.accuracy)\n", "let test_started = nn.clock()\n", "let evaluation = evaluate_network(network, testing, batch_size)\n", "print(\"BENCHMARK_TEST_SECONDS\")\n", "print(nn.clock() - test_started)\n", "print(\"BENCHMARK_TEST_LOSS\")\n", "print(evaluation.loss)\n", "print(\"BENCHMARK_TEST_ACCURACY\")\n", "print(evaluation.accuracy)\n", "print(\"BENCHMARK_MEMORY\")\n", "print(nn.memory())\n", "dev.capture(\"Final model\", network)\n", "print(state.summary())\n" ], "execution_count": null, "outputs": [] } ] }