Dataset partitions, folds, and mini batches

A Dataset is an immutable source and a selected sequence of original source rows. Partitioning assigns protected training, validation, and test roles. Mini batch schedules preserve those roles and share the same source storage.

Training, validation, and test roles

Training rows determine parameter updates and fitted preprocessing. Validation rows support model selection and research decisions. Test rows support the final evaluation after those decisions are fixed. All Dataset classifier fits and Dataset standardizer fits reject validation and test roles as training input. Selection and batching retain those protected roles. Raw features(), labels(), and one_hot() results are Tensors without Dataset membership identity: Tensor differentiation, Tensor preprocessing, and other estimators cannot infer their role. Keep the original views, call require_training() before extracting training Tensors, and audit held out selections explicitly in custom loops.

import aner.dataset

let parts = dataset.iris().partition(validation: 0.2, test: 0.2, seed: 42)
let training = parts.train()
let validation = parts.validation()
let testing = parts.test()
print(parts.audit())
print(training.role())
print(training.provenance())

The short partition form stratifies by class and preserves attached groups. The full form adds stratify: Bool and group_aware: Bool; supply both flags together. Fractions must be finite and nonnegative, with a positive held out fraction and a sum below one. Every requested partition must contain rows. A missing .validation() or .test() selection raises an error.

For independent rows, total validation and test counts are rounded with floor(n × fraction + 0.5). Class quotas use proportional allocation with deterministic corrections to obtain those exact total counts. Each requested stratified partition must retain every class; infeasible sizes raise an error. Within each class, SplitMix64 and unbiased Fisher and Yates produce reproducible selections. Views preserve original source row order.

DatasetSplit remains the original two way API for the teaching catalogue. Its pinned Iris and Wine memberships are unchanged. Its training and test views now retain protected roles. Official MNIST training uses a training/validation partition; its official test source remains separate.

Source identity and partition integrity

A row identity consists of the immutable source SHA256 and its original zero based row ID. Independent loads of the same source have the same identity. Row positions in a shuffled batch are distinct from original row IDs.

parts.audit() verifies that every eligible row appears exactly once, partition roles are distinct, source and class/feature schema match, and attached whole groups remain disjoint when group preservation is enabled. The report includes source and selection fingerprints. data.provenance() also retains partition seed, fractions, fold settings, role, row count, and group selection fingerprint.

import aner.dataset

let parts = dataset.wine().partition(validation: 0.2, test: 0.2, seed: 2026)
dataset.require_disjoint(parts.train(), parts.validation())
dataset.require_disjoint(parts.train(), parts.test())
print(parts.audit())

Mini batch classifiers configured with ml.training retain cumulative fitted row/group membership, preprocessing membership, and earlier validation selection membership. They reject overlapping evaluation or selection views, missing required group metadata, and use of earlier selection rows as a final test. This history persists in version 5 checkpoints. The legacy full batch classifier evaluation remains compatible: it permits evaluation on training rows and reports that overlap in its summary.

These checks establish membership integrity. They do not deduplicate observations, infer subject identity, or establish statistical independence from feature values.

Explicit group identity

Provide observed subject, acquisition, or session identifiers when multiple rows belong to the same sampling unit. Group keys are Int64 or nonempty String values; key types remain distinct. String keys must be NUL free and at most 1,022 UTF-8 bytes. Every selected row requires a key.

import aner.dataset

let observations = dataset.from_tensors(
    [[1.0], [2.0], [3.0], [4.0], [5.0], [6.0], [7.0], [8.0]],
    [[0.0], [1.0], [0.0], [1.0], [0.0], [1.0], [0.0], [1.0]])
let grouped = observations.with_groups(List("a", "a", "b", "b", "c", "c", "d", "d"))
let parts = grouped.partition(validation: 0.25, test: 0.0, seed: 42)
print(parts.audit())

This is a synthetic example of supplied group identity. The allocator shuffles groups reproducibly, processes larger groups first, and uses a size/class balance objective. It preserves entire groups and exact eligible row coverage. Requested fractions and class proportions are targets: indivisible groups can prevent exact counts, and the heuristic does not guarantee a globally optimal allocation. Insufficient groups or failure to retain every requested class raises an explicit error. Try another seed or change a declared policy when such an allocation is infeasible.

Groups must be attached consistently before leakage checks. require_disjoint rejects shared attached groups even when their row IDs differ. Canonical MNIST IDX files contain no writer identifiers; Aner does not manufacture writer groups.

Cross validation folds

import aner.dataset

let folds = dataset.iris().kfold(count: 5, seed: 2026)
print(folds.audit())
let first = folds.fold(index: 0)
print(first.train().rows())
print(first.validation().rows())

Every eligible row serves as validation exactly once across 2 to 100 folds. A fold's training view contains the other rows. Class stratification and whole group preservation are enabled by the short form; the full form supplies both flags explicitly. Validation and test roles cannot construct folds. The official test source remains outside cross validation.

Lazy mini batch schedules

import aner.dataset

let training = dataset.iris().partition(validation: 0.2, test: 0.0, seed: 42).train()
for batch in training.batches(size: 32, seed: 42, shuffle: true) {
    print(batch.rows())
    let input = batch.features()
    let target = batch.one_hot()
}

Each selected training row occurs once in a schedule. An incomplete final batch is retained. A fixed seed reproduces the ordered source row sequence; shuffle: false retains the view's existing order. schedule.count() and schedule.batch(index) expose the same iteration order. Schedules allocate row indices; feature and target tensors materialize only for the requested batch.

The configured classifier uses epoch seed seed + completedEpoch. Continuation retains epoch position, optimizer state, batch settings, and source order. Exact continuation requires the same numerical backend and precision.

Streaming preprocessing

import aner.dataset

let parts = dataset.iris().partition(validation: 0.2, test: 0.0, seed: 42)
let scaler = parts.train().fit_standardizer()
let validation_features = dataset.transform(scaler, parts.validation().features())

The Dataset overload fits population statistics directly over source rows in bounded passes. It rejects held out roles and avoids a full feature Tensor. Constant columns use scale 1. Reuse these fixed statistics for validation and testing.

Time and space complexity

Let n be eligible rows, d feature columns, c classes, g groups, k partitions or folds, b batch size, and a class quota corrections. Source storage is shared by views. The row quota implementation scans class scores for each correction; its cost is stated explicitly.

Output selections are sorted into canonical source row order; construction costs include this sorting. Group aware audits use ordered group maps. Bounds assume bounded identifier lengths; feature/class schema comparisons add their own linear cost.

Group balancing admits at most 50 million estimated scoring units. Dataset sources retain at most one million rows; each materialized Tensor retains its one million element bound. Batch feature and one hot sizes must both fit that bound. These limits are workload safeguards rather than a complete process memory quota.