Datasets

mtlearn.datasets contains PyTorch helpers for paired image datasets and datasets that generate targets from input images with a callable.

Public PyTorch dataset helpers for mtlearn.

class mtlearn.datasets.GeneratedTargetImageDataset(root_dir, target_fn, *, num_rows=None, num_cols=None, grayscale=True, invert_in=False, extensions=('.jpg', '.jpeg', '.png', '.bmp', '.tif', '.tiff'), dtype=torch.float32, scale_in=True, scale_out=True, suffix_in='')

Bases: Dataset

Read source images and generate each target with target_fn.

Samples are returned as (input_tensor, target_tensor, filename) with channel-first tensors. Resize is optional; when configured, the source image is resized before calling target_fn. The generated target must match the post-resize source spatial shape.

Parameters:

• root_dir (str)

• target_fn (Callable[[np.ndarray], np.ndarray])

• num_rows (int | None)

• num_cols (int | None)

• grayscale (bool)

• invert_in (bool)

• extensions (tuple[str, ...])

• dtype (torch.dtype)

• scale_in (bool)

• scale_out (bool)

• suffix_in (str)

train_test_split(test_size=0.25, shuffle=True, random_state=42)

Return (train_subset, test_subset) using stable dataset indices.

class mtlearn.datasets.PairedImageDataset(root_dir, num_rows=None, num_cols=None, *, grayscale_in=True, grayscale_target=True, invert_in=False, invert_target=False, extensions=('.jpg', '.jpeg', '.png', '.bmp', '.tif', '.tiff'), dtype=torch.float32, scale_in=True, scale_out=True, prefix_in='', prefix_target='', suffix_in='_in', suffix_target='_target')

Bases: Dataset

Read matched input/target image pairs from one directory.

Samples are returned as (input_tensor, target_tensor, filename) with channel-first tensors. Resize is optional; when configured, both input and target images are resized to (num_rows, num_cols). Without resize, the input and target spatial shapes must already match.

Parameters:

• root_dir (str)

• num_rows (int | None)

• num_cols (int | None)

• grayscale_in (bool)

• grayscale_target (bool)

• invert_in (bool)

• invert_target (bool)

• extensions (tuple[str, ...])

• dtype (torch.dtype)

• scale_in (bool)

• scale_out (bool)

• prefix_in (str)

• prefix_target (str)

• suffix_in (str)

• suffix_target (str)

classmethod from_folders(input_dir, target_dir, *, ordering='textual', unmatched='error', extensions=('.jpg', '.jpeg', '.png', '.bmp', '.tif', '.tiff'), **kwargs)

Match direct files in two directories by filename stem.

ordering="textual" preserves textual IDs; "numeric" normalizes decimal IDs. Duplicate IDs within a role always fail. unmatched is error, warn or ignore and controls incomplete pairs. Missing counterparts are reported in missing_input_ids and missing_target_ids. Extensions are case-normalized. Reader options follow from_pairs().

Configuration reconstruction rescans the directories and checks the ordered ID digest. Use from_manifest() for a saved selection that must ignore unrelated new files.

classmethod from_manifest(path, *, roots, split=None, expected_fingerprint=None)

Reconstruct a nonempty saved selection with the manifest’s exact reader.

Resolve only saved records, optionally in one split’s order. The compact source configuration stores the manifest path, expected manifest fingerprint, roots and split rather than embedding every pair. Worker recreation rejects changed manifest contracts. Root locations may be changed explicitly for relocation.

This loads metadata and checks source existence/containment, without decoding or verifying source hashes. Call SplitManifest.validate_files explicitly before execution. Use subsets_from_ids(..., allow_empty=True) when empty views are needed. The dataset’s effective reader must remain unchanged before exporting its configuration.

classmethod from_pairs(pairs, **kwargs)

Read explicit (sample_id, input_path, target_path) records.

IDs must be unique, nonempty strings. ordering="provided" preserves record order; "textual" sorts strings. "numeric" sorts decimal IDs numerically and normalizes them, so "01" and "1" collide. Indexing checks file existence without decoding images.

Reader options shared with from_folders() and from_suffixes():

• grayscale_in and grayscale_target default to True. False reads RGB; ordinary reading converts to uint8 and discards alpha channels.

• strict_grayscale_uint8=True requires native 2D uint8 input and target arrays, with both grayscale flags enabled.

• binary_target=True requires a native 2D uint8 target whose values belong to either {0, 1} or {0, 255}. It maps positive values to one, requires grayscale_target=True and rejects invert_target=True.

• invert_in and invert_target default to False and apply 255 - image before resizing.

• num_rows and num_cols default to None. Supply both to resize inputs with area interpolation and targets with nearest interpolation. Input and target spatial sizes must match before resizing.

• dtype defaults to torch.float32. scale_in and scale_out default to True and divide converted tensors by 255; binary targets bypass scale_out. Integer tensors can promote to a floating dtype.

Return a PairedImageDataset with sample_ids, immutable records (sample_id, input_path and target_path fields), and pairs containing only input/target paths. Items are (input, target, sample_id) on CPU with shapes (C, H, W) and (C_target, H, W). The returned reader provides get_config() and get_preprocessing_contract(). The latter returns copied input and target mappings describing decoding, channel order, alpha and orientation policies, inversion, resize, interpolation, binarization and scaling. dtype records conversion; output_dtype records the effective dtype after scaling. Native file shape and dtype are recorded separately by audit_image_pairs().

classmethod from_suffixes(root_dir, *, suffix_in='_in', suffix_target='_target', prefix_in='', prefix_target='', extensions=('.png', '.jpg', '.pgm'), ordering='textual', unmatched='error', **reader_options)

Build strict pairs from literal filename prefixes and suffixes.

Index direct files in textual ID order without decoding. IDs are nonempty and preserved exactly. Suffixes must be distinct and nonempty; a file matching both roles or duplicate IDs within a role always fails. unmatched accepts error, warn or ignore for incomplete pairs only. Report exclusions through missing_input_ids and missing_target_ids. Extensions are case-normalized.

Reader options follow from_pairs(). Configuration roundtrips rescan this source and check its ordered ID digest. To freeze a historical selection against unrelated added files, use from_manifest().

train_test_split(test_size=0.25, shuffle=True, random_state=42)

Return (train_subset, test_subset) using stable pair indices.

class mtlearn.datasets.SplitValidationReport(sample_ids, sample_counts, group_counts, unassigned_ids, checks)

Bases: object

Describe the checked universe, split counts and unassigned sample IDs.

Counts distinguish samples from groups. group_counts is None when grouping is disabled. checks identifies enforced policies; a false policy does not claim that the data violate it. Containers returned by to_dict are copies.

Parameters:

• sample_ids (tuple[str, ...])

• sample_counts (Mapping[str, int])

• group_counts (Mapping[str, int] | None)

• unassigned_ids (tuple[str, ...])

• checks (Mapping[str, bool])

to_dict()

Return a JSON-compatible copy of the universe, counts and checks.

sample_ids: tuple[str, ...]

sample_counts: Mapping[str, int]

group_counts: Mapping[str, int] | None

unassigned_ids: tuple[str, ...]

checks: Mapping[str, bool]

mtlearn.datasets.validate_split_ids(sample_ids, split_ids, groups=None, *, require_complete=False, require_complete_groups=True, allow_empty=False)

Validate split membership without reading files or dataset items.

Parameters:

• sample_ids – Ordered nonempty source universe of unique nonempty strings. IDs are opaque; "01" and "1" remain distinct.

• split_ids – Mapping from nonempty split names to ordered ID iterables. No ID may occur twice within or across splits.

• groups – Optional complete mapping from source IDs to nonempty string group IDs. None disables grouping; an empty mapping does not.

• require_complete – Require every source ID to be assigned.

• require_complete_groups – Require every member of a selected group to be assigned, relative to the supplied universe. Groups never cross splits.

• allow_empty – Permit individual empty splits. At least one must be nonempty.

Returns:

A SplitValidationReport with source order and unassigned IDs preserved.

Raises:

• TypeError – An argument has an unsupported container or policy type.

• ValueError – IDs, groups, coverage or split membership are inconsistent.

mtlearn.datasets.subsets_from_ids(dataset, split_ids, *, sample_ids=None, groups=None, require_complete=False, require_complete_groups=True, allow_empty=False)

Return Subset views in exactly the requested split and sample order.

Use explicit sample_ids or resolve dataset.sample_ids and nested Subset indices. No call to __getitem__ discovers identities. Repeated source positions are invalid for partitioning; training samplers may still repeat positions. Remaining arguments follow validate_split_ids().

The original dataset and each item’s return type are preserved without copying tensors. A dataset without identity metadata requires explicit sample IDs.

mtlearn.datasets.split_ids_by_group(sample_ids, groups, group_counts, seed=42)

Partition all source groups using an isolated NumPy RandomState.

Parameters:

• sample_ids – Ordered unique nonempty string IDs.

• groups – Complete sample-ID to group-ID mapping.

• group_counts – Mapping of split names to positive integer group counts. Counts must sum to all groups. Mapping order assigns permuted groups.

• seed – Integer in [0, 2**32); booleans are rejected.

Returns:

Split names mapped to ordered ID lists. Group names are sorted textually before permutation; source order is preserved within each chosen group.

No global Python, NumPy or Torch random state changes. Counts refer to groups, not images or percentages. Persist the returned IDs to identify the selection.

class mtlearn.datasets.PairAudit(data, signatures=())

Bases: object

Hold detached metadata from one explicit audit of original files.

sample_ids preserves source order. sources records named relative paths, SHA-256, native shape and dtype, and optional callback metadata. preprocessing describes the effective reader separately. Public containers are defensive copies. In-process file signatures detect ordinary source changes before creating a manifest; they are excluded from content fingerprints. Obtain this object with audit_image_pairs(). Native metadata describes OpenCV IMREAD_UNCHANGED output, before the dataset reader’s transformations.

property preprocessing

Return a copy of the effective input and target reader contracts.

property sample_ids

Return audited textual IDs in source order as a tuple.

property sources

Return copied input/target file references and callback metadata by ID.

to_dict()

Return JSON-compatible audit data without local file signatures.

mtlearn.datasets.audit_image_pairs(dataset, roots, validate_sample=None)

Audit explicit image pairs with streamed hashes and one decoded pair at a time.

Parameters:

• dataset – Dataset returned by an explicit PairedImageDataset factory. Its records and effective reader options define the audited contract.

• roots – Mapping containing input and target directories. Every file must resolve within its role’s root, including through symbolic links.

• validate_sample – Optional callback (record, input_array, target_array). Arrays are read-only native OpenCV IMREAD_UNCHANGED results: color channels are BGR/BGRA, not the reader’s RGB tensors. Return None or a JSON mapping with finite numeric values; raise to reject a sample. The callback must not retain arrays or mutate sources or reader configuration.

Returns:

A PairAudit. Native input/target spatial sizes must match; binary masks, grayscale, uint8 and matching channel counts are not imposed by the audit.

Hashing uses bounded blocks. Only one decoded pair is retained by this function. File signatures are rechecked across the audit and when constructing a manifest; a detected change requires a fresh audit. This is a snapshot check, not a lock against later filesystem changes. No audit is added to dataset item access.

class mtlearn.datasets.SplitManifest(data)

Bases: object

Store a validated, portable image-selection document with defensive copies.

The format is mtlearn.dataset-manifest, schema version 1. It records the complete audited universe, splits, optional groups, policies, native source metadata, effective reader options, preprocessing contracts and provenance. Construct from from_audit(), load(), or a complete schema mapping.

File references use canonical relative POSIX paths and named roots. Root locations are supplied at execution time. Loading validates structure without reading images. Call validate_files() explicitly before using a saved selection, and validate_preprocessing() for an independently built reader.

fingerprint(kind='manifest', *, split=None)

Hash a validated identity component, optionally for one named split.

input and target hash ID/file-hash pairs sorted by textual ID. They exclude paths and selection order. splits hashes ordered membership, selected groups and policies. preprocessing_input and preprocessing_target describe each role’s effective reader operations. manifest includes all schema fields except provenance, including relative paths and audit metadata, and guards compact worker reconstruction.

Absolute roots and timestamps in provenance are excluded. JSON map order and formatting do not affect hashes; list order does. File hashes identify original bytes, not decoded tensors or morphological cache entries. A split-scoped hash excludes other splits’ sources and groups. Never use a whole-manifest hash as an indiscriminate morphology-cache key.

fingerprints(*, split=None)

Return all six identity components for the universe or a selected split.

Preprocessing components are shared across splits; content and membership are restricted when split is supplied. No files are opened or hashed here.

classmethod from_audit(audit, split_ids, groups=None, *, preprocessing=None, require_complete=True, require_complete_groups=True, allow_empty=False, provenance=None)

Create a manifest from audited sources and validated split membership.

Preprocessing is derived from the audited reader. An optional supplied contract must match it exactly. Split policies follow validate_split_ids(), with complete coverage enabled by default. Optional provenance is a JSON mapping, for example a seed, algorithm name or decoder version; it is excluded from fingerprints. Reuse an audit for several partitions only while its files remain unchanged. A reduced selection that cuts a group requires require_complete_groups=False; preserve the full manifest separately.

property groups

Return a copied sample-to-group mapping, or None when grouping is off.

classmethod load(path)

Load and validate schema and contracts without reading source images.

Reject unknown schemas, duplicate JSON keys, nonfinite numbers, invalid paths and inconsistent IDs, splits or reader contracts. Integer readers with scaling require the saved effective floating dtype to match the current Torch default. No legacy schema is inferred.

pairs(*, roots, split=None)

Resolve saved records to explicit (sample_id, input_path, target_path) tuples.

The saved order is preserved and directories are not rescanned for new samples. Check existence and containment, rejecting symbolic links outside their roots. This operation does not decode or verify file hashes; use validate_files for that explicit preflight. A selected empty split returns an empty list.

property preprocessing

Return copied effective reader contracts for input and target.

property sample_ids

Return the complete audited universe in source order as a tuple.

save(path)

Write JSON through a unique temporary file and atomic replacement.

The parent directory must exist. A failed replacement preserves the previous file and removes the temporary file. The manifest itself remains unchanged.

property splits

Return copied split names and ordered sample-ID lists.

to_dict()

Return a detached, JSON-compatible copy of the validated schema.

validate_files(*, roots, split=None, raise_on_error=True)

Verify saved original bytes without decoding or modifying image files.

Return checked sample IDs, a file count and issues identifying sample and role. By default divergences raise ValueError. With raise_on_error=False, collect file-level issues in the report. Invalid root configuration still raises. Validation must be repeated if sources can change after this snapshot.

validate_preprocessing(dataset_or_contract)

Check an explicit reader, its Subset, or an effective contract mapping.

Return a successful role report or raise ValueError naming changed roles. Checking file hashes alone cannot detect inversion, scaling or resize changes.