CFP Preparation and Storage

Import these public components from mtlearn.layers.cfp. They separate fixed CPU morphology from learned parameters, normalization and optional retention. For fitting dataset statistics and using cached batches, see Connected Filter Preprocessing. Layer methods such as fit_stats, set_stats and forward_prepared are documented in Layers.

Preparation

class mtlearn.layers.cfp.CFPPreprocessor(*, tree_specs, features, attribute_dtype=numpy.float32, morphology_module=<module 'mtlearn.morphology' from '/home/runner/work/MTLearn/MTLearn/mtlearn/python/mtlearn/morphology.py'>)

Bases: object

Build CPU topology and raw attributes, one image/channel/tree at a time.

classmethod from_layer(layer)

Copy only tree/feature contracts, never parameters, stats or device.

prepare_image(image, tree_key=None, *, input_id=None)

Validate and prepare one 2D image using the existing uint8 conversion.

Inputs may be tensors or NumPy arrays. Multiple trees require an explicit tree_key. No images or returned payloads are retained by this object.

prepare_batch(images, *, store=None, sample_ids=None)

Prepare one dense BCHW batch; default to no persistent retention.

Explicit stores key the effective uint8 pixels, shape, tree, attributes, dtype and process-local implementation identity. Logical IDs only label the returned batch. Keep batch size small to bound active memory.

prepare(source, *, store=None, manifest=None, source_version=None, preprocessing_version=None, split='train', sample_ids=None, collect_stats=None, cancel=None, num_workers=0, max_in_flight=None, worker_threads=1, max_sample_bytes=67108864, max_retries=0, progress=None)

Prepare a finite map-style source, returning only progress/statistics.

A persistent store requires named/versioned source and preprocessing contracts. Pass collect_stats=layer.get_statistics_contract() for a train-only snapshot. With a DiskStore, num_workers > 0 prepares in spawned CPU processes. Source/targets stay in the caller; only canonical uint8 images enter the bounded queue (max_in_flight, default num_workers). max_sample_bytes limits each canonical input, not native workspace/RSS. max_retries retries ordinary task errors on the same pixels; abrupt worker death and quota exhaustion require an explicit resume. A cancel callable/event is checked between samples and while waiting. Resume by repeating the same call; completed content is verified/reused.

prepare_u8(image_np, tree_key, *, input_id=None)

Prepare an already canonical CPU uint8 image without normalization.

class mtlearn.layers.cfp.PreparedMorphology(tree_spec, feature_spec, info, raw_attributes, input_id=None, format_version=4)

Bases: object

One CPU image/channel/tree. Tensor buffers are read-only by contract.

Mappings are copied and protected, while tensors are shared without copying. Consumers must not mutate those tensors. input_id is a logical label, not a content fingerprint or permission to reuse changed pixels.

Parameters:

• tree_spec (TreeSpec)

• feature_spec (FeatureSpec)

• info (Mapping[str, Any])

• raw_attributes (Mapping[Any, torch.Tensor])

• input_id (str | None)

• format_version (int)

property nbytes: int

Bytes of unique CPU storages, including storage behind any views.

validate(*, full=False)

Check the schema; optionally scan values and traversal consistency.

Return type:

None

Parameters:

full (bool)

class mtlearn.layers.cfp.PreparedBatch(samples, sample_ids=None)

Bases: object

Morphology grouped as samples[batch][channel][tree_key].

All channels share a spatial shape. Logical sample IDs affect CFPContext, never cache identity. Tensors are shared and read-only by contract. A batch owns its active data even after the corresponding store entries are evicted.

Parameters:

• samples (tuple)

• sample_ids (tuple | None)

validate_for(layer)

Check compatibility before transferring tensors or scoring any sample.

Extra trees/attributes are allowed, enabling preparation shared by models with different feature subsets. Required raw dtypes must match the layer.

class mtlearn.layers.cfp.PreparationResult(store_path, manifest, sample_count, status, statistics=None, statistics_id=None)

Bases: object

Progress and optional frozen statistics; never a list of tensor payloads.

Parameters:

• store_path (str | None)

• manifest (str | None)

• sample_count (int)

• status (str)

• statistics (StatisticsSnapshot | None)

• statistics_id (str | None)

class mtlearn.layers.cfp.PreparedDataset(*args, **kwargs)

Bases: Dataset

Resolve one manifest sample at a time, optionally paired with a live source.

With source, input content is verified on every access and its target is returned. Without a source this is an immutable prepared snapshot, independent of current image files and labels. The source must match manifest order; apply samplers or Subset to this PreparedDataset for training order changes. P3 consumption uses DataLoader num_workers=0; each DiskStore is process-local.

mtlearn.layers.cfp.collate_prepared(samples)

Collate prepared samples (and optional targets) without copying raw tensors.

Storage policies

class mtlearn.layers.cfp.NullStore

Bases: _ConsumerTracker

Return prepared data to callers without retaining it.

Stores are local to one process and intended for sequential use. Active-byte metrics cover returned preparation handles, including handles kept alive by CFP autograd; they exclude normalized/device tensors and allocator overhead.

put(key, prepared)

Return a consumer handle; callers must treat its tensors as read-only.

get_or_prepare(key, factory)

Resolve an immutable content key or invoke the zero-argument factory.

clear()

Drop retained entries without invalidating consumer handles or counters.

class mtlearn.layers.cfp.MemoryStore(max_bytes)

Bases: NullStore

Retain unique CPU storages up to max_bytes (not a process RSS limit).

Entries larger than the budget are returned without admission. Eviction drops only the store’s reference; active batches and their backward remain valid. Keys supplied directly to this low-level store must identify immutable content. CFPPreprocessor calculates keys from canonical pixels and preparation options.

put(key, prepared)

Return a consumer handle; callers must treat its tensors as read-only.

clear()

Drop retained entries without invalidating consumer handles or counters.

class mtlearn.layers.cfp.DiskStore(path, *, max_disk_bytes=None, max_ram_bytes=0, readonly=False, mmap=True, validation='always', immutable=False, max_validation_entries=1024, min_free_disk_bytes=0)

Bases: object

Versioned local SSD entries with bounded RAM and restricted CPU loading.

Set a tensor-file quota explicitly for a writer. max_ram_bytes=0 is the default. Quota counts published files and temporary tensor files; SQLite, lock files, filesystem allocation and OS page cache are reported separately. One writer may coexist with readonly readers. Use each handle sequentially in the process that opened it, and close it (or use a context manager).

recover()

Recover fully published orphans; discard owned partial temporary files.

Valid registered files are checked lazily on reads. Recovery visits one entry at a time and never retains a dataset of mapped tensors.

get(key)

Read a content key, verifying file checksum before restricted loading.

get_or_prepare(identity, factory)

Resolve canonical content, or prepare/repair it without touching model stats.

put(identity, prepared)

Publish a complete entry atomically; never delete a valid entry for quota.

clear()

Release only the RAM cache. Persistent entries and active handles survive.

statistics(manifest, contract)

Recombine persisted scalar summaries in manifest order, without trees.

Normalization snapshot

class mtlearn.layers.cfp.StatisticsSnapshot(contract, _statistics, sample_count=None)

Bases: object

An immutable statistical contract with defensive copies of CPU scalars.

statistics returns copies: callers cannot change the installed snapshot by mutating returned tensors. Source sample_count may be unknown for legacy checkpoints; it counts logical images, not tree nodes or unique contents.

Parameters:

• contract (Mapping[str, Any])

• _statistics (Mapping[str, Any])

• sample_count (int | None)

classmethod combine(snapshots)

Combine snapshots in iterable order without retaining per-image data.