Transforms

Added in version 1.5.0.

The s3dgraphy.transforms package collects graph rewriting passes used by the GraphML exporter (and, in some cases, by hand before export) to produce a minimal, formally clean GraphML without losing resolver fidelity.

Three modules live here, each with a focused responsibility.

compact — reverse-propagation compaction

When stratigraphic units in the same swimlane repeat the same has_author / has_license / has_embargo declaration node by node, the workbook is redundant: every unit re-states something already true for the whole epoch. The compact transform lifts those declarations to the swimlane-level Paradata Node Group (SL_PD) and removes the per-unit copies. The resolver behaviour is identical before and after — every node still returns the same value, but via the swimlane-level lookup instead of a local edge.

Two passes:

s3dgraphy.transforms.compact.prune_redundant_propagative_edges()

Removes per-node has_author / has_license / has_embargo edges (and has_property edges to temporal PropertyNodes) when the swimlane-level resolver returns the same value anyway. Pure local pruning — no new nodes are created.

s3dgraphy.transforms.compact.hoist_propagative_metadata()

Promotes a shared per-node declaration into an SL_PD when every stratigraphic unit whose primary swimlane is Epoch E declares the same single target. A new SL_PD is created only if no free SL_PD already covers the swimlane; otherwise the existing one is reused.

s3dgraphy.transforms.compact.compact_propagative_metadata()

Convenience orchestrator that runs hoist then prune. Idempotent: a second call is a no-op.

Conservative invariants:

• Only AUTHOR, LICENSE, EMBARGO are promoted. Chronology PropertyNodes are pruned when redundant but never hoisted — that would require PropertyNode deduplication, out of scope.

• Strat nodes associated with multiple epochs (survive_in_epoch in addition to has_first_epoch) are skipped by hoist — their primary swimlane is ambiguous.

• Hoist requires every strat unit in a swimlane to declare the same target. Partial overlap never hoists, to avoid silently crediting units that had no declaration.

Locked in by tests/test_compact_metadata.py (8 synthetic scenarios plus idempotence of the orchestrator).

materialize_continuity — diamond materialization on export

The GraphML importer expands continuity diamonds (BRs) into has_first_epoch + survive_in_epoch edges on the strat node they terminate. The Blender side then works with the edge chain and never carries the diamond in memory.

On export, we have to re-create the diamond, otherwise the round-trip breaks — the importer would assume default life rules and the user’s life-bound configuration would be lost on the next reload.

s3dgraphy.transforms.materialize_continuity.materialize_continuity_diamonds()

Family-aware rules using the single source of truth from s3dgraphy.classification:

• REAL strat types (REAL_US_TYPES) live forever by default. A diamond is emitted iff the actual last_epoch is not the most recent epoch in the graph (the node has a bounded life).

• VIRTUAL strat types (VIRTUAL_US_TYPES) live only in their birth epoch by default. A diamond is emitted iff the graph contains at least one survive_in_epoch edge on the node — its life has been extended beyond birth, which only a diamond can express in GraphML.

The synthetic diamond is wired to last_epoch via a synthetic has_first_epoch edge. The exporter’s position calculator (EpochSwimlanesGenerator) uses it to decide which swimlane row to place the diamond in. At re-import, the diamond’s y-position sits inside last_epoch’s vertical range and the importer’s gate epoch.max_y > continuity_y_pos correctly bounds survive_in_epoch to the right span.

All injected content is tagged injected_by="materialize_continuity" so it is treated as auxiliary (see aux_tracking below) and removed by the volatile save.

aux_tracking — Hybrid-C auxiliary lifecycle

The auxiliary layer in EM-tools / s3dgraphy covers DosCo documents, emdb property tables, pyArchInit SQLite, sources-list xlsx and configurable resource-folders. Auxiliary files attach to existing host nodes via a stable key ID (D.NN for documents, US / USV ids for stratigraphic units). They never create new top-level entities by themselves: they either

1. add enrichment child nodes and edges (e.g. PropertyNode children added by emdb, LinkNode children added by DosCo), or

2. update attributes on the host node (e.g. DosCo setting DocumentNode.url, sources-list setting author / license / embargo).

The s3dgraphy.transforms.aux_tracking module provides the bookkeeping primitives so a later exporter pass can either keep the auxiliary layer ephemeral (volatile save, default) or commit it into the GraphML (bake).

Two complementary tracks:

Tagging enrichment children

mark_as_injected(obj, injector_id)() / is_injected(obj)() — tag enrichment children with attributes['injected_by']. injector_id is free-form, conventionally "<kind>:<source-path>" (e.g. "DosCo:/path/to/DosCo").

Recording attribute overrides

record_attribute_override(node, attr, injector_id, original)() + freeze_aux_value(node, attr)() — capture the pre-aux value when an auxiliary mutates a host-node attribute. Used by the volatile save’s reversal policy.

Volatile save policy:

apply_override_reversal_policy(graph)() walks every host node with _aux_overrides and, for each attribute, reverts to the original value only if the current value is still the aux value; otherwise it drops the override record (the user re-edited the attribute manually after the auxiliary applied — user wins).

Bake save policy:

clear_aux_tags(graph)() drops every injected_by tag and every _aux_overrides record. The enrichment layer becomes graph-native going forward.

Side-effect summary:

Function	Side effect
mark_as_injected	Writes obj.attributes['injected_by']
record_attribute_override	Appends to node.attributes['_aux_overrides']
freeze_aux_value	Updates the override’s current aux value
apply_override_reversal_policy	Mutates node attributes, drops override entries
strip_injected_content	Removes nodes/edges tagged injected_by
clear_aux_tags	Removes the two bookkeeping keys

All other helpers (aux_overridden_attrs, iter_injected, push_orphan / iter_orphans / clear_orphans) are read-only.

Exporter dispatch

GraphMLExporter and GraphMLPatcher take a persist_auxiliary flag:

exporter.export("out.graphml")                          # volatile (default)
exporter.export("out.graphml", persist_auxiliary=True)  # bake

• False (volatile): apply the reversal policy and strip injected content before emitting. The on-disk GraphML reflects only the graph-native state. On the next reload the auxiliaries re-inject cleanly.

• True (bake): emit everything verbatim and clear the injected_by / _aux_overrides bookkeeping. The enrichment layer is promoted to graph-native.

Idempotence:

The bake then reload roundtrip is idempotent. The volatile save is idempotent in the absence of user edits between iterations (covered by tests/test_aux_roundtrip_graphml.py).

When to run the transforms

The standard 1.5 export pipeline is:

1. Run any project-specific edits in memory.

2. Optional: compact_propagative_metadata(graph) for a smaller, formally cleaner GraphML.

3. GraphMLExporter.export(path) — implicitly calls materialize_continuity_diamonds and the volatile save policy.

The compact pass is not run by the exporter automatically — it is a conscious editorial decision (you might want per-unit declarations for granular tracking).

See also

• Property Propagation — the resolver whose behaviour the transforms preserve.

• Temporal Inference Engine — the temporal layer that compact defers to (chronology PropertyNodes are pruned but never hoisted).

• CHANGELOG.md root entry [Unreleased] / [1.5.x] for the detailed test matrix.

API Reference

Reverse-propagation compaction for GraphML export.

Given a graph where stratigraphic units repeat the same author / license / embargo declarations at the node level, lift those declarations into the swimlane-level Paradata Node Group (SL_PD). The resolver behaviour is preserved: every node still returns the same value, but via the swimlane-level lookup instead of a local edge. The result is a smaller, more formally structured GraphML file.

Two complementary passes are exposed:

• prune_redundant_propagative_edges() removes per-node has_author / has_license / has_embargo / has_property (temporal) edges whose target is already reachable via the swimlane-level resolver. It is purely local — no new nodes are created.

• hoist_propagative_metadata() promotes a shared per-node declaration into an SL_PD when every stratigraphic unit in a swimlane points to the same target instance. A new SL_PD is created only if no free SL_PD already covers the swimlane; otherwise the existing one is reused.

compact_propagative_metadata() runs hoist then prune.

Conservative by design:

• Only AUTHOR, LICENSE, EMBARGO are currently promoted. Chronology PropertyNodes are pruned when redundant but never hoisted, because that would require deduplicating PropertyNode instances.

• Strat nodes associated with multiple epochs (survive_in_epoch in addition to has_first_epoch) are skipped by hoist — their primary swimlane is ambiguous.

• Hoist requires every strat unit in a swimlane to declare the same single target. Partial overlap never hoists, to avoid silently crediting units that had no declaration.

s3dgraphy.transforms.compact.prune_redundant_propagative_edges(graph) → Dict[str, int]

Remove per-node has_author / has_license / has_embargo edges (and has_property edges to temporal PropertyNodes) whose value matches what the swimlane-level resolver returns for the same node. The graph is mutated in place.

Returns a report: {rule_id: edges_removed}.

s3dgraphy.transforms.compact.hoist_propagative_metadata(graph) → Dict[str, int]

Promote a shared per-node declaration to the swimlane-level SL_PD.

For each epoch E and each rule in (author, license, embargo): if every stratigraphic unit whose primary swimlane is E carries the same single has_<rule> target UUID (no more, no less, no multi-epoch nodes, no empty sets), hoist the declaration:

1. Ensure a free SL_PD exists for E (create if missing).

2. Attach the shared target to the SL_PD via is_in_paradata_nodegroup.

3. Remove the per-unit has_<rule> edges.

The resolver continues to return the same value for every node, via the swimlane path. Returns a report of how many edges were hoisted per rule.

Chronology is not hoisted (would require PropertyNode deduplication across units).

s3dgraphy.transforms.compact.compact_propagative_metadata(graph) → Dict[str, Dict[str, int]]

Run hoist_propagative_metadata() then prune_redundant_propagative_edges() on graph. Returns a combined report with two top-level keys: "hoisted" and "pruned".

Export-time inverse of the importer’s continuity-diamond expansion.

The GraphML importer (import_graphml.py:1601+) reads a continuity diamond (BR) and derives has_first_epoch + survive_in_epoch edges on the strat node it terminates. The Blender side then works with the edge chain and never carries the diamond in memory: a node lives in Epoch_N (has_first_epoch) and survives into Epoch_M (survive_in_epoch), and that is all the in-memory graph knows.

When exporting back to GraphML, we have to RE-CREATE the diamond, otherwise the round-trip semantics break — the importer would assume default life rules and the user’s life-bound configuration would be lost on the next load.

Family-aware rules (single source of truth: s3dgraphy.classification):

• REAL strat types (REAL_US_TYPES) live forever by default. When no terminator is present the importer adds survive_in_epoch for every epoch more recent than the birth epoch. We must emit a diamond iff the actual last_epoch is NOT the most recent epoch in the graph (the node has a bounded life).

• VIRTUAL strat types (VIRTUAL_US_TYPES) live ONLY in their birth epoch by default. Without a terminator the importer adds no survive_in_epoch. We must emit a diamond iff the in-memory graph has at least one survive_in_epoch edge on the node — its life has been extended beyond birth, which only a diamond can express in GraphML.

Placement: the synthetic diamond is wired to last_epoch via a synthetic has_first_epoch edge. That’s how the exporter’s position calculator (EpochSwimlanesGenerator) decides which swimlane row a node falls into. At re-import, the diamond’s y_pos sits inside last_epoch’s vertical range and the importer’s gate epoch.max_y > continuity_y_pos correctly bounds survive_in_epoch to the right span.

All injected content is tagged injected_by="materialize_continuity" so s3dgraphy.transforms.aux_tracking.revert_injector() (re-exposed here as dematerialize_continuity()) can drop it at the end of the export, leaving the in-memory graph identical to its pre-export state.

Idempotent: a strat node that already has an incoming edge from a BR (user-authored diamond in the graph) is skipped — we never duplicate an existing terminator.

s3dgraphy.transforms.materialize_continuity.materialize_continuity(graph) → Dict[str, int]

Inject synthetic ContinuityNode diamonds + edges so a re-import of the exported GraphML reproduces the in-memory life chain.

Self-healing: any leftover synthetic content from a previous failed run (tagged injected_by=INJECTOR_ID) is swept first.

Idempotent vs user-authored continuity: strat nodes that already have an incoming edge from a BR are left untouched.

Returns {"nodes", "edges", "skipped_user_authored", "swept"}.

s3dgraphy.transforms.materialize_continuity.dematerialize_continuity(graph) → Dict[str, int]

Remove every node and edge tagged injected_by="materialize_continuity", plus any dangling edges that would result.

Thin pinned wrapper over s3dgraphy.transforms.aux_tracking.revert_injector() so callers don’t have to remember the injector id.

Returns {"nodes", "edges"}.

Hybrid-C auxiliary-lifecycle tracking (Phase 1 of HYBRID_C_AUXILIARY_LIFECYCLE.md).

The “auxiliary” layer in EMtools/s3Dgraphy covers DosCo documents, emdb property tables, pyArchInit SQLite, sources-list xlsx and configurable resource-folders. Auxiliary files attach to existing host nodes via a stable key ID (D.NN for documents, US / USV ids for stratigraphic units). They never create new top-level entities by themselves: they only

1. add enrichment child nodes and edges (e.g. PropertyNode children added by emdb, LinkNode children added by DosCo),

2. update attributes on the host node (e.g. DosCo setting DocumentNode.url, sources-list setting author / license / embargo).

This module provides the bookkeeping primitives so a later exporter pass can either keep the auxiliary layer ephemeral (volatile save, default) or commit it into the GraphML (bake).

Two complementary tracks:

• mark_as_injected(obj, injector_id) / is_injected(obj) — tag enrichment children with attributes['injected_by'].

• record_attribute_override(node, attr, injector_id, original) / aux_overridden_attrs(node) — record each auxiliary-induced attribute change so the volatile save can revert it (or keep the current value if the user re-edited it manually afterwards).

Additional helpers:

• strip_injected_content(graph, keep_ids=None) — remove every node and edge tagged injected_by from a graph in-place. Returns the number of items removed.

• apply_override_reversal_policy(graph) — walk every host node with _aux_overrides and, for each attribute, revert to the original value only if the current value is still the aux value; otherwise drop the override record (the user re-edited).

• clear_aux_tags(graph) — clear both injected_by tags and _aux_overrides across the whole graph. Used by the bake path to promote the enrichment layer to graph-native.

All functions are side-effect-free unless documented otherwise.

s3dgraphy.transforms.aux_tracking.mark_as_injected(obj, injector_id: str) → None

Tag an enrichment node or edge as coming from an auxiliary.

injector_id is a free-form string, conventionally in the form "<kind>:<source-path>": "DosCo:/path/to/DosCo", "emdb:/path/x.xlsx", "pyarchinit:/path/x.sqlite", "sources_list:/path/sources.xlsx", "resource_folder:/path/resources".

The tag lives on the object’s attributes dict under the key "injected_by". Nodes and edges both expose an attributes dict in s3Dgraphy.

s3dgraphy.transforms.aux_tracking.is_injected(obj) → str | None

Return the injector id stamped on obj, or None if the object is graph-native (no injected_by tag).

s3dgraphy.transforms.aux_tracking.record_attribute_override(node, attr_name: str, injector_id: str, original_value: Any) → None

Record that an auxiliary touched node.<attr_name> and that its pre-auxiliary value was original_value.

Call this before applying the auxiliary change so the original_value you pass is the real pre-aux value. If the attribute did not exist prior to the aux, pass MISSING_SENTINEL (or the module-level _MISSING).

The override is stored on the node’s attributes dict under the special key "_aux_overrides" as a dict:

{attr_name: {"injector": injector_id, "aux_value": <set by apply_override_reversal_policy>, "original": original_value}}

The aux_value slot is filled by freeze_aux_values() just after the injector finishes its pass, so that apply_override_reversal_policy can later compare the current value against the aux value (to detect user edits) without requiring the injector to track it by itself.

s3dgraphy.transforms.aux_tracking.freeze_aux_value(node, attr_name: str) → None

Called by the injector AFTER it has written the new value to node.<attr_name>. Copies the freshly-written value into the override record so the reversal policy can later compare against it.

If no override record exists (injector didn’t call record_attribute_override first), this is a no-op — the attribute is treated as graph-native.

s3dgraphy.transforms.aux_tracking.aux_overridden_attrs(node) → Dict[str, Dict[str, Any]]

Return the _aux_overrides dict for node (empty dict if none). Keys are attribute names; values are {"injector", "original", "aux_value"} triples.

s3dgraphy.transforms.aux_tracking.strip_injected_content(graph) → Dict[str, int]

Remove every node and edge whose attributes['injected_by'] is set. Returns {"nodes": N, "edges": M}.

Edges are removed first (to avoid dangling endpoints), then the nodes are removed. Any edge that becomes dangling because its endpoint was removed is also dropped.

This mutates graph in place.

s3dgraphy.transforms.aux_tracking.apply_override_reversal_policy(graph) → Dict[str, int]

Walk every node with _aux_overrides and apply the Hybrid-C volatile-save policy per-attribute:

• If current value == aux_value → restore original value (the user never re-edited this attribute after the aux applied).

• If current value != aux_value → keep the current value (the user re-edited manually after the aux) and drop the override entry; the attribute becomes graph-native going forward.

Returns {"reverted": R, "kept": K, "unseen": U} where R is the number of attributes rolled back to their pre-aux value, K is the number kept because the user re-edited, U is the number of entries whose aux_value was never frozen (injector didn’t call freeze_aux_value()); for those we conservatively keep the current value.

This mutates graph in place (removes _aux_overrides on each node after processing, since the policy is applied once per save).

s3dgraphy.transforms.aux_tracking.clear_aux_tags(graph) → Dict[str, int]

Drop all injected_by tags and _aux_overrides from every node and edge in graph. Used by the bake path to promote the enrichment layer to graph-native.

Returns {"injected_cleared": N, "overrides_cleared": M}.

s3dgraphy.transforms.aux_tracking.push_orphan(graph, injector_id: str, key_id: str, payload: dict) → None

Record that an auxiliary row with key key_id could not be attached to any host node in the graph. Orphans live under graph.attributes['aux_orphans'] as a list of dicts {"injector", "key_id", "payload"}.

The UI layer (EMtools) can render these per-auxiliary and offer a “create host node” action.

s3dgraphy.transforms.aux_tracking.iter_orphans(graph, injector_id: str | None = None) → Iterable[dict]

Yield orphan entries in graph.attributes['aux_orphans']. When injector_id is provided, only entries matching that injector are yielded.

s3dgraphy.transforms.aux_tracking.clear_orphans(graph, injector_id: str | None = None) → int

Drop orphan entries. Returns how many were removed.

With injector_id: only entries for that injector (used when an auxiliary is unregistered). Without: clears the whole list (used by the bake path).

s3dgraphy.transforms.aux_tracking.revert_injector(graph, injector_id: str) → Dict[str, int]

Remove every node / edge injected by injector_id and apply the volatile-save reversal policy for the attribute overrides attributed to the same injector only. Other injectors’ content stays untouched.

Semantics (scoped variant of the strip + policy pair used by GraphMLExporter on volatile save):

• A node / edge carrying attributes["injected_by"] == injector_id is removed. Edges touching a removed host-node endpoint are also dropped to avoid dangling references.

• For each host-node attribute in _aux_overrides where the recorded injector matches injector_id:

  • current value == frozen aux value → restore the original pre-aux value;

  • current value differs → user re-edited after the aux applied → keep current value, drop the override record;

  • no aux_value frozen → conservatively keep the current value and drop the override record.

• Orphan entries filed under this injector are cleared.

Returns a report {"nodes", "edges", "reverted", "kept", "unseen", "orphans_cleared"}.

This is the helper used when the user unregisters a single auxiliary without rebuilding the whole graph — for example the DosCo importer exposing an “Unregister DosCo” button in the UI. Unlike strip_injected_content(), which removes all injected content regardless of source, this preserves content from other auxiliaries so multi-aux setups behave predictably.