EM Data Tree

See also

In the Extended Matrix language manual:

• Extended Matrix — language manual — overview of the formal notation feeding the .graphml files loaded here.

• EM workspace preparation — DosCo folder layout and iteration pattern.

Fig. 2 EM Data Tree Panel

This panel (Fig. 2) allows to create the first connection between Blender and the Extended Matrix (.graphml file). Press the Add GraphML and locate the .graphml file wiin the Path section (NB: before closing the path window remember to uncheck relative path within the settings. Alternatively, it is possible to paste the entire path within the empty line).

Fig. 3 GraphML import

When a GraphML is loaded (Fig. 3), on the left side of the EM Data Tree window the GraphML ID will appear (for example, the ID: GT16). On the same line, on the right side, a green square will show up. The green color cofirms that a connection between the GraphML and EMtools has been established.

Note

To correctly link the .graphml file with the EM Data Tree panel it is mandaotry to insert, at least, the GraphML ID on the title of the Swimlane node (1.5 dev4 palette); the first node that needs to be imported in the yEd space to start the creation of an Extended Matrix (Fig. 4).

Example:

Context name [ID:xx;ORCID:xx;LICENSE:CC-BY-ND]

Great Temple [ID:GT16;LICENSE:CC-BY-ND]

Fig. 4 EM graphml file title example

Next to the green (or red) square, three buttons complete the right side of the line (Fig. 3): the Update button allows to refresh the .graphml file, if changes have been applied on the EM graph during the modelling session; the Activate EM button consent to explore only the selected GraphML; and the pubblish button appears when the multigraph mode is enabled.

This version of EMtools allows to activate the Multigraph Mode, this option consent to upload and visualize multiple graphs. To upload a new graph and explore its information user can follow all the steps already explained.

Note

This version of EMtools include info boxes. When the info button is selected an info box will appears with more information related to that specific part of the tool.

Fig. 5 Example of EM info box

Once the connection has been established, EMTools will summarize the most important information (US/USV; Epochs; Properties; Sources) within a simple table under the Path section (Fig. 3).

The Remove GraphML button allows to remove one or more EMs from the EM Data Tree list.

Note

Loading DosCo documents into the graph

The panel commands described below load documents from your DosCo folder into the EM graph as nodes. The link between the file on disk and the graph node is preserved; if you re-iterate your DosCo (add new entries, update existing ones), reload from this panel to surface the changes. For the DosCo concept, folder layout and iteration pattern, see Iterating the DosCo in the Extended Matrix language manual.

In this panel (Fig. 3) users can also link the path to the DosCo folder, where sources are stored. To locate sources, users must follow the same guidelines previously outlined for the localization of the EM file.

If the EM graph presents a connection with and external database, EMTools allows to import databases to maintain data connection also within Blender.

To establish the connection with EMtools:

• expand the Auxiliary files section and press Add;

• select the type (Generic Excel, PyArchInit, EMdb Excel);

• indicate the exact location of the Auxiliary file and click on the Accept button.

Note

When EMdb Excel type is selected a Format menu appears, select the correct format from the list.

Auxiliary files — concept

Auxiliary files are not part of the EM graph’s core structure. The graph (.graphml) defines stratigraphy, proxies, sources and their logical relations; auxiliary files provide additional, tabular information that EMtools attaches to existing graph nodes at import time. They live outside the graph and can be re-imported, re-mapped or detached without altering the topology of the reconstruction.

The three currently supported auxiliary types behave the same way conceptually but differ in their source format:

• Generic Excel — any .xlsx with a header row; mapping is defined by the user.

• pyArchInit — a pyArchInit SQLite database; mapping is pre-configured for the pyArchInit US table (see Auxiliary Resources: pyArchInit and External Data Connections).

• EMdb Excel — a tabular database structured according to the EMdb conventions; mapping is driven by a JSON mapping file (see JSON mapping template below).

Important

Auxiliary files in this sense (tabular data attached at import time via a mapping) must not be confused with auxiliary stratigraphic nodes in the EM language — Continuity and related node types that belong to the formal notation. The former is a data plumbing feature of EMtools; the latter is a modelling primitive of the language. See the EM language manual for the latter.

Typical use cases:

• enrich existing US/USV nodes with material qualia, dating ranges, excavator notes that live in a separate spreadsheet;

• link photo paths and metadata to stratigraphic units without redrawing the graph in yEd (see Linking Photos as Auxiliary Files for the step-by-step procedure);

• merge field-database records (pyArchInit) into an already authored graph.

JSON mapping template

For the EMdb Excel (and Generic Excel) auxiliary types, EMtools consumes a JSON mapping file that declares how each column of the spreadsheet should be projected onto graph properties. A minimal mapping has the following shape:

{
  "version": "1.0",
  "sheet_name": "Stratigraphy",
  "start_row": 2,
  "id_column": "A",
  "mappings": [
    {"column": "A", "target": "human_id",      "type": "identifier"},
    {"column": "B", "target": "node_type",     "type": "node_type"},
    {"column": "C", "target": "epoch",         "type": "epoch"},
    {"column": "D", "target": "material",      "type": "qualia"},
    {"column": "E", "target": "photo_path",    "type": "document_link"}
  ]
}

Each row of the spreadsheet (from start_row onward) is matched against an existing graph node via id_column. For each entry in mappings, EMtools writes the value of the indicated column onto the target property of the matched node (or creates the property if absent). Supported type values mirror the kinds of data the EM language recognises (identifier, node_type, epoch, qualia, document_link, …); the exact list depends on the importer in use and is documented alongside the operator in API Reference.

Todo

Ship a canonical emdb_mapping.template.json next to the addon and reference it here once finalised. Until then the shape above is the reference; live working examples can be inspected in the Basilica Iulia dataset used by Bulk Import via the Excel Mapping Tool.

See also

• Bulk Import via the Excel Mapping Tool — bulk import via the JSON mapping tool, end-to-end on a 2 000+ row dataset.

• Auxiliary Resources: pyArchInit and External Data Connections — pyArchInit as auxiliary file (live link to the field database).

• Linking Photos as Auxiliary Files — link photos as auxiliary files.

Starting from version 1.5, EMtools allows to link external resource folders containing photos, 3D scans, documents and other media files to your Extended Matrix project. This feature is particularly useful when working with large image collections that need to be referenced and previewed directly from within Blender.

Setting Up Resource Folders

To link a resource folder to your Extended Matrix:

1. Expand the Auxiliary files section in the EM Data Tree panel

2. Select an existing auxiliary file or press Add to create a new one

3. In the Resources: field, click the folder icon and navigate to your media folder

4. Important: Before confirming the path, make sure to check the Relative Path option in the file browser settings

Warning

Always use relative paths when setting up resource folders!

• Correct: //Resources or //../../SharedFolder/Photos

• Wrong: C:\Users\YourName\Project\Resources (Windows-specific)

• Wrong: /Users/yourname/Project/Resources (macOS-specific)

Relative paths ensure that your project works correctly when:

• Opening the file on different computers

• Syncing through cloud services (OneDrive, Dropbox, Google Drive)

• Sharing the project with collaborators

• Moving the project to a different location

If a warning appears in red indicating that the path is absolute, reconfigure the path using the relative format.

The Thumbnail System

Once a resource folder is configured, EMtools can automatically generate thumbnail previews of all images in that folder. The thumbnail system creates a local cache that speeds up image browsing and reduces memory usage.

How it works:

• Thumbnails are stored in a folder named EM_thumbs/ next to your .blend file

• Each resource folder gets its own subfolder (e.g., Resources_abc12345)

• The system remembers which images have been processed to avoid duplicates

• If you sync your project via cloud storage, thumbnails are automatically shared across computers

Generating Thumbnails

1. In the Auxiliary Files panel, locate the text Thumbnails for the resource folder? Click below.

2. Click the (Re)generate thumbnails button

3. EMtools will scan all images in the resource folder (including subfolders) and create previews

4. Progress information is displayed in the Blender console

5. The line Thumbs: Resources_xxxxxxxx shows the name of the cache folder that was created

Note

Supported image formats: JPG, PNG, BMP, TIFF, PDF (first page)

The generation process only creates thumbnails for new or modified images. If you click (Re)generate again, existing thumbnails are skipped automatically.

Viewing Thumbnail Cache

You can verify that thumbnails were created correctly:

• The Thumbs: Resources_xxxxxxxx line displays the cache folder name

• Click the folder icon next to this line to open the cache folder in your file manager

• Inside you’ll find the index.json file and thumbnail images organized in subfolders

Using Thumbnails in EMtools

Once thumbnails are generated, they are automatically displayed when:

• Browsing DocumentNodes linked to images

• Using the US/USV Manager to view associated documentation

• Working with the Stratigraphy visualization tools

The thumbnail preview provides quick access to your images without having to open them in external applications.

Working Across Multiple Computers

If you use cloud storage to sync your project:

1. First computer: Set up resource folders using relative paths and generate thumbnails

2. Wait for cloud sync to complete (the EM_thumbs/ folder will be uploaded)

3. Second computer: Open the same .blend file

4. Thumbnails are automatically recognized and ready to use - no need to regenerate them!

Tip

If you work on multiple computers, make sure that:

• The .blend file and resource folders maintain the same relative structure

• Cloud sync is complete before opening the project

• You always use the // prefix for relative paths

Best Practices

• Keep resources organized: Use clear folder names like Photos, 3DScans, Documents

• Generate thumbnails periodically: After adding many new images, regenerate to update the cache

• Check the hash: The Thumbs: Resources_xxxxxxxx code should be identical on all your computers if using relative paths

• Backup regularly: Include both the .blend file and the EM_thumbs/ folder in your backups

Note

For more advanced usage and technical details about the thumbnail system, click the help icon (?) next to the Thumbs: line in the interface.

Troubleshooting

Thumbnails not appearing

• Verify that the resource folder path is correct and accessible

• Click (Re)generate thumbnails to rebuild the cache

• Check the Blender console for error messages

Warning about absolute path

• Reconfigure the resource folder using a relative path with // prefix

• Example: change C:\Project\Resources to //Resources

Different cache folder on another computer

• This happens when using absolute paths instead of relative paths

• Solution: reconfigure all resource folders using relative paths

• The hash (Thumbs: Resources_xxxxxxxx) should match across computers

Thumbnails folder not syncing

• Check that your cloud storage service is actively syncing the EM_thumbs/ folder

• Some cloud services may need manual folder selection for sync

Utils sub-panel

A third section, the Utils sub-panel (labelled Utilities & Settings in older builds), is included within the EM Data Tree panel. Here, users can: convert an EM made with an old version of the formalism, create the default collections, rename Proxies and enable Experimental Features.

In the first case (Convert 1.x->1.5 button) EMtools will normalise older GraphML files authored with the 1.x palette to the 1.5 visual conventions — see Convert 1.x->1.5 below for details.

Within this section, EMtools includes also a button, Create Standard Collections, that allows to automatically create the set of default collections (Proxy, RM, CAMS) related to a reconstruction process with Extended Matrix.

By pressing Manage Proxies' Prefixes button, EMtools will automatically rename Proxies according to the GraphML ID (NB: this step is mandatory to mutually connect GraphML and Proxies. User must select geometries before applying the tool).

By pressing the Enable Experimental Features button, a set of Experimental Features will be activated within the sections of the EM Data Tree panel (Fig. 6).

Convert 1.x->1.5

The Convert 1.x->1.5 button in the EM Data Tree → Utils sub-panel migrates older Extended Matrix GraphML files (authored with the 1.x generation of the yEd palette) to the 1.5 visual conventions used by the current importer and tooling.

The button drives the graphml.convert_borders operator, which reads a .graphml file and rewrites its node visual attributes in-place on a new file:

• Sets the border width to 4.0 for the EM target shapes (rectangle, hexagon, ellipse, octagon, parallelogram).

• Applies the 1.5 canonical border colours based on shape type and background:

  • rectangle → #9B3333 (US / negative units)

  • hexagon → #31792D (USV)

  • ellipse → #31792D (USV variants)

  • parallelogram → #248FE7 (documents / extractors)

  • octagon → #D8BD30 (Special Find) when on a light background, #B19F61 (Virtual Special Find) when on a black background.

The result is written next to the original file with a _converted.graphml suffix; the source file is left untouched.

1. Expand the Utils sub-panel in the EM Data Tree.

2. Click Convert 1.x->1.5 — Blender’s file browser opens.

3. Select the legacy .graphml file and confirm. EMtools produces <name>_converted.graphml in the same folder and reports the output path in the Blender status bar.

4. Load the converted file with Add GraphML as you would any other graph (see the Loading DosCo documents into the graph note above for the load workflow).

The operator is a visual / palette normaliser, not a full datamodel migration. In particular:

• It does not invent or upgrade paradata. If the older graph lacked family / is_series attributes on extractors and combiners, the converted file reflects the same absence.

• It does not rename relation edges whose semantics changed between releases (e.g. is_after direction canonicalisation introduced in v1.5.3). Such edges are preserved as-is and may surface as importer warnings.

• It does not re-link external documents. DosCo references in the original are preserved as identifiers; ensure the corresponding files are reachable in your DosCo folder.

• For projects authored before EM 1.0 (pre-formalisation, free-form yEd), conversion is not reliable: the original is best used as a visual reference for manually rebuilding the graph in the current palette.

For the missing pieces above, the importer side picks up some of the slack: s3dgraphy.importer.import_graphml is forgiving on older label conventions and re-maps known legacy types on the fly during import, and the post-import passes in s3dgraphy.transforms (hoist_propagative_metadata, prune_redundant_propagative_edges, compact_propagative_metadata) can be used to normalise paradata attribution programmatically.

A single, end-to-end s3dgraphy.utils.convert_legacy_em_graph wrapper that combines the palette-normalisation step performed by this button with the in-memory cleanup transforms is on the s3dgraphy roadmap. Until it lands, the procedure above (button → load → optional transforms → re-export) is the recommended path. The anchor for this section (convert-legacy-em-graph) is stable and will continue to point here when the wrapper is published.

Fig. 6 Experimental Features enabled (red rectangles with numbers 1-3)

Note

As highlighted in the red warning, this set of features is experimental and it should not be used within the regular documentation process of the EM (Fig. 6).

Here a brief presentation of the Experimental Features, the numebers on the list refer to the numbers specified in Fig. 7:

1. On the upper part of the panel the Switch to 3D GIS button allows to instantly switch from the EM mode to a 3D GIS mode where users can link an external database to EMtools (Fig. 7). The activation of the 3D GIS mode consents to connect an External database to the 3D environment of Blender via EMtools. User have three different type of external databases (Generic Excel, PyArchInit, EMdb Excel) to choose from. By pressing the corresponding button a diverse set of options will appear. After setting all the required information the external db will be import within EMtools.

2. Within the DosCo Folder section (Fig. 6) the More option menu appears, this new part of the add-on permits to populate Extractors, documents and Combiners using DosCo files.

3. In the Utilities & Settings (Fig. 6) the activation of the Enable Experimental Features button activates the Experimental Tools section with Rebuild Graph Indices, Benchmark Property Functions, and the collapsible Create a GraphML wizard.

   • Create a GraphML: A 3-step panel-based wizard for generating Extended Matrix GraphML files from Excel data — either filled manually using downloadable templates or produced by AI-assisted extraction from archaeological reports. The wizard works entirely in memory until export:

     • Step 1 — Convert Stratigraphy: Loads stratigraphy.xlsx (24-column template) and creates an s3dgraphy graph in memory.

     • Step 2 — Enrich with Paradata (optional): Loads em_paradata.xlsx and adds per-property provenance chains (PropertyNode → ExtractorNode → DocumentNode) to matching nodes.

     • Step 3 — Export GraphML: Saves the graph to a .graphml file. Import this file via File > Import EM file to populate the Blender scene.

     The panel also provides template download buttons and an AI Extraction Prompt section with a language selector and a one-click Copy AI Prompt to Clipboard button.

     Warning: This feature is experimental. Always verify the generated GraphML before using it in production.

Fig. 7 3D GIS mode

GraphML Warnings

When a GraphML file is imported, EMtools validates its structure and content. Any issues are surfaced in a collapsible GraphML Warning box below the graph entry.

Common warnings:

• Missing site ID — the swimlane header does not contain the mandatory ID: field (for example Great Temple [ID:GT16]). Fix this in yEd before reloading.

• Placeholder epoch dates — an epoch still contains the xx placeholder in its start/end fields. Replace with real dates.

• Structural issues — malformed nodes, dangling edges, unknown node types reported by the importer. Each message indicates the offending node ID.

The warnings are read-only: they describe what the importer detected. To clear them, fix the .graphml file in yEd and reload with the File Refresh button in the EM Data Tree list. The Data Funnel guide button links to the Extended Matrix manual section with authoring best-practices.

GraphML Merge Conflict Resolution

When an XLSX stratigraphy file is merged into an already-loaded graph (for example, to incrementally add units discovered in a later season), the EM Setup panel performs a diff and surfaces conflicts — nodes where the incoming data and the existing graph disagree.

The Conflict Resolution panel appears in the EM tab while a merge is active. For each conflict the user can choose:

• Keep Existing — ignore the incoming value for this node.

• Use Incoming — overwrite the existing value.

• Per-Field Choice — accept some fields from the existing graph and others from the incoming XLSX, useful when only a subset of attributes has changed.

Epoch compatibility is verified before merge: if the XLSX references epochs that do not exist in the graph (or have incompatible ranges), a blocking Epoch Report is shown at the top of the panel. Fix the XLSX (or adjust graph epochs) and retry.

Apply Merge commits the choices into the graph; Cancel Merge discards them.