Spool
In the proposed workflow, DASCore’s Spool acts as a bridge between stored patch (time-series) data and the DASDAE Inventory. The spool finds the relevant patches using indexed patch metadata; the selected patches then use Inventory for richer context querying, selectively transferring coordinates/metadata, etc.
Inventory Workflow
Spool operations
Everything this page describes is implemented.
An inventory can be attached to a spool, which then carries it as context for the operations that want one. Attaching is deliberately inert: it costs nothing per patch and adds nothing to what the spool’s patches carry. It does clear enrichment already set up, because swapping the source underneath a configured enrichment would silently rewrite every patch’s metadata; the new one has to be asked for. Three methods do the work. enrich copies inventory metadata onto each patch as it is extracted, conform_to_inventory resolves the index against the inventory so that the spool holds exactly what the inventory describes, and remove_inventory drops the inventory along with any enrichment set up from it. enrich and conform_to_inventory each take an inventory directly, defaulting to the attached one, so attaching is a convenience rather than a required first step.
import dascore as dc
inventory = dc.inventory("inventory.yaml")
# Attaching is cheap: the spool carries the inventory and nothing more.
spool = dc.spool("/data/archive").attach_inventory(inventory)
raw = (
spool
# Resolve the index against the inventory: drop the patches it does
# not describe, and subdivide those straddling an epoch boundary.
.conform_to_inventory()
# Get rid of channels with undefined coupling
.unselect(coupling=None)
# Coordinate selection from corresponding coordinates
.select(latitude=(45, 50), elevation=(100, 250))
# Each patch now arrives with its inventory metadata attached.
.enrich()
)
patch = raw[0]Keeping the three apart matters because they cost very different things. Attaching is free. Conforming is a one-time pass over metadata, expensive on a large archive but paid once. Enrichment is per-patch work paid on every extraction — resolving the context, then building a coordinate per projected track — so a spool that is only being selected on, or whose patches are consumed for their data alone, should not pay it. That is why enrichment is something asked for rather than something attaching implies.
Patches the inventory does not describe
A patch resolves to no inventory entry when it carries no acquisition_key, when it carries one the inventory does not contain, or when its time falls outside every matching epoch. Both conform_to_inventory and enrich have to say what happens then, and they answer differently because they are different verbs: conforming decides membership, enrichment decides metadata.
conform_to_inventory applies an on_unresolved policy: "raise" (the default) fails immediately and names the files, "warn" drops them but reports what was dropped, and "drop" discards them silently — the right choice only when the inventory intentionally covers a subset of a larger archive. A patch is judged over its whole span, so one reaching outside a matching epoch at either end is undescribed rather than partly described; keeping that edge simple is deliberate, as a mature workflow should rarely produce one.
enrich never drops a patch. A patch it cannot resolve comes out unchanged rather than missing, under an on_unresolved policy of "warn" (the default), "raise", or "ignore" for silence. Removing rows is conforming’s job, and keeping it there is what lets enrichment stay lazy: nothing is resolved when enrich is called, so len(spool) and get_contents() keep their meaning and the cost stays proportional to the patches actually extracted. An archive an inventory partly covers therefore works without conforming — spool.enrich(inventory) enriches what it can and says what it could not — and spool.conform_to_inventory().enrich() is how to have both. Both spool-level verbs spell this policy on_unresolved, differing only in the actions their verb allows: enrich takes "warn"/"raise"/"ignore", and conform_to_inventory takes "warn"/"raise"/"drop". The name differs from Patch.enrich’s on_missing because the scopes differ — on_missing governs a requested name the inventory does not define, on_unresolved a whole patch it does not describe.
A patch which straddles an epoch boundary is a different condition and is not covered by on_unresolved: the inventory describes it twice rather than not at all. Where the change is one of optical path, conform_to_inventory subdivides it; where it is one of acquisition, it raises, because no valid file spans a configuration change. Per-channel gaps such as uncovered geometry are different again and stay governed by the partial-coverage semantics — missing values, never errors.
Resolution and epoch boundaries are pure metadata, so conforming filters and subdivides the spool’s dataframe: len(spool) and get_contents() reflect exactly what iteration will yield, and only data materializes lazily on extraction. Subdivision is exact along the sample grid — each piece opens at the first sample at or after its boundary, so the pieces of a patch hold every sample it held and hold none of them twice, whatever instant the boundary falls on. A patch which must be subdivided but records no sampling interval raises instead of guessing, since a guess would silently lose the sample beside the boundary.
A spool combined with another (+) keeps an inventory attached to either operand, and likewise the enrichment set up from it; the two carry over independently, so attaching the same inventory to the other operand cannot turn a working union into an error. Two operands answering either question differently have no combined meaning and raise; attach to the combined spool instead. What carries over then applies to every patch in the result, including those the other operand contributed; under on_unresolved those the inventory does not describe come out unenriched with a warning rather than failing the extraction.
Selecting On Inventory Metadata
Each channel resolves to a single point on the optical path, and that point decides which intervals it belongs to. Channels have physical extent — gauge length is recorded on the acquisition — but membership is decided by the channel’s mapped center, so every query has one well-defined answer and boundary channels follow the same half-open rule as the tracks themselves. Selection places the channels with the same projection enrichment uses and then judges the values with the same predicate the index applies to a stated attribute, so a selector cannot mean one thing here and another in either: every channel a selection keeps is one enrichment would give the value asked for.
Typed tracks are selected by their own name, which matches the track’s identity field. The name is the one the optical path calls the track, so it is the same name a qualified field uses (optical_components, not optical_component) and the same one enrichment projects — one vocabulary rather than a singular for selection and a plural for everything else:
spool.select(coupling="trench") # coupling_type
spool.select(geometry="borehole-A") # geometry name
spool.unselect(optical_components="lead-in") # component nameA sequence selects the union, and None matches channels where the track is undefined. A coordinate carries one dtype and a string array has no null, so absence is stored as the empty string and None is the query spelling for it; numeric tracks use NaN and membership groups False:
spool.select(coupling=["trench", "conduit"])
spool.unselect(coupling=None)Other fields of a track use qualified names, which are exact field references and therefore arrive by dict-unpacking. Keyword arguments combine with AND, so both predicates below are evaluated against the one coupling record a channel resolves to:
spool.select(coupling="trench", **{"coupling.medium": "soil"})Annotation groups become coordinates named by the group, so they are selected like any other coordinate — no separate entry point:
spool.select(rock_type="granite") # categorical group
spool.select(rock_type=["granite", "shale"]) # union
spool.select(noisy=True) # membership group
spool.select(frost_depth=(1.0, 2.0)) # numeric groups take ranges, as coords doGroup names are therefore subject to collision rules: a group named after a structural coordinate or a typed track raises when the inventory is checked, and a group colliding with a coordinate a particular patch already carries raises at enrichment.
Inventory.get_names is what makes this sayable: it returns the names an inventory could contribute to a patch, split by where each lands — attrs for the acquisition and interrogator facts, coords for the coordinate labels its CRS defines, its annotation groups, and its typed-track fields. Spool.select uses it to tell an inventory field apart from a misspelled attr, so an unknown name gets one message and a real name selection cannot use yet gets another. The attrs side is read off the models, so a field added to the acquisition or interrogator is selectable without a second list to maintain; the coords side is computed from the inventory, since only it knows which tracks its paths describe. Contributing a name is not promising a value for it — a name is listed when some acquisition or path could define it, and one none does simply resolves to nothing.
Selection on the acquisition-level facts — gauge_length, interrogator.model, and the rest of the attrs side — is whole-patch metadata work and is implemented. Precedence there is per row: a patch which states the name is judged by the index exactly as it would be without an inventory, and only the rows leaving it unstated are resolved, once per epoch rather than once per patch. So an archive whose headers are complete never consults the inventory, and one whose headers are empty pays for its epochs rather than its files. A patch the inventory does not describe, or describes twice because it straddles a change of acquisition or optical path, is not selected: select is a filter, and a patch with no single answer is no more selected than one which lacks the attr entirely. A patch crossing an epoch bound its answers survive unchanged does have a single answer, and is judged on it.
Spool.unselect is the complement of select — everything the matching selection would have removed — and reaches whatever select reaches. The complement is taken against select itself rather than by negating each predicate, so one keyword cannot come to mean different things in the two. It refuses the patches’ own coordinates: a coordinate range decides how much of each patch to keep rather than which patches to keep, so its complement is a hole in the middle of each patch rather than a filter over them. Removing part of a patch’s span is a trim, and the way to ask for one is to select the ranges to keep, or to use Patch.unselect on each patch — a patch is the one place a range complement makes sense, because it can have samples taken out of its middle.
A name the index already uses keeps its own meaning, whether or not the inventory could also place it along the fiber. select(distance=...) is the patch’s own axis, so attaching an inventory never moves a name out of the namespace it has always been in; optical path distance is reached through enrichment (enrich(coords=("distance",))) instead.
Inventory-backed select and unselect on the coords side operate on channels, not whole patches. Contents are trimmed to the matching channels, and when the matching region is disjoint along the fiber — a path that passes in and out of the selected coordinate range, or an undefined-coupling zone in the middle of a path — the patch is subdivided so that each contiguous matching interval becomes its own patch. Selection can therefore change both the shape and the number of patches in the spool. A channel query complements exactly, unlike a patch’s: the channels run along one dimension, so what a selection did not keep is expressible as channels rather than as a shape no array can hold. Naming acquisition-level facts alongside channel-level ones still yields one complement — a patch the attributes never matched keeps every channel, since the selection it complements never held it.
Optical-path validity edges work the same way along time: a fiber can break mid-recording, so a patch straddling a path epoch boundary appears in the index as its pieces, each resolving cleanly to its own window, and the data is split when extracted. Spool.split_by (below) applies the same subdivision mechanism to any inventory-derived coordinate.
Which channels match is decided on each patch’s own sample grid, so a patch which records no channel spacing — one already cut into non-contiguous pieces, say — raises rather than being trimmed on a guess, exactly as a patch with no time step does when an epoch boundary would subdivide it. The dimension trimmed is the one the acquisition’s distance_map places channels by, and it must be a real dimension of the patch: what a non-dimensional coordinate says about the dimension it runs along is not recorded in the index, and a spool whose patches disagree about which dimension that is has no single one to trim.
Because a channel selection removes channels rather than re-describing them, it is not re-planned from its sources the way a chunk is: chunking such a spool again along the fiber keeps what the selection kept, rather than collapsing back onto the whole patches and quietly restoring the channels it dropped. samples and relative are refused alongside a fiber coordinate — both describe the patch’s own axis, while these say what is attached to each channel of it — and a bare ... selects everything here as everywhere. None is the one selector which means something different on this side: it matches the channels a track says nothing about, rather than selecting all of them.
Acquisition boundaries are stricter: a configuration change requires stopping and restarting the acquisition, so no valid file spans one. On a straddling file conform_to_inventory raises, naming the file and the boundary; the usual fix is a correction — set the epoch time to the file boundary, the instrument’s own record of the restart.
Subdivision reflects exactly what the path metadata says: a noisy geometry track can shatter what is conceptually one interval into several patches separated by small gaps. Apply Spool.chunk afterward to consolidate nearly-contiguous patches (tolerance parameter). chunk merges on indexed attrs, and enrichment happens after merging rather than before — a spool enriches each patch as it is extracted, so the merge never sees inventory metadata and cannot use it to keep fragments apart. Keeping fragments from different epochs separate is therefore the subdivision’s job, not enrichment’s: merging across a path epoch boundary is legal and will be re-subdivided at the next resolution.
Spool.enrich sets up the same operation Patch.enrich performs, applied to each patch as it is extracted: spool.enrich() uses the attached inventory, spool.enrich(inventory) attaches and uses the one it is given, and every Patch.enrich keyword passes straight through. Nothing is computed at the call itself; the argument names are checked and the arguments held, with their values checked by each patch’s own enrichment. Calling it again replaces the arguments rather than adding to them. Enrichment survives select, sort, and chunk — a derived spool is the same data — and remove_inventory or attaching a different inventory ends it.
Another useful function is Spool.split_by. It expands the spool into one patch per value of an inventory-derived coordinate — most often an annotation group — and can greatly expand the number of patches. Intervals of one group may overlap, but a channel resolves to a single value of it, the same one enrichment would project, so the outputs of one call divide the fiber rather than share it; two different groups may still cut it differently, which is what makes a nested split worth doing. It raises if no inventory is attached, and likewise for a name the inventory could not contribute — a misspelling has no values to split into, and an empty spool would not say so.
by_zone = spool.split_by("zone", include=("hole_1", "trench_?a"), exclude=("along_fence",))Every kind of group splits: a categorical one by each of its strings, a numeric one by each distinct measurement, and a membership group into the channels it includes and those it does not. Absence is not a value, so the channels a group says nothing about make no output of their own — the one exception being a membership group, where False is a statement about every channel rather than the absence of one. include and exclude are globs matched against each value written as a string, so one vocabulary covers all three kinds, and exclude wins where both match, which lets a family be named and one member carved out of it in either order.
Each output patch is stamped with the value it was split on as an attr named after the group (zone="hole_1"), so overlapping siblings stay distinguishable and later operations can select on it directly — both the patch and the row get_contents shows for it carry the value. Pass stamp=False to leave existing attrs untouched, which is what nested splits want: splitting by zone and then by quality should not overwrite the first result.
Patch operations
# Enrich with all applicable attrs and coords (the default)
patch = patch.enrich(inventory)
# Or select specific attrs and coords
patch = patch.enrich(
inventory,
attrs=("sample_rate", "gauge_length", "interrogator.model"),
coords=("x", "y", "z"),
on_missing="null",
)
# Enrich with attrs only
patch = patch.enrich(inventory, coords=False)Patch.enrich is the single entry point for copying inventory metadata onto a patch — it resolves the full chain internally (the acquisition’s distance_map to optical distance, then the path tracks), and optical distance itself is requestable like any coord: enrich(coords=("distance",)). The attrs and coords keywords each accept True (all applicable), a tuple of names, or False to skip. It is the same operation Spool.enrich applies to each patch extracted from the spool. The patch never retains a reference to the inventory; any operation that needs inventory context takes it as an explicit argument.
Blanket attrs=True copies observing-system facts only: the fields the inventory is authoritative for — gauge_length, pulse_width, interrogator.serial_number, and the like — carried under their exact inventory spellings, with dotted names for nested facts. Excluded from the blanket form are data_type, data_category, and data_units, which describe the data as it now stands rather than the system that recorded it, and sample_rate and spatial_interval, which the patch’s own coordinates already state — nothing should be redundant between coordinates and attrs, and decimating changes both, so restoring the as-acquired values would contradict the coordinates they duplicate. Naming any of them explicitly (attrs=("data_type",)) still copies it, and means exactly what it says: restore the as-acquired value. The test that sorts the two cases is whether a patch function can legitimately change the field without the inventory being wrong. If it cannot, it is a system fact and belongs to blanket enrichment; if it can, the patch is the authority and enrichment waits to be asked.
Disagreements are governed by conflicts, which takes the same values and behaves the same way as the flag Spool.chunk uses for conflicting attrs. Enrichment conceptually combines [inventory_values, existing_values], so "keep_first" (the default) lets the inventory win, which makes re-enriching a refresh rather than an error; "raise" fails on genuine disagreement — both sides present and unequal — naming the attr and both values; and "drop" discards the disagreeing attrs instead of choosing between them. Filling an empty attr is never a conflict, and neither are equal values. "raise" is the misresolution guard: a file header whose interrogator.serial_number disagrees with the resolved acquisition’s usually means the acquisition_key resolved to the wrong place. It applies uniformly to every field rather than through per-field special cases, and is worth opting into when first attaching an inventory to an unfamiliar archive.
enrich’s on_missing governs only explicitly requested names the inventory does not define — "raise" (default), "null" (the dtype-appropriate missing marker: NaN for floats, None for strings), or "ignore" (omit silently). The blanket attrs=True/coords=True forms copy what is applicable and never trigger on_missing. Whole-patch resolution failures belong to the spool-level policy above.
Three policy vocabularies coexist here on purpose. on_unresolved and DASCore’s WARN_LEVELS are a volume axis: their members take the same action and differ only in whether they announce it. on_missing is a result axis: "raise" fails, "null" fills a marker, and "ignore" leaves the name off. The two axes share raise and ignore, which mean the same thing in both — fail, and proceed quietly — so the vocabulary is one word set rather than three; "null" is the member with no counterpart, because filling a marker is an action rather than a volume. conflicts keeps chunking’s "keep_first"/"raise"/"drop" because matching that flag is worth more than internal tidiness.
The patch resolves inventory context from its acquisition_key, or from an explicit acquisition_key passed to enrich. Because acquisition metadata is scalar per patch, enrich requires the patch to lie within a single acquisition and optical-path validity window and raises if it straddles a boundary — select or split the patch first.
Patches whose time axis is no longer physical (e.g., lag-time correlations) can still resolve by passing an explicit time argument alongside the codes, selecting the acquisition epoch and optical-path context as of that moment. time is accepted only when the patch has no physical time coordinate; for a patch with a real time axis, enrich raises — select or split instead.
patch = patch.enrich(
inventory,
acquisition_key="DAS.L001.01.RAW",
time="2024-05-01T00:00:00",
coords=("x", "y", "z"),
)Kinds of Patch Attribute
Enrichment is easier to reason about once patch attributes are sorted into kinds. Every attribute is exactly one of the following four, and the kind decides who writes it.
- The identity key.
acquisition_keyalone, holdingnetwork.fiber_array.location.acquisition. It flows patch → inventory as the resolution query, is set by whoever authored the patch (a reader, a hive path, the user), and enrichment never touches it. - Observing-system facts. Everything the inventory is authoritative for, flowing inventory → patch under the exact inventory spellings. Blanket enrichment fills these, governed by
conflicts. - Data state and patch-native bookkeeping.
data_type,data_category, anddata_unitsdescribe the data as it now stands;tagandhistoryare the patch’s own record of itself. Readers and processing functions maintain both. Enrichment touches data state only when explicitly asked, and never touchestagorhistory. - Storage provenance.
source_path,source_format, andsource_versionrecord where the bytes live, as opposed to where the signal came from. They belong to the spool and are read fromget_contents()rather than from patch attrs — a convention rather than an enforced rule. The separation is deliberate: a patch merged from three files has oneacquisition_keyand no single source path.
Working with Archives
Native files carry no network, fiber-array, location, or acquisition codes, and rewriting every file to embed them is costly and format-dependent. Instead, encode the metadata in the archive layout itself using hive-style directory names. DASCore’s file scanning parses any path segment of the form key=value as a string patch attribute applied to every file below it. Since paths are already stored in the spool index, this adds no scanning cost, and partition-aware tools such as PyArrow, DuckDB, and polars can query the same layout directly without DASCore in the loop (assuming they know how to read the files in question).
An archive laid out this way might look like:
archive/
├── inventory.yaml
├── acquisition_key=DAS.R2D1.01.HHZ/
│ ├── 2024-05-01_000000.h5
│ ├── 2024-05-01_010000.h5
│ └── 2024-05-01_020000.h5
├── acquisition_key=DAS.R2D2.01.HHZ/
│ ├── 2024-05-01_000000.h5
│ ├── 2024-05-01_010000.h5
│ └── 2024-05-01_020000.h5
└── field_notes/
├── deployment_log.md
└── tap_tests.csvGeneric key=value parsing sets acquisition_key directly — no special machinery — making it simple to “bolt on” a DASDAE inventory to virtually any DFOS archive that can be reorganized (or symlinked) into this layout. It is also branch-agnostic: the second token may name a fiber array or a station, and the resolver distinguishes them through the disjoint code namespaces. Component-level queries are wildcard matches against the composite: spool.select(acquisition_key="DAS.*").
The composite is the only supported spelling. A layout splitting the identity across segments (network=DAS/fiber_array=R2D1/...) would produce four string attrs which mean nothing on their own: network and station are no longer patch attributes, and nothing reassembles the parts. Write the whole key in one segment.
A few rules keep the scheme predictable:
- All four
acquisition_keyparts must be present for inventory resolution. Partial keys are legal at the scanning layer and remain readable as plain string attrs, but missing parts are never guessed or defaulted; patches without a completeacquisition_keyhave no valid inventory entry. - Path-derived attributes are always strings. Anything richer belongs in the inventory.
- A path segment wins over an attribute of the same name stored in the file. Renaming a directory is how metadata is attached or corrected without rewriting data, so the path is treated as the more recent statement. Filling an attribute the file leaves empty is the ordinary case and passes silently. Overriding a value the file actually states is not: it usually means a file was moved into the wrong directory, so scanning warns and names the attributes overridden along with an example of each.
- Merged or derived products have no single ancestor, so there is nothing true to assert with provenance keys; store them under plain directories (path segments without
key=valueassert nothing), where their file-embedded attributes — including a clearedacquisition_key— stand as written. - Keys and values should stick to code-safe characters (letters, digits,
-);=,/, and.are not allowed in values.acquisition_keyis the one exception to the dot ban: its value must be exactly four dot-separated code-safe tokens, of which only the location may be empty — a blank location is carried as an empty token (acquisition_key=DAS.L001..RAW). - Values should be short, station-like codes. This keeps
acquisition_keyreadable and total path lengths well clear of the Windows 260-character default limit (deep archives on Windows may still need long paths enabled).