DAS Data Modeling

CWP Seminar 2023

Distributed Acoustic Sensing



Background: DASDAE

  • Collection of DFOS libraries and applications
  • Focus on research and education
  • Enable DFOS interoperability and fusion

Background: DASCore

DASCore: Goals


  • Common building block for future DASDAE codes
  • Provide reference processing implementations
  • Implement basic, customizable visualizations
  • Read/write common data formats
  • Manage data sets

Data Structures

Data Structures

Data Structures

Data Structures

Data Structures

Data Structures

Data Structures

Data Structures

Motivation

  • Patch and Spool model data
    • Processing, data interfaces, partial metadata
  • How to model “full” metadata?
  • Simply mirror a standard?
  • Which standard?

Motivation

DAS RCN consensus:

“No standard data and metadata format exists”






- Jerry Carter (director of data services, Earthscope)

Motivation


https://xkcd.com/927/

Requirements


Earthscope

  • Common technology
  • Data can be managed separate from metadata
  • Applicable to many geophysical monitoring scenarios

Requirements


Derrick

  • Versioning and time snapshots
  • Track provenance
  • Single source of truth
  • Version migrations
  • Plain-textable

Existing Standards


Earthscope’s DAS-RCN metadata definition

  • Useful; many concepts outlined
  • Incomplete, possibly abandoned?
  • Definition not implementation

Existing Standards


StationXML

  • Specific to point sensors (e.g., seismometers)
  • Doesn’t account for DAS information
  • XML

Existing Standards


ProdML

  • Excellent coverage of DFOS concepts
  • Focused on oil/gas
  • Stores data/metadata together
  • Not clear how well requirements are met

Existing Standards


SEGY

  • Binary parsing nightmare (extended headers)
  • Stores data/metadata together
  • Not good

Existing Standards




Others?

Implementation: Approach

  • Use Pydantic
    • schema creation
    • serialization
    • validation
  • Try to capture RCN metadata concepts
  • Use ProdML terminology
  • Focus on flexibility: we are going to get this wrong
  • Build prototype, solicit feedback

Implementation: Base


class Base:
  resource_id: str
  creation_time: datetime
  author_id: str
  notes: str
  version: int = 1
  schema_version: int = 1

Implementation: Base


class Base:
  resource_id: str
  creation_time: datetime
  author_id: str
  notes: str
  version: int = 1
  schema_version: int = 1
  
  @migration(1, 2)
  def migration(self, ...):
      ...

Implementation: Acquisition

Acquisition extends attrs and coords of a Patch

class Acquisition(Base):
  instrument_configuration_id: str
  cable_segment_id: str
  data_units: str
  start_time: datetime
  end_time: datetime
  network: str
  station: str
  dims: str
  tag: str
  history_id: str

Implementation: Inventory


  • Contains all Base objects
  • Indices for object type
  • Updates occur on exactly one object
  • Stores history
  • Can be restored to specific time

Implementation: Inventory


class Inventory(Base):
  contents: dict = {resource_id: [object, ...]}
  classes: dict = {class_name: [resource_id, ...]}
  • Objects are immutable
  • Each resource_id can have multiple objects
  • Updated objects are appended to resource_id list
  • Last items in list are “current”
  • Can rewind time

Implementation: Inventory


class Inventory(Base):
  contents: dict = {resource_id: [object, ...]}
  classes: dict = {class_name: [resource_id, ...]}
  • Allows efficient lookup and iteration by type

Implementation: CableSegment


class CableSegment(Base):
  length: float
  length_units: str
  cable_id: str
  environment_id: str
  orientation_id: str
  contents: list = [
    segment_id, 
    splice_id, 
    termination_id, 
    connector_id,
  ]

Implementation: What’s Wrong?

Focus Groups


  • Down hole exploration
  • Streamers
  • Hydro/Cryosphere
  • Urban/Industrial

https://github.com/dasdae/dascore