-
Notifications
You must be signed in to change notification settings - Fork 2
Data Model Reference
- General Classes
- Mission Configuration
- Product Classes, Selection Rules and Products
- Processor Classes, Versions and Configurations
- Order Entry and Job Generation
- User Management
Last updated: 2025-01-28 (document version 1.7, valid for prosEO V1.1.0)
The prosEO Metadata Database holds all knowledge about products, processors, processor configurations, orders and jobs in the processing system. An overview diagram is given in Figure 2‑1 below.
Figure 2‑1: Product, processor, order and job metadata database
The table below describes the purpose of the entities contained in the Metadata Database:
| Entity (Class) | Description |
|---|---|
| Configuration | A specific processor configuration, tied to a specific ConfiguredProcessor object. It mainly consists of a set of configuration files and template information for creating Job Order files for the associated processor. |
| ConfiguredProcessor | A specific version of a Processor combined with a specific Configuration object (i. e. a specific set of configuration data for the given processor version). A ConfiguredProcessor is what must be specified for the generation of the products of a prosEO order. |
| Job | A collection of job steps required to fulfil an order for a specific period of time (e. g. one orbit). Note: As a future extension a selection by geographical area in addition to or instead of a period of time is envisioned. |
| JobStep | A single processor execution to produce a defined output product based on a defined set of required input product (modelled as ProductQuery objects). A JobStep can be executed as soon as all its ProductQuerys are satisfied. |
| Orbit | The description of an orbit identified by its start and stop times (e. g. the "spacecraft midnight" events of the Sentinel-5P satellite). There must be no time gap between subsequent orbits of the same spacecraft. Orbit times (and all derived times in prosEO) are given in UTC-STS (leap seconds spread evenly over the last 1000 seconds of the day) and to a microsecond precision. The public static variable orbitTimeFormatter gives a standard format for parsing and formatting orbit times. |
| OrderTimer | A timer for the generation of new processing orders from OrderTemplates, e. g. for the creation of background products. |
| PersistentObject | The root class of all persistent objects. It provides an ID, a version and a set of mission-specific parameters for the object. |
| ProcessingOrder | A customer order to process a specific set of ProductClasses for a specific period of time using a specific set of ConfiguredProcessors. An order may have properties like a product quality indicator (test vs operational), specific product delivery endpoints, specific (potentially mission-dependent) product generation attributes (e. g. a Copernicus collection number) etc. |
| Processor | A specific version of a ProcessorClass. Each permissible combination of a specific Processor with a specific Configuration is modelled as a ConfiguredProcessor. A Processor can consist of multiple tasks (note: for Sentinel-5P only one task per Processor is expected). |
| ProcessorClass | A type of processor capable of generating products of a specific set of ProductClasses. There can be only one ProcessorClass capable of generating products of any one ProductClass. |
| Product |
An instantiation of a ProductClass for a specific period of time (e. g. one orbit) and of a specific version. It is allowed to generate multiple products for the same ProductClass and time period, since equality of products is defined based on their attribute values. Products are considered equal if either of the following alternatives holds:
Note that the generation time is not considered relevant for product equality, because two processings with the same set of attributes as listed above are expected to produce the same output. |
| ProductClass | A class of products pertaining to a specific Mission, e. g. the L2_O3 products of Sentinel-5P. A ProductClass can describe final (deliverable) products as well as intermediate products. For a ProductClass its dependency on base products can be described using SelectionRules. Alternatively a ProductClass may be composed of other product classes (e. g. the S5P NPP products consist of three separate single-band NPP sub-products). |
| ProductQuery | A ProductQuery models the need of a JobStep to use a Product of a certain ProductClass for a specific time period. The time period can be defined by conditions, and a Product is said to satisfy a ProductQuery, if it matches the given conditions. A JobStep can be executed as soon as all its ProductQuerys are satisfied. |
| SimplePolicy | A product retrieval policy consisting of a retrieval mode and a time interval with delta times as defined in Annex B of ESA's Generic IPF Interface Specifications. Note: As a future extension selection policies based on spatial coverage are envisioned. |
| SimpleSelectionRule | A rule defining the required input ProductTypes for a certain output ProductType using an ordered list of SelectionPolicys. When selecting applicable Products for a JobStep the matching SelectionPolicy defines whether a Product satisfies a ProductQuery. Simple Selection Rules may be restricted to be applicable for only a well-defined set of Configured Processors. |
| Task | A single, separately adressable execution step of a Processor. Tasks may be flagged as "external", in which case the input data products are forwarded to an external entity for processing, and the output products are retrieved from that external entity. (Level 4 "Task" from Generic IPF Interface Specifications, sec. 4.1.3) |
| Workflow | A Workflow is a template for a processing order. For On-demand processing it allows to select a suitable ConfiguredProcessor based on a given input product type (and possibly also an output product type). For Systematic Production it supports automatic generation of processing orders based on incoming lower-level products. Within a given mission, the combination of workflow name and version must be unique. This class must not be confused with the generic concept of a "workflow" in the sense of chained processing steps, since such a concept does not exist in prosEO. The class name is derived from ESA's On-Demand Production Interface Delivery Point (ODPRIP) API. |
Table 2‑1: Metadata database classes
The processing system is capable to serve multiple missions. Therefore, a mission metadata model has been designed for the configuration of individual missions operating one or more spacecrafts each:
Figure 2‑2: Data model extension for multi-mission configuration
The table below details the classes for the multi-mission configuration extension:
| Entity (Class) | Description |
|---|---|
| ArchiveType | Interface protocol to use for a Product Archive |
| Mission | An Earth Observation mission (currently only Sentinel-5P). |
| Orbit | The description of an orbit identified by its start and stop times (e. g. the "spacecraft midnight" events of the Sentinel-5P satellite). There must be no time gap between subsequent orbits of the same spacecraft. Orbit times (and all derived times in prosEO) are given in UTC-STS (leap seconds spread evenly over the last 1000 seconds of the day) and to a microsecond precision. The public static variable orbitTimeFormatter gives a standard format for parsing and formatting orbit times. |
| Payload | The payload of a spacecraft, e. g. the TROPOMI instrument of Sentinel-5P. A spacecraft may have multiple payloads. |
| ProcessingOrder | A customer order to process a specific set of ProductClasses for a specific period of time using a specific set of ConfiguredProcessors. An order may have properties like a product quality indicator (test vs operational), specific product delivery endpoints, specific (potentially mission-dependent) product generation attributes (e. g. a Copernicus collection number) etc. |
| ProcessorClass | A type of processor capable of generating products of a specific set of ProductClasses. There can be only one ProcessorClass capable of generating products of any one ProductClass. |
| ProductArchive | Available archive locations for input products or auxiliary files. Product archives are by design mission-neutral, but the credentials may actually restrict an archive to a specific mission. In such a case, if the same endpoint shall be used for multiple missions, the product archives must be differentiated in their short codes. |
| ProductClass | A class of products pertaining to a specific Mission, e. g. the L2_O3 products of Sentinel-5P. A ProductClass can describe final (deliverable) products as well as intermediate products. For a ProductClass its dependency on base products can be described using SelectionRules. Alternatively a ProductClass may be composed of other product classes (e. g. the S5P NPP products consist of three separate single-band NPP sub-products). |
| Spacecraft | A spacecraft used for a specific Mission. A Mission may operate more than one spacecraft. |
Table 2‑2: Multi-mission configuration classes
Several aspects of the data model are shown in greater detail in the following diagrams.
Figure 2‑3: Data model for product classes and products
Figure 2‑4: Data model for processor classes, processors and configurations
Figure 2‑5: Data model for order entry and job generation
Figure 2‑6: Data model for user management
The tables below describe each of the classes for the prosEO Metadata Database (note that attribute multiplicities are not shown due to the way this document is generated):
| Entity (Class) | Description | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PersistentObject |
The root class of all persistent objects. It provides an ID, a version and a set of mission-specific parameters for the object.
|
|||||||||||||||
| Parameter |
This class allows to add mission-specific parameters to any persistent object. A parameter consists of a type (of the enum ParameterType) and a value with a class that corresponds to the type.
|
|||||||||||||||
| ParameterType |
The Java data type of the parameter
|
| Entity (Class) | Description | |||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mission |
An Earth Observation mission (currently only Sentinel-5P).
|
|||||||||||||||||||||||||||||||||||||||
| Spacecraft |
A spacecraft used for a specific Mission. A Mission may operate more than one spacecraft.
|
|||||||||||||||||||||||||||||||||||||||
| Payload |
The payload of a spacecraft, e. g. the TROPOMI instrument of Sentinel-5P. A spacecraft may have multiple payloads.
|
|||||||||||||||||||||||||||||||||||||||
| Orbit |
The description of an orbit identified by its start and stop times (e. g. the "spacecraft midnight" events of the Sentinel-5P satellite). There must be no time gap between subsequent orbits of the same spacecraft.
|
|||||||||||||||||||||||||||||||||||||||
| ProductArchive |
Available archive locations for input products or auxiliary files. Product archives are by design mission-neutral, but the credentials may actually restrict an archive to a specific mission. In such a case, if the same endpoint shall be used for multiple missions, the product archives must be differentiated in their short codes.
|
|||||||||||||||||||||||||||||||||||||||
| ArchiveType |
Interface protocol to use for a Product Archive
|
Table 3‑1: Mission configuration class detail
| Entity (Class) | Description | |||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ProductClass |
A class of products pertaining to a specific Mission, e. g. the L2_O3 products of Sentinel-5P. A ProductClass can describe final (deliverable) products as well as intermediate products. For a ProductClass its dependency on base products can be described using SelectionRules. Alternatively a ProductClass may be composed of other product classes (e. g. the S5P NPP products consist of three separate single-band NPP sub-products).
|
|||||||||||||||||||||||||||||||||||||||||||||
| ProductVisibility |
Visibility of products to external users (internally all products are visible at all times)
|
|||||||||||||||||||||||||||||||||||||||||||||
| ProcessingLevel |
Processing level, usually indicating the number of processing steps required to create a product of this class from unprocessed ("level 0") data. The enumeration values have been derived from a number of sources (listed below).
|
|||||||||||||||||||||||||||||||||||||||||||||
| SimpleSelectionRule |
A rule defining the required input ProductTypes for a certain output ProductType using an ordered list of SelectionPolicys. When selecting applicable Products for a JobStep the matching SelectionPolicy defines whether a Product satisfies a ProductQuery. Simple Selection Rules may be restricted to be applicable for only a well-defined set of Configured Processors.
|
|||||||||||||||||||||||||||||||||||||||||||||
| SimplePolicy |
A product retrieval policy consisting of a retrieval mode and a time interval with delta times as defined in Annex B of ESA's Generic IPF Interface Specifications.
|
|||||||||||||||||||||||||||||||||||||||||||||
| DeltaTime |
Class representing a single overlapping time period
|
|||||||||||||||||||||||||||||||||||||||||||||
| PolicyType |
Available policy types as defined in ESA's Generic IPF Interface Specifications.
|
|||||||||||||||||||||||||||||||||||||||||||||
| Product |
An instantiation of a ProductClass for a specific period of time (e. g. one orbit) and of a specific version. It is allowed to generate multiple products for the same ProductClass and time period, since equality of products is defined based on their attribute values.
Note that the generation time is not considered relevant for product equality, because two processings with the same set of attributes as listed above are expected to produce the same output.
|
|||||||||||||||||||||||||||||||||||||||||||||
| ProductQuality |
Quality annotation for products (determined by processor configuration)
|
|||||||||||||||||||||||||||||||||||||||||||||
| ProductionType |
Production context for the generation of a product (from the ESA PRIP API specification, ESA-EOPG-EOPGC-IF-3, issue 1.4, sec. 3.2)
|
|||||||||||||||||||||||||||||||||||||||||||||
| DownloadHistory |
History of product downloads; while originially bound to a ProductFile, the entry persists even after deletion of the ProductFile to give a full history of the Product downloads
|
|||||||||||||||||||||||||||||||||||||||||||||
| ProductFile |
The data file and pertinent auxiliary files for a product at a given processing facility. Each product has at most one data file representation at each of the processing facilities.
|
|||||||||||||||||||||||||||||||||||||||||||||
| StorageType |
The available storage types
|
Table 3‑2: Product class, selection rule and product class detail
| Entity (Class) | Description | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ProcessorClass |
A type of processor capable of generating products of a specific set of ProductClasses. There can be only one ProcessorClass capable of generating products of any one ProductClass.
|
||||||||||||||||||||||||||||||||||||
| Processor |
A specific version of a ProcessorClass. Each permissible combination of a specific Processor with a specific Configuration is modelled as a ConfiguredProcessor. A Processor can consist of multiple tasks (note: for Sentinel-5P only one task per Processor is expected).
|
||||||||||||||||||||||||||||||||||||
| Task |
A single, separately adressable execution step of a Processor. Tasks may be flagged as "external", in which case the input data products are forwarded to an external entity for processing, and the output products are retrieved from that external entity. (Level 4 "Task" from Generic IPF Interface Specifications, sec. 4.1.3)
|
||||||||||||||||||||||||||||||||||||
| Configuration |
A specific processor configuration, tied to a specific ConfiguredProcessor object. It mainly consists of a set of configuration files and template information for creating Job Order files for the associated processor.
|
||||||||||||||||||||||||||||||||||||
| ConfigurationFile |
A processor configuration file; configuration files must be provided statically to the processor container and will not be stored in the Storage Manager
|
||||||||||||||||||||||||||||||||||||
| InputFile |
Input files for inclusion in generated Job Order Files; these files will be stored in the Storage Manager and provided from there, if they have the file name type "Physical".
|
||||||||||||||||||||||||||||||||||||
| ConfiguredProcessor |
A specific version of a Processor combined with a specific Configuration object (i. e. a specific set of configuration data for the given processor version). A ConfiguredProcessor is what must be specified for the generation of the products of a prosEO order.
|
||||||||||||||||||||||||||||||||||||
| Workflow |
A Workflow is a template for a processing order. For On-demand processing it allows to select a suitable ConfiguredProcessor based on a given input product type (and possibly also an output product type). For Systematic Production it supports automatic generation of processing orders based on incoming lower-level products. Within a given mission, the combination of workflow name and version must be unique.
|
||||||||||||||||||||||||||||||||||||
| WorkflowOption |
Description of possible options, which can be set as "Dynamic Processing Parameters" in Job Orders created using the embedding Workflow
|
||||||||||||||||||||||||||||||||||||
| WorkflowOptionType |
Allowed types for workflow options
|
Table 3‑3: Processor class, version and configuration class detail (for InputFIlter and ClassOutputParameter see sec. 3.5)
| Entity (Class) | Description | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ProcessingOrder |
A customer order to process a specific set of ProductClasses for a specific period of time using a specific set of ConfiguredProcessors. An order may have properties like a product quality indicator (test vs operational), specific product delivery endpoints, specific (potentially mission-dependent) product generation attributes (e. g. a Copernicus collection number) etc.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| InputFilter |
Filter conditions to apply to an input product of a specific product class in addition to filter conditions contained in the applicable selection rule
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ClassOutputParameter |
Set of parameters to apply to a generated product of the referenced product class replacing the general output parameters
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| InputProductReference |
Identification of the input product to use for On-Demand Production; its intention is to be unambiguous (as per ODPRIP ICD), however this is not guaranteed given the limited number of specification options (uniqueness may be guaranteed when specifying a file name, but it is not, when only specifying a sensing time range). The input product will be retrieved from some LTA, if it is not readily available in prosEO.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| NotificationEndpoint |
The endpoint to send order completion notifications to
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| OrderState |
Possible states for a processing order; recommended state transitions:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![]() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| OrderSlicingType |
Possible methods for partitioning the order time period into individual job time periods for product generation
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Job |
A collection of job steps required to fulfil an order for a specific period of time (e. g. one orbit).
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| JobState |
Enumeration describing possible job states.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![]() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| JobStep |
A single processor execution to produce a defined output product based on a defined set of required input product (modelled as ProductQuery objects). A JobStep can be executed as soon as all its ProductQuerys are satisfied.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| JobStepState |
The possible processing states for a job step
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![]() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ProductQuery |
A ProductQuery models the need of a JobStep to use a Product of a certain ProductClass for a specific time period. The time period can be defined by conditions, and a Product is said to satisfy a ProductQuery, if it matches the given conditions. A JobStep can be executed as soon as all its ProductQuerys are satisfied.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ProcessingFacility |
A processing facility for running prosEO jobs and storing prosEO product files, e. g. at a cloud service provider.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| FacilityState |
For the user as well as for planning and product access purposes it is important to know, whether a processing facility is operational or not. To this end the enumeration FacilityState provides the following values:
Allowed state transitions are DISABLED <--> STOPPED --> STARTING --> RUNNING --> STOPPING --> STOPPED.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![]() |
Table 3‑4: Order entry and job generation class detail
| Entity (Class) | Description | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| User |
A prosEO user (actually the user's credentials).
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Authority |
An authority (also called a "privilege") is an atomic entitlement for a user to access a prosEO method or domain object.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Group |
A group of users
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| GroupAuthority |
A group authority (also called a "privilege") is an atomic entitlement for a group of users to access a prosEO method or domain object.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| UserRole |
Low-level user roles, which can be bundled for user groups.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Quota |
Monthly data volume granted to and actually used by a user
|
Table 3‑5: User management class detail



