Unit 5: More Use Case Examples

The previous unit demonstrated how to retrieve (reference metadata) configuration details for a dataflow and iterate over these attributes to drive process steps.

These metadata can be utilized to customize the behavior of diverse statistical processes, including data collection, validation, and mapping.

In this unit, we'll take a closer look at this configuration process, as well as the ways in which pysdmx can assist in creating the physical data model for a dataflow, facilitating data validation, data mapping and generating the filesystem structure, along with the required metadata in all cases. We'll conclude by going over the process of using vtlengine (VILT) for validation.

Configure your processes

In a scenario where we receive a data submission for validation, mapping, and integration, each step can be configured differently.

Select each option for more information.

Configuration options

Configuration options

Configuration options may depend on the ingested data or business unit practices. For instance, consider validation:

• The data received might include only what has changed compared to the previous submission. Alternatively, it could be a complete dataset, requiring different validation approaches.
• In case of validation problems, businesses may choose to quarantine only the invalid data, proceeding to the next step with the subset of valid data. Others may prefer quarantining the entire submission.
  
  

Configuration steps

Configuration steps

These configuration options can be captured using SDMX reference metadata.

To do this:

1. Create a Metadata Structure Definition with the configuration options using concepts and coded concepts.
2. Define the type(s) of attachment targets (e.g., a dataflow, a provision agreement).
3. Define a metadataflow (e.g., with the ID DCO for Dataflow Configuration Options) for which metadata reports will be provided.
4. Provide metadata reports (metadatasets) attached to the desired targets, defining their different configuration options.

Dataflow example

Dataflow example

For example, for the BIS_MACRO dataflow maintained by BIS, options could include:

• partial_update set to the boolean value true (indicating acceptance of only new or updated data).
• on_validation_error set to code F (Fail), signifying that the entire submission must be quarantined in case of validation issues.
• structure_map set to the URN of the structure map to be used for mapping data from the BIS_MACRO dataflow structure to its target structure.
• on_mapping_error set to code I (Ignore), as only a subset of data is mapped.

Create physical data model

Pysdmx can assist in creating the physical data model for a dataflow in a metadata-driven fashion, relying solely on the metadata stored in an SDMX Registry. For this scenario, Data Structure metadata are needed in the SDMX Registry:

The basic steps to follow to create the physical data model are:

1. Connect to the registry
2. Retrieve the schema information
3. Create the physical data model
4. Map the SDMX data types
5. Fine tune the physical model

A description of each of these steps, along with python code can be found on the pysdmx site.

Validate your data

Pysdmx can be used to facilitate data validation in a metadata-driven approach, relying solely on the metadata stored in an SDMX Registry.

There are various types of validation, and we'll focus on structural validation in this scenario. Structural validation ensures that the structure of data meets the expectations.

For this scenario, the necessary metadata depends on the desired thoroughness of validation. At a minimum, we need the data structure information. However, for more comprehensive validation, we may consider additional constraints from the dataflow or provision agreement.

We'll examine more about the required metadata on the next screen.

Required metadata

Let's look at each of the necessary metadata for the scenario presented on the previous screen.

Select each option to learn more.

Critical artifacts and steps

In summary, the following SDMX artifacts need to be in the SDMX Registry: AgencyScheme, Codelist, ConceptScheme, and Data Structure. For more thorough validation, additional metadata like Data Constraint, Dataflow, DataProviderScheme, DataStructure, and ProvisionAgreement is needed.

The basic steps to follow are:

1. Connect to the registry
2. Retrieve the schema information
3. Validating data
4. Validating the components
5. Validating the data type
6. Validating with facets
7. Validating coded components
8. Validating mandatory components

A description of each of these steps, along with python code can be found on the pysdmx site.

Map your data

Pysdmx facilitates mapping data in a metadata-driven fashion, relying solely on the metadata stored in an SDMX Registry.

Select each question below to learn more.

Required metadata

For our example, the objective is to store data in folders organized by dataflows. In each dataflow folder, we want to have sub-folders by data providers. Access to folders should be granted via appropriate roles with access requests approved by the manager of the organizational unit owning the dataflow.

Pysdmx can aid in generating the filesystem structure in a metadata-driven fashion, relying solely on metadata stored in an SDMX Registry.

Select each option to learn more about the required metadata.

Using VTL for Validation

Pysdmx supports reading data and metadata to generate and operate on datapoints using vtlengine. Numerous types of operations can be performed and the metadata requirements change depending upon the operations. Nevertheless, the steps remain the same:

1. Read the data
2. Extract the data and the data structure
3. Prepare the dictionary
4. Define the expressions and execution

More information on pysdmx and vtlengine integration is available here.

What do you know?

Let's complete one final question before concluding. Which of the following metadata indicate which providers supply data for a dataflow?

Select your answer and then select Submit.