Preface to the data management handbook

This data management handbook comes from an initiave started by MET and that has been funded by the S-ENDA project

1. Introduction

The purpose of the Data Management Handbook (DMH) is threefold:

1. to provide an overview of the principles for data management to be employed;

2. to help personnel identify their roles and responsibilities for good data management; and

3. to provide personnel with practical guidelines for carrying out good data management.

Data management is the term used to describe the handling of data in a systematic and cost-effective manner. The data management regime should be continuously evolving, to reflect the evolving nature of data collection. Therefore this DMH is a living document that will be revised and updated from time to time in order to maintain its relevance.

The DMH is a strategic governing document and should be used as part of the quality framework the organisation is using.

The primary focus of this DMH is on the management of dynamic geodata. Dynamic geodata is weather, environment and climate-related data that changes in space and time and is thus descriptive of processes in nature. Examples are weather observations, weather forecasts, pollution (environmental toxins) in water, air and sea, information on the drift of cod eggs and salmon lice, water flow in rivers, driving conditions on the roads and the distribution of sea ice. Dynamic geodata provides important constraints for many decision-making processes and activities in society.

This introduction (Introduction) lays forth the background and principles for the data management regime. Chapters 2-5 describe the implementation of the main building blocks: structuring and documenting data (???), data services (???), portals and documentation aimed at users (???), and governance issues (???). Each chapter starts with a brief statement of its purpose, followed by a description of what is implemented at present, as well as the planned developments for the short-term (<2 years) and expected developments for the longer term (2-5 years).

Practical guidelines for carrying out good data management are addressed in Chapter 6.

The intended audience for this DMH is any personell involved in the process of making data available for the end user. This process can be viewed as a value chain that moves from the producer of the data to the data consumer (i.e., the end user). The process is further described in The data value chain (???).

The handbook can be used in three ways:

1. Read Introduction to learn about the background and principles of data management;

2. Read Chapters 2-5 to learn about how data management is currently implemented and how it is expected to evolve in the next few years; and

3. Read ??? to find how-to’s and practical guidelines.

1.1 The principles of data management for dynamic geodata

Principles of standardised data documentation, publication, sharing and preservation have been formalised in the FAIR Guiding Principles for scientific data management and stewardship [RD3] through a process facilitated by FORCE11. FAIR stands for findability, accessibility, interoperability and reusability.

By following the FAIR principles it is easier to obtain a common approach to data management, or a unified data management model. One of the main motivations for implementing a unified data management is to better serve the users of the data. Primarily, this can be approached by making user needs and requirements the guide for determining what data we provide and how. For example, it will be described below how the specification of datasets should be determined. By implementing the data management practices described here, it is expected that users will experience:

• Ease of discovering, viewing and accessing datasets;

• Standardised ways of accessing data, including downloading or streaming data, with reduced need for special solutions on the user side;

• Reduced storage needs;

• Simple and standard access to remote datasets and catalogues, with own data visualisationl and analysis tools;

• Ability to compare and combine data from internal and external sources;

• Ability to apply common data transformations, like spatial, temporal and variables subsetting and reprojection, before downloading anything;

• Possibility to build specialized metadata catalogues and data portals targeting a specific user community.

1.1.1 External data management requirements and forcing mechanisms

Any organisation that strives to implement FAIR data management model has to relate to external forcing mechanisms concerning data management at several levels. At the national level, the organisation must comply with national regulations as decided by the government. Some of these are indications of expected behaviour (e.g., OECD regulations) and some are implemented through a legal framework. The Norwegian government has over time promoted free and open sharing of public data. Mechanisms for how to do this are governed by the Geodataloven (implemented as Geonorge), which is a national implementation of the European INSPIRE directive (to be amended in 2019). INSPIRE defines a federated multinational Spatial Data Infrastructure (SDI) for the European Union, similar to NSDI in the USA or UNSDI under the United Nations. The goal is to provide a standardised access to data and provide the necessary tools to be able to work with the data in a unified manner. In short, these legal frameworks require standardised documentation (at discovery and use level; these concepts are described later) and access (through specified protocols) to the data identified.

Other external requirements and forcing mechanisms that are organisation-specific are provided in ???.

1.1.2 The data value chain

The process of getting the data from the data producer to the consumer can be viewed as a value chain. An example of a data value chain is presented in ???. Typically, data from a wide variety of providers are used in the value chain. Traditionally, the data used have been transmitted on request from one data centre to another, and used in the specific processing chains that requested the data. The focus on reuse of data in various contexts has been missing.

Datasets and metadata are what travels through the value chain. At the end of the data management value chain are the data consumers.

1.1.3 Dataset

A dataset is a collection of data. In the context of the data management model, the storage mode of the dataset is irrelevant, since access mechanisms can be decoupled from the storage layer as experienced by a data consumer. Typically, a dataset represents a number of variables in time and space. A more detailed definition is provided in the Glossary of Terms. In order to best serve the data through web services, the following principles are useful for guiding the dataset definition:

1. A dataset can be a collection of variables stored in, for example, a relational database or as flat files;

2. A dataset is defined as a number of spatial and/or temporal variables;

3. A dataset should be defined by the information content and not the production method;

4. A good dataset does not mix feature types, i.e., trajectories and gridded data should not be present in the same dataset.

Point 3 implies that the output of, e.g., a numerical model may be divided into several datasets that are related. This is also important in order to efficiently serve the data through web services. For instance, model variables defined on different vertical coordinates should be separated as linked datasets, since some OGC services (e.g., WMS) are unable to handle mixed coordinates in the same dataset. One important linked dataset relation is the parent-child relationship. In the numerical model example, the parent dataset would be the model simulation. This (parent) dataset encompasses all datasets created by the model simulation such as, e.g., two NetCDF-CF files (child datasets) with different information content.

Most importantly, a dataset should be defined to meet the consumer needs. This means that the specification of a dataset should follow not only the content guidelines just listed, but also address the consumer needs for data delivery, security and preservation.

1.1.4 Metadata

Metadata is a broad concept. In our data management model the term "metadata" is used in several contexts, specifically the five categories that are briefly described in table_title.

The tools and facilities used to manage the information for efficient discovery and use are further described in ???.

1.1.5 A data management model based on the FAIR principles

The data management model is built upon the following principles:

• Standardisation – compliance with established international standards;

• Interoperability – enabling machine-to-machine interfaces including standardised documentation and encoding of data;

• Integrity – ensuring that data and data access can be maintained over time, and ensuring that the consumer receives the same data at any time of request;

• Traceability – documentation of the provenance of a dataset, i.e., all actions taken to produce and maintain the dataset and the usage of the data in downstream systems;

• Modularisation – enabling replacement of one component of the system without necessitating other changes.

The model’s basic functions fall into three main categories:

1. Documentation of data using discovery and use metadata. The documentation identifies who, what, when, where, and how, and shall make it easy for consumers to find and understand data. This requires application of information containers and utilisation of controlled vocabularies and ontologies where textual representation is required. It also covers the topic of data provenance which is used to describe the origin and all actions done on a dataset. Data provenance is closely linked with workflow management. Furthermore, it covers the relationship between datasets. Application of ontologies in data documentation is closely linked to the concept of linked data.

2. Publication and sharing of data focuses on making data accessible to consumers internally and externally. Application of standardised approaches is vital, along with cost effective solutions that are sustainable. Direct integration of data in applications for analysis through data streaming minimises the complexity and overhead in dissemination solutions. This category also covers persistent identifiers for data.

3. Preservation of data includes short and long term management of data, which secures access and availability throughout the lifespan of the data. Good solutions in this area depend on expected and actual usage of the data. Preservation of data includes the concept of data life cycle, i.e., the documented flow of data from initial storage through to obsolescence and permanent archiving (or deletion) and preserving the metadata for the same data (even after deleting).

1.2. Human roles in data management

1.2.1 Context

Data is processed and interpreted to generate knowledge (e.g., about the weather) for end users. The knowledge can be presented as information in the form of actual data, illustrations, text or other forms of communication. In this context, an illustration is a representation of data, whereas data means the numerical values needed to analyse and interpret a natural process (i.e., calibrated or with calibration information; it must be possible to understand the meaning of the numerical value from the available and machine-readable information).

Definition

Data here means the numerical values needed to analyse and interpret a natural process (i.e., calibrated or with calibration information, provenance, etc.; it must be possible to understand the meaning of the numerical value from the available and machine-readable information).

Advanced users typically consume some type of data in order to process and interpret it, and produce new knowledge, e.g., in the form of a new dataset or other information. The datasets can be organised in different levels, such as the WMO WIGOS definition for levels of data. Less advanced users apply information based on data (e.g., an illustration) to make decisions (e.g., clothing adapted to the forecast weather).

1.2.2 User definitions

We define two types of users:

1. Producers: Those that create / produce data

2. Consumers: Those that consume data

A consumer of one level of data is typically a producer of data at the next level. A user can both consume data and produce data, or just have one of these roles (e.g., at the start/end of the production chain).

1.2.2.1 Data consumers

We split between three types of data consumers: (1) advanced, (2) intermediate, and (3) simple. These are defined below.

1.2.2.1.1 Advanced consumers

Advanced consumers require information in the form of data and metadata (including provenance) to gain a full understanding of what data exists and how to use it (discovery and use metadata), and to automatize the generation of derived data (new knowledge generation), verification (of information), and validation of data products.

Example questions:

• Need all historical weather data, that can be used to model / predict the weather in the future

Specific consumers:

• Researcher (e.g., for climate projections within the "Klima i Norge 2100" research project)

1.2.2.1.2 Intermediate consumers

Intermediate consumers need enough information to find data and understand if it can answer their question(s) (discovery and use metadata). Also, they often want to cross reference a dataset with another dataset or metadata for inter-comparative verification of information.

Example questions:

• Is this observation a record / weather extreme (coldest, warmest, wettest)?

• What was the amount of rain in last month in a certain watershed?

Specific consumers:

• Klimavakt (MET)

• Developer (app, website, control systems, machine learning, etc.)

• Energy sector (hydro, energy prices)

• External partners

1.2.2.1.3 Simple consumers

Simple consumers do not have any prior knowledge about the data. Information in the form of text or illustrations is sufficient for their decision making. They do not need to understand either data or metadata, and they are most likely looking for answers to simple questions.

Example questions:

• Will it be raining today?

• Can the event take place, or will the weather impeed it?

• When should I harvest my crops?

Specific consumers:

• Event organizer

• Journalist

• Farmer, or other people who work with the land like tree planters

Note

An advanced consumer may discover information pertaining a role as a simple consumer. Such a user may, for some reason, be interested in tracking the data in order to use it together with other data (interoperability) or to verify the information. Therefore, it is important to have provenance metadata pointing to the basic data source(s) also at the simplest information level.

1.2.2.2 Data producers

A producer is an advanced consumer at one level of data that generate new information at a higher level. This new information could be in the form of actual data or simple information, such as an illustration or a text summary. It is essential that any information can be traced back to the source(s).

1.2.2.3 Data Management Roles

Between the data providers and data consumers are the processes that manage and deliver the datasets (cf. ???). A number of human roles may be defined with responsibilities that, together, ensure that these processes are carried out in accordance with the data management requirements of the organisation. The definition and filling of these roles depend heavily on the particular organisation, and each organisation must devise its own best solution.

1.3 Introduction to the data management at (insert organisation name here)

1.3.1 Background at (insert organsitation name here)

1.3.2 External data management requirements and forcing mechanisms specific to (insert organisation name here)

​

1.3.3 Data Management roles at (insert organisation name here)

1.4 Summary of data management requirements

The data management regime described in this DMH follows the Arctic Data centre model and shall ensure that:

1. There are relevant metadata for all datasets, and both data and metadata are available in a form and in such a way that they can be utilised by both humans and machines

   • There are sufficient metadata for each dataset for both discovery and use purposes

   • Discovery metadata are indexed and can be retrieved from available services in a standard way and with standard protocols

   • There are interfaces for discovery, visualisation and download, as well as portals for human access, that operate seamlessly across institutions

   • The data are described in a relevant, standardised and managed vocabulary that supports machine-machine interfaces

   • Datasets have attached a unique and permanent identifier that enables traceability

   • Datasets have licensing that ensures free use and reuse wherever possible

   • Datasets are available for download in a standard form according to the FAIR guiding principles and through standard protocols that are accepted and utilised in the user environment

   • There are authentication and authorisation mechanisms that ensure access control to data with restrictions, and that are compatible with and coupled to relevant public authentication solutions (FEIDE, eduGAIN, Google, etc.)

2. There is an organisation that provides for the management of each dataset throughout its lifetime (life cycle management)

   • There is documentation that describes physical storage, lifetime of each dataset, degree of storage redundancy, metadata consistency methods, how dataset versioning is implemented and unique IDs to ensure traceability

   • The organisation provides seamless access to data from distributed data centres through various portals

   • The above and a business model at dataset level are described in a Data Management Plan (DMP)

3. There are services or tools that provide the following functionalities on the datasets:

   • Transformations

     • Subsetting

     • Slicing of gridded datasets to points, sections, profiles

     • Reprojection

     • Resampling

     • Reformatting

   • Visualisation (time series, mapping services, etc.)

   • Aggregation

   • Upload of new datasets (including enabling and configuring data access services)

2. Structuring and Documenting Data for Efficient Discovery and Use

2.1 Purpose

In order to properly find, understand and use geophysical data, standardised encoding and documentation are required, i.e., metadata.

Both discovery metadata and use metadata can be embedded in the files produced for a dataset through utilisation of self-explaining file formats. If properly done by the data producer, publication and preservation of data through services is simplified and can be automated.

2.2 Implementation

An essential prerequisite for structuring and documenting data is the specification of the dataset(s), cf. ???. The dataset is the basic building block of our data management model; all the documentation and services described in this DMH are built on datasets. The dataset specification is the first step in structuring one’s data for efficient management, and it is mandatory.

2.3 Structuring and documenting data at [insert organisation here]

2.3.1 Current practice in structuring and documenting data

Table 2. Data types available at [insert institution here], with the fileformats supported. The primary fileformat is marked in bold

2.3.2 Planned developments in the near-term (< 2 years)

2.3.3 Expected evolution in the longer term (> 2 years)

3. Data Services

3.1 Purpose

The purpose of this chapter is to describe services that benefit from the standardisation performed in the previous step. The information structures described in the previous chapter pave the way for efficient data discovery and use through tools and automated services. Implementation of the services must be in line with the institute’s delivery architecture.

Data services include:

1. Data ingestion, storing the data in the proper locations for long term preservation and sharing;

2. Data cataloging, extracting the relevant information for proper discovery of the data;

3. Configuration of visualisation and data publication services (e.g., OGC WMS and OpeNDAP).

3.2 Implementation

When planning and implementing data services, there are a number of external requirements that constrain choices, especially if reuse of solutions nationally and internationally is intended and wanted for the data in question. At the national level, important constraints are imposed by the national implementation of the INSPIRE directive through Norge digitalt.

3.2.1 Legacy

3.2.2 Planned developments in near-term (< 2 years)

3.2.3 Expected evolution in the longer term (> 2 years)

4. User Portals and Documentation

4.1 Purpose

The purpose of this chapter is to describe the human interfaces which data consumers would use to navigate data and the related services. A portal is an entry point for data consumers, enabling them to discover and search for datasets and services, and providing sufficient documentation and guidance to ensure that they are able to serve themselves using the interactive and machine interfaces offered.

Here, we can distinguish between a general portal for all publishable datasets from the institution and targeted portals that offer a focused selection of data, which may include external datasets. Targeted portals cater to specific user groups and may have a limited lifetime, but also can be long-term commitments.

4.2 Implementation of user portals at NINA

Table 3: User portals in use at NINA

4.2.1 Current implementation

4.2.2 Planned developments in near-term (< 2 years)

4.2.3 Expected evolution in the longer term (> 2 years)

5. Data Governance

5.1 Purpose

This chapter describes how we organise and steer data management activities in order to ensure that:

1. The guidelines described above are implemented throughout the organisation;

2. Our data management practices are in line with and contribute to the institute’s strategic aims;

3. Our data management regime is subject to review, analysis and revision in a timely manner.

These higher level aspects of data management are often referred to as data governance. A useful definition is:

"Data governance … is the overall management of the availability, usability, integrity and security of data used in an enterprise. A sound data governance program includes a governing body or council, a defined set of procedures and a plan to execute those procedures."

In this chapter we address many aspects of this definition, but a full description of data governance touches on management structures that are beyond the scope of this handbook.

5.2 Data life cycle management

Data life cycle management is steered by documentation describing how data generated or used in an activity will be handled throughout the lifetime of the activity and after the activity has been completed. This is living documentation that follows the activity and specifies what kind of data will be generated or acquired, how the data will be described, where the data will be stored, whether and how the data can be shared, and how the data will be retired (archived or deleted). The purpose of life cycle management is to safeguard the data, not just during their “active” period but also for future reuse of the data, and to facilitate cost-effective data handling.

This DMH recommends the following concepts of life cycle management to be implemented for the institution:

• An institution specific Data Management Handbook (DMH) based on a common general template;

• Extended discovery metadata for data in internal production chains (these are metadata elements that provide the necessary information for life cycle management just described); and

• A Data Management Plan (DMP) document (a DMP is expected for datasets produced in external projects, but may also be useful for internal datasets, as a supplement to the extended discovery metadata).

The goal is that life cycle management information shall be readily available for every dataset managed by the institute. How these concepts are implemented are described in the subsections below.

5.2.1 Data Management Plan

A Data Management Plan (DMP) is a document that describes textually how the data life cycle management will be carried out for datasets used and produced in specific projects. Generally, these are externally financed projects for which such documentation is required by funding agencies. However, larger internal projects covering many datasets may also find it beneficial to create a specific document of this type.

Currently, agencies funding R&D (such as NFR and the EU) do not strictly require a DMP from the start of any project. However, for projects in the geosciences, data management is an issue that must be addressed, and the agencies strongly recommend a DMP solution. For example, NFR publishes guidelines for the contents of a DMP, including links to tools (templates and online services); these guidelines are recommended for any data management project or activity and will in time become a requirement according to NFR.

5.3 Data governance at NINA

5.3.1 Current implementation

5.3.1.1 Organisational Roles

5.3.1.2 Status DMH

5.3.1.3 Status Discovery metadata

5.3.1.3 Status DMP

5.3.2 Planned developments in the near-term (< 2 years)

Revise DMH annually or when needed.

5.3.3 Expected evolution in the longer term (> 2 years)

Revise DMH annually or when needed.

6. Practical Guides

This chapter includes how-to’s and other practical guidance for data producers.

6.1 Create a Data Management Plan (DMP)

The funding agency of your project will usually provide requirements, guidelines or a template for the DMP. If this is not the case or for datasets that are not part of a project use the template provided by your institution or the template based on the recommendations by Science Europe.

6.1.1 Using easyDMP

1. Log in to easyDMP, use Dataporten if your institution supports that, otherwise pick one of the other login methods.

2. Click on + Create a new plan and pick a template

3. By using the Summary button from page two and on, you can get an overview of all the questions.

6.1.2 Publishing the plan

Currently you can use the export function in easyDMP to download an HTML or PDF version of the DMP and use it further. This might change if "Hosted DMP" gets implemented.

6.2 Submitting data as NetCDF-CF

6.2.1 Workflow

1. Define your dataset (see dataset and ???)

2. Create a NetCDF-CF file (see Creating NetCDF-CF files)

   • Add discovery metadata as global attributes (see global attibutes section)

   • Add variables and use metadata following the CF conventions

3. Store the NetCDF-CF file in a suitable location, and distribute it via thredds or another dap server (see, e.g., How to add NetCDF-CF data to thredds)

4. Register your dataset in a searchable catalog (see How to register your data in the catalog service)

   • Test that your dataset contains the necessary discovery metadata and create an MMD xml file (see Generation of MMD xml file from NetCDF-CF)

   • Test the MMD xml file (see Test the MMD xml file)

   • Push the MMD xml file to the discovery metadata catalog (see Push the MMD xml file to the discovery metadata catalog)

6.2.2 Creating NetCDF-CF files

By documenting and formatting your data using NetCDF following the CF conventions and the Attribute Convention for Data Discovery (ACDD), MMD files can be automatically generated from the NetCDF files. The CF conventions is a controlled vocabulary providing a definitive description of what the data in each variable represents, and the spatial and temporal properties of the data. The ACDD vocabulary describes attributes recommended for describing a NetCDF dataset to data discovery systems. See, e.g., netCDF4-python docs, or xarray docs for documentation about how to create netCDF files.

The ACDD recommendations should be followed in order to properly document your netCDF-CF files. The below tables summarize required and recommended ACDD and some additional attributes that are needed to properly populate a discovery metadata catalog which fulfills the requirements of international standards (e.g., GCMD/DIF, the INSPIRE and WMO profiles of ISO19115, etc.).

6.2.2.1 Notes

Keywords describe the content of your dataset following a given vocabulary. You may use any vocabularies to define your keywords, but a link to the keyword definitions should be provided in the ``keywords_vocabulary`` attribute. This attribute provides information about the vocabulary defining the keywords used in the ``keywords`` attribute. Example:

:keywords_vocabulary = "GCMDSK:GCMD Science Keywords:https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/sciencekeywords, GEMET:INSPIRE Themes:http://inspire.ec.europa.eu/theme, NORTHEMES:GeoNorge Themes:https://register.geonorge.no/metadata-kodelister/nasjonal-temainndeling" ;

Note that the GCMDSK, GEMET and NORTHEMES vocabularies are required for indexing in S-ENDA and Geonorge. You may find appropriate keywords at the following links:

• GCMDSK

• GEMET

• NORTHEMES

The keywords should be provided by the ``keywords`` attribute as a comma separated list with a short name defining the vocabulary used, followed by the actual keyword, i.e., ``short_name:keyword``. Example:

:keywords = "GCMDSK:Earth Science > Atmosphere > Atmospheric radiation, GEMET:Meteorological geographical features, GEMET:Atmospheric conditions, NORTHEMES:Weather and climate" ;

See https://adc.met.no/node/96 for more information about how to define the ACDD keywords.

A data license provides information about any restrictions on the use of the dataset. To support a linked data approach, the ``license`` element should be supported by a ``license_resource`` element, providing a link to the license definition. Example:

:license = "CC-BY-4.0" ;
:license_resource = "http://spdx.org/licenses/CC-BY-4.0" ;

6.2.2.2 List of Attributes

This section provides lists of ACDD elements that are required and recommended, as well as some extra elements that are needed to fully support our data management needs. The right columns of these tables provide the MET Norway Metadata Specification (MMD) fields that map to the ACDD (and our extension to ACDD) elements. Please refer to MMD for definitions of these elements, as well as controlled vocabularies that should be used. Note that the below tables are automatically generated - check https://github.com/metno/py-mmd-tools/blob/master/py_mmd_tools/mmd_elements.yaml if anything is unclear.

In order to check your netCDF-CF files, and to create MMD xml files, you can use the nc2mmd.py script in the py-mmd-tools Python package.

The following ACDD elements are required:

The following ACDD elements are (highly) recommended:

The following elements are ACDD extensions that are needed to improve (meta)data interoperability. Please refer to the documentation of MMD for more details:

6.2.3 How to add NetCDF-CF data to thredds

This section should contain institution specific information about how to add netcdf-cf files to thredds.

6.2.4 How to register your data in the catalog service

In order to make a dataset findable, a dataset must be registered in a searchable catalog with appropriate metadata. The (meta)data catalog is indexed and exposed through CSW.

The following needs to be done:

1. Generate an MMD xml file from your NetCDF-CF file (see Generation of MMD xml file from NetCDF-CF)

2. Test your mmd xml metadata file (see Test the MMD xml file)

3. Push the MMD xml file to the discovery metadata catalog (see Push the MMD xml file to the discovery metadata catalog)

6.2.4.1 Generation of MMD xml file from NetCDF-CF

Clone the py-mmd-tools repo and make a local installation with eg pip install .. This should bring in all needed dependencies (we recommend to use a virtual environment).

Then, generate your mmd xml file as follows:

cd script ./nc2mmd.py -i <your netcdf file> -o <your xml output directory></programlisting>

See ./nc2mmd.py --help for documentation and extra options.

You will find Extensible Stylesheet Language Transformations (XSLT) documents in the MMD repository. These can be used to translate the metadata documents from MMD to other vocabularies, such as ISO19115:

./bin/convert_from_mmd -i <your mmd xml file> -f iso -o <your iso output file name></programlisting>

Note that the discovery metadata catalog ingestion tool will take care of translations from MMD, so you don’t need to worry about that unless you have special interest in it.

6.2.4.2 Test the MMD xml file

Install the dmci app, and run the usage example locally. This will return an error message if anything is wrong with your MMD file.

6.2.4.3 Push the MMD xml file to the discovery metadata catalog

For development and verification purposes:

curl --data-binary "@<PATH_TO_MMD_FILE>" https://dmci-*.s-enda.k8s.met.no/v1/insert</programlisting>

where * should be either dev or staging.

For production (the official catalog):

curl --data-binary "@<PATH_TO_MMD_FILE>" https://dmci.s-enda.k8s.met.no/v1/insert</programlisting>

6.3 Searching data in the Catalog Service for the Web (CSW) interface

6.3.1 Using OpenSearch

6.3.1.1 Local test machines

The vagrant-s-enda environment found at vagrant-s-enda provides OpenSearch support through PyCSW. To test OpenSearch via the browser, start the vagrant-s-enda vm (vagrant up) and go to the following address:

• http://10.10.10.10/pycsw/csw.py?mode=opensearch&service=CSW&version=2.0.2&request=GetCapabilities

6.3.1.2 Online catalog

For searching the online metadata catalog, the base url (http://10.10.10.10/) must be replaced by https://csw.s-enda.k8s.met.no/:

• http://csw.s-enda.k8s.met.no/?mode=opensearch&service=CSW&version=2.0.2&request=GetRecords&elementsetname=full&typenames=csw:Record&resulttype=results

6.3.1.3 OpenSearch examples

To find all datasets in the catalog:

• https://csw.s-enda.k8s.met.no/?mode=opensearch&service=CSW&version=2.0.2&request=GetRecords&elementsetname=full&typenames=csw:Record&resulttype=results

Or datasets within a given time span:

• http://csw.s-enda.k8s.met.no/?mode=opensearch&service=CSW&version=2.0.2&request=GetRecords&elementsetname=full&typenames=csw:Record&resulttype=results&time=2000-01-01/2020-09-01

Or datasets within a geographical domain (defined as a box with parameters min_longitude, min_latitude, max_longitude, max_latitude):

• https://csw.s-enda.k8s.met.no/?mode=opensearch&service=CSW&version=2.0.2&request=GetRecords&elementsetname=full&typenames=csw:Record&resulttype=results&bbox=0,40,10,60

Or, datasets from any of the Sentinel satellites:

• https://csw.s-enda.k8s.met.no/?mode=opensearch&service=CSW&version=2.0.2&request=GetRecords&elementsetname=full&typenames=csw:Record&resulttype=results&q=sentinel

6.3.2 Advanced geographical search

PyCSW opensearch only supports geographical searches querying for a box. For more advanced geographical searches, one must write specific XML files. For example:

• To find all datasets containing a point:

<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?>
<csw:GetRecords
    xmlns:csw="http://www.opengis.net/cat/csw/2.0.2"
    xmlns:ogc="http://www.opengis.net/ogc"
    xmlns:gml="http://www.opengis.net/gml"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    service="CSW"
    version="2.0.2"
    resultType="results"
    maxRecords="10"
    outputFormat="application/xml"
    outputSchema="http://www.opengis.net/cat/csw/2.0.2"
    xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd" >
  <csw:Query typeNames="csw:Record">
    <csw:ElementSetName>full</csw:ElementSetName>
    <csw:Constraint version="1.1.0">
      <ogc:Filter>
        <ogc:Contains>
          <ogc:PropertyName>ows:BoundingBox</ogc:PropertyName>
          <gml:Point>
            <gml:pos srsDimension="2">59.0 4.0</gml:pos>
          </gml:Point>
        </ogc:Contains>
      </ogc:Filter>
    </csw:Constraint>
  </csw:Query>
</csw:GetRecords>

• To find all datasets intersecting a polygon:

<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?>
<csw:GetRecords
    xmlns:csw="http://www.opengis.net/cat/csw/2.0.2"
    xmlns:gml="http://www.opengis.net/gml"
    xmlns:ogc="http://www.opengis.net/ogc"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    service="CSW"
    version="2.0.2"
    resultType="results"
    maxRecords="10"
    outputFormat="application/xml"
    outputSchema="http://www.opengis.net/cat/csw/2.0.2"
    xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd" >
  <csw:Query typeNames="csw:Record">
    <csw:ElementSetName>full</csw:ElementSetName>
    <csw:Constraint version="1.1.0">
      <ogc:Filter>
        <ogc:Intersects>
          <ogc:PropertyName>ows:BoundingBox</ogc:PropertyName>
          <gml:Polygon>
            <gml:exterior>
              <gml:LinearRing>
                <gml:posList>
                  47.00 -5.00 55.00 -5.00 55.00 20.00 47.00 20.00 47.00 -5.00
                </gml:posList>
              </gml:LinearRing>
            </gml:exterior>
          </gml:Polygon>
        </ogc:Intersects>
      </ogc:Filter>
    </csw:Constraint>
  </csw:Query>
</csw:GetRecords>

• Then, you can query the CSW endpoint with, e.g., python:

import requests
requests.post('https://csw.s-enda.k8s.met.no', data=open(my_xml_request).read()).text

6.3.3 Web portals

GeoNorge.no

TODO: describe how to search in geonorge, possibly with screenshots

6.3.4 QGIS

MET Norway’s S-ENDA CSW catalog service is available at https://csw.s-enda.k8s.met.no. This can be used from QGIS as follows:

1. Select Web > MetaSearch > MetaSearch menu item

2. Select Services > New

3. Type, e.g., csw.s-enda.k8s.met.no for the name

4. Type https://csw.s-enda.k8s.met.no for the URL

Under the Search tab, you can then add search parameters, click Search, and get a list of available datasets.

Acknowledgements

At various stages during the writing of the first version of this handbook, we have solicited comments on the manuscript from coworkers at MET Norway (in alphabetical order): Åsmund Bakketun, Arild Burud, Lara Ferrighi, Håvard Futsæter and Nina Larsgård. Their comments and advice are gratefully acknowledged. In addition, we thank members of the top management at MET Norway, including Lars-Anders Breivik, Bård Fjukstad, Jørn Kristensen, Anne-Cecilie Riiser, Roar Skålin and Cecilie Stenersen, who have provided valuable criticism and advice.

While working on the second version of this handbook, valuable input has come from Matteo De Stefano from NINA. This input has made it possible to transform this handbook into a tool that can be adopted by institutions outside of METNorway.

Glossary of Terms and Names

List of Acronyms

This list contains acronyms used throughout the DMH. The column "General/Specific" indicates if the acronyms are used in the general part of the DMH (from the template) or if they are organisation specific.

Appendix A: list of referenced softwares & services