Data Sync Scripts
datasync
Provide subcommands for synchronizing different resources, see subcommands
Usage
datasync [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--install-completion |
Install completion for the current shell. | No | - |
--show-completion |
Show completion for the current shell, to copy it or customize the installation. | No | - |
Commands
| Name | Description |
|---|---|
coat |
Export COAT data to parquet |
nva |
Commands to handle NVA tasks |
ubw |
Export UBW APIs to Parquet in S3 bucket |
dms |
|
ninagen |
Commands to handle NINAGEN tasks |
pit-registering-salmon |
|
grass-gis |
|
services |
Miljødata Infrastructure as Code pipelines |
gbif-backbone |
export GBIF Backbone data to DuckDB database |
ipt |
Provide commands to deal with IPT |
mediebank |
Commands to handle Mediebank employee... |
Subcommands
coat
Export COAT data to parquet
Usage
datasync coat [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
get-packages-and-resources
Run the COAT extraction pipeline.
Usage
datasync coat get-packages-and-resources [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--endpoint-url |
AWS S3 endpoint URL | Yes | - |
--access-key |
AWS S3 access key | Yes | - |
--secret-key |
AWS S3 secret key | Yes | - |
--bucket |
AWS S3 bucket name | Yes | - |
--prefix |
AWS S3 prefix (folder path) for storing data | No | coat |
--region |
AWS S3 region | No | us-east-1 |
--api-key |
CKAN API key | Yes | - |
--base-url |
COAT CKAN API base URL | No | https://data.coat.no/api/3/action |
--dataset-name |
Local pipeline name (used for dlt's local working/state directory) | No | coat |
get-plausible-analytics
Run the Plausible analytics pipeline for data.coat.no.
Usage
datasync coat get-plausible-analytics [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--endpoint-url |
AWS S3 endpoint URL | Yes | - |
--access-key |
AWS S3 access key | Yes | - |
--secret-key |
AWS S3 secret key | Yes | - |
--bucket |
AWS S3 bucket name | Yes | - |
--prefix |
AWS S3 prefix (folder path) for storing data | No | coat |
--region |
AWS S3 region | No | us-east-1 |
--api-key |
Plausible Stats API key | Yes | - |
--site-id |
Plausible site ID (e.g. data.coat.no) | No | data.coat.no |
nva
Commands to handle NVA tasks
Usage
datasync nva [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
run
Sync NVA data from REST API to target
Usage
datasync nva run [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--resources / --no-resources |
No | no-resources |
|
--projects / --no-projects |
No | no-projects |
|
--persons / --no-persons |
No | no-persons |
|
--categories / --no-categories |
No | no-categories |
|
--funding-sources / --no-funding-sources |
No | no-funding-sources |
|
--base-url |
Base URL for the NVA API | No | https://api.nva.unit.no/ |
--duckdb-name |
No | nva_sync |
|
--institution-code |
NVA institution code | No | 7511.0.0.0 |
--endpoint-url |
AWS S3 endpoint URL | Yes | - |
--access-key |
AWS S3 access key | Yes | - |
--secret-key |
AWS S3 secret key | Yes | - |
--bucket |
AWS S3 bucket name | Yes | - |
--prefix |
AWS S3 prefix (folder path) for storing data | Yes | - |
--region |
AWS S3 region | No | us-east-1 |
filter-data
Filter NVA parquet files and export to S3
Usage
datasync nva filter-data [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--resource-name |
Name for the filtered resource | Yes | - |
--data-s3-path |
S3 path for input data | Yes | - |
--storage-s3-endpoint |
S3 endpoint URL for storage | Yes | - |
--storage-access-key |
S3 access key for storage | Yes | - |
--storage-secret-key |
S3 secret key for storage | Yes | - |
--storage-bucket |
S3 bucket for storage | Yes | - |
--storage-prefix |
S3 prefix for storage | Yes | - |
--filter |
Filter by column value (format: column=value). Can specify multiple filters. | No | - |
--show-columns |
Display available column names and exit | No | False |
search-resources-api
Fetch and filter NVA data with flexible search parameters
Usage
datasync nva search-resources-api [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--resource-name |
Name for the DLT resource | Yes | - |
--base-url |
Base URL for the NVA API | No | https://api.nva.unit.no/ |
--filter, -f |
Filter parameters as key=value pairs. Valid keys: {'contributor_name', 'publisher', 'status', 'created_before', 'from', 'order', 'funding', 'course', 'institution_should', 'publisher_id', 'title', 'grant_id_not', 'has_file', 'scientific_report_period_before', 'license_not', 'contributor_name_should', 'unit', 'publication_year_since', 'grant_id', 'id_should', 'orcid', 'id', 'project_not', 'scientific_report_period_since', 'doi', 'published_between', 'title_not', 'contributor', 'fields', 'publisher_not', 'publisher_id_should', 'orcid_not', 'contributor_name_not', 'modified_since', 'doi_not', 'tags_not', 'funding_source_should', 'page', 'issn', 'funding_source', 'user_affiliation_not', 'series', 'instance_type', 'lang', 'id_not', 'series_not', 'isbn_not', 'doi_should', 'user', 'project_should', 'query', 'Accept', 'scientific_value', 'unit_not', 'publisher_id_not', 'exclude_parent_type', 'license', 'category_should', 'parent_publication_year', 'parent_type', 'instance_type_should', 'status_should', 'scopus_identifier', 'exclude_subunits', 'unit_should', 'title_should', 'instance_type_not', 'abstract', 'funding_identifier_should', 'context_type_should', 'aggregation', 'institution_not', 'funding_source_not', 'top_level_organization', 'size', 'publication_language', 'contributor_not', 'user_not', 'orcid_should', 'user_should', 'issn_not', 'tags', 'parent_publication', 'funding_identifier_not', 'context_type_not', 'created_since', 'published_since', 'course_not', 'contributor_should', 'institution', 'user_affiliation', 'user_affiliation_should', 'category', 'parent_publication_should', 'published_before', 'publication_language_not', 'context_type', 'license_should', 'course_should', 'series_should', 'tags_should', 'exclude_parent_publication_year', 'publisher_should', 'grant_id_should', 'abstract_should', 'isbn_should', 'publication_year_before', 'sort', 'has_isbn', 'modified_before', 'has_issn', 'publication_year', 'publication_language_should', 'cristin_identifier', 'status_not', 'issn_should', 'project', 'abstract_not', 'category_not', 'funding_identifier', 'isbn'} | No | [] |
--apply-filter / --no-apply-filter |
Apply extract_useful_info.sql transformation | No | apply-filter |
--storage-endpoint-url |
S3 endpoint URL for storage | Yes | - |
--storage-access-key |
S3 access key for storage | Yes | - |
--storage-secret-key |
S3 secret key for storage | Yes | - |
--storage-bucket |
S3 bucket for storage | Yes | - |
--storage-prefix |
S3 prefix for storage | Yes | - |
--storage-region |
S3 region for storage | No | us-east-1 |
--add-timestamp / --no-add-timestamp |
Whether to write last successful run timestamp to S3 | No | add-timestamp |
get-pubs-per-researcher
Fetch publications for researchers at NINA
Usage
datasync nva get-pubs-per-researcher [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--employees-parquet |
Path to employees parquet file | Yes | - |
--storage-s3-path |
S3 endpoint URL for storage | Yes | - |
--storage-access-key |
S3 access key for storage | Yes | - |
--storage-secret-key |
S3 secret key for storage | Yes | - |
--storage-bucket |
S3 bucket for storage | Yes | - |
--storage-prefix |
S3 prefix for storage | Yes | - |
--storage-region |
S3 region for storage | No | us-east-1 |
--test / --no-test |
If set, will only fetch publications for the first 3 researchers | No | no-test |
ubw
Export UBW APIs to Parquet in S3 bucket
Usage
datasync ubw [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
run
No description available
Usage
datasync ubw run [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--access-key |
No | - | |
--secret-key |
No | - | |
--endpoint-url |
No | - | |
--bucket |
No | - | |
--prefix |
No | - | |
--base-url |
No | - | |
--auth |
No | - |
dms
No description available
Usage
datasync dms [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
generate-csw-metadata
No description available
Usage
datasync dms generate-csw-metadata [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--base-url |
No | - | |
--access-key |
No | - | |
--secret-key |
No | - | |
--endpoint |
No | - | |
--bucket |
No | - | |
--publish-url |
No | - | |
--limit |
No | - | |
--search |
Filter resources by title using a LIKE expression | No | - |
generate-geoapi-config
No description available
Usage
datasync dms generate-geoapi-config [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--base-url |
No | - | |
--publish-url |
No | - | |
--search |
Filter resources by title using a LIKE expression | No | - |
generate-maps-json
Generate a maps.json file from map resources in the DMS parquet files. The output follows the format used by the NINA map-editor. The URL for each map is read from the uri field of the resource. The file is written to S3 as a publicly accessible file.
Usage
datasync dms generate-maps-json [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--base-url |
No | - | |
--access-key |
No | - | |
--secret-key |
No | - | |
--endpoint |
No | - | |
--where |
Provide an additional SQL filter | No | 1=1 |
--bucket |
S3 bucket for output (e.g., 'my-bucket') | No | - |
--output |
S3 key path for output JSON file | No | /dms/maps/maps.json |
ninagen
Commands to handle NINAGEN tasks
Usage
datasync ninagen [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
snp-database-normalize
Convert SNP excel sheet to parquet
Usage
datasync ninagen snp-database-normalize [OPTIONS] FILE [SHEET]
Arguments
| Name | Description | Required |
|---|---|---|
FILE |
path to the file | Yes |
SHEET |
Name of the Excel Sheet to use | No |
Options
| Name | Description | Required | Default |
|---|---|---|---|
--header-row |
XLSX Row number that contains the header | No | 1 |
--allele-start-column |
XLSX column that contains the first allele | No | F |
snp-analysis-to-parquet
Convert SNP csv of an analysis to a parquet file
Usage
datasync ninagen snp-analysis-to-parquet [OPTIONS] FILE
Arguments
| Name | Description | Required |
|---|---|---|
FILE |
Path to the csv file | Yes |
Options
No options available
pit-registering-salmon
No description available
Usage
datasync pit-registering-salmon [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
run
Download PIT data from BioMark's API to a .duckdb file.
Usage
datasync pit-registering-salmon run [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--duckdb-path |
No | biomark_pit_registering_salmon_v1.duckdb |
|
--place |
Site location (kongsfjord, sylte, vigda, agdenes, vatne) | No | - |
--begin-date |
Start date for data download in YYYY-MM-DD format | No | - |
--end-date |
End date for data download in YYYY-MM-DD format | No | - |
--tags / --no-tags |
Download tags data | No | no-tags |
--readers / --no-readers |
Download readers voltage data | No | no-readers |
--environment / --no-environment |
Download environment data | No | no-environment |
--all-locations / --no-all-locations |
Download data from all accessible locations | No | no-all-locations |
--base-url |
No | https://data3.biomark.com/api/v1/ |
|
--yesterday / --no-yesterday |
Set date range to yesterday only | No | no-yesterday |
--dataset-name |
No | main |
grass-gis
No description available
Usage
datasync grass-gis [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
register-layers
No description available
Usage
datasync grass-gis register-layers [OPTIONS] PARQUET_FILE_PATH
Arguments
| Name | Description | Required |
|---|---|---|
PARQUET_FILE_PATH |
Yes | |
PROJECT_NUMBER |
Yes | |
GISBASE |
Yes |
Options
No options available
services
Miljødata Infrastructure as Code pipelines
Usage
datasync services [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
services-to-parquet
Convert metadata.yml definitions to a set of parquet that can be imported in the DMS
Usage
datasync services services-to-parquet [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--org |
No | ninanor |
|
--repo |
No | - | |
--bucket |
No | - | |
--endpoint |
No | - | |
--access-key |
No | - | |
--secret-key |
No | - | |
--prefix |
No | /dms/tables |
|
--git-username |
No | - | |
--git-token |
No | - |
dashboard
Produce a Homer Dashbord using the Miljødata Infrastructure as Code repository as data source
Usage
datasync services dashboard [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--org |
No | ninanor |
|
--repo |
No | - | |
--config-org |
No | ninanor |
|
--config-repo |
No | - | |
--bucket |
No | - | |
--endpoint |
No | - | |
--access-key |
No | - | |
--secret-key |
No | - | |
--git-username |
No | - | |
--git-token |
No | - | |
--prefix |
No | /dms/services |
dashboard-test
Test the parsing of the services metadata and the generation of the dashboard configuration
Usage
datasync services dashboard-test [OPTIONS]
Arguments
No arguments available
Options
No options available
gbif-backbone
export GBIF Backbone data to DuckDB database
Usage
datasync gbif-backbone [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
import-all
Import GBIF Backbone data into a DuckDB database.
Usage
datasync gbif-backbone import-all [OPTIONS]
Arguments
No arguments available
Options
No options available
ipt
Provide commands to deal with IPT
Usage
datasync ipt [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
run
Convert IPT resources to geoparquet, register them in the DMS, publish metadata and configurations
Usage
datasync ipt run [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--skip-data / --no-skip-data |
Ignore data conversion step, perform only metadata | No | no-skip-data |
--skip-dms / --no-skip-dms |
Skip publishing to DMS | No | no-skip-dms |
--skip-csw / --no-skip-csw |
Skip publishing to CSW | No | no-skip-csw |
--skip-geoapi / --no-skip-geoapi |
Skip publishing to pygeoapi | No | no-skip-geoapi |
--limit |
Only import a certain amount of records | No | - |
--search |
execute only on resources which contains that string | No | - |
validate-iso
Validate an XML file against ISO 19115 schema
Usage
datasync ipt validate-iso [OPTIONS] FILE
Arguments
| Name | Description | Required |
|---|---|---|
FILE |
Yes |
Options
No options available
mediebank
Commands to handle Mediebank employee portraits
Usage
datasync mediebank [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
No options available
Subcommands
employees-portraits
Sync employee portraits from Mediebank to S3
Usage
datasync mediebank employees-portraits [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
--client-id |
Mediebank API client id | Yes | - |
--client-secret |
Mediebank API client secret | Yes | - |
--token-url |
OAuth2 token URL | No | https://login.sdl.no/oauth/token |
--audience |
OAuth2 audience | No | https://api.ntb.no |
--api-url |
Mediebank API base URL | No | https://api.ntb.no/media/v1/mb |
--collection |
Portraits collection id | No | 67480 |
--employees-parquet |
Path to the employees parquet file | No | https://s3-ext-1.nina.no/dms/nina/employees.parquet |
--s3-endpoint-url |
S3 endpoint URL | Yes | - |
--s3-access-key |
S3 access key | Yes | - |
--s3-secret-key |
S3 secret key | Yes | - |
--s3-bucket |
S3 bucket | Yes | - |
--s3-prefix |
S3 prefix for the portraits | No | nina.no/ansattbilder |