Full workflow for importing ERA5-Land into DHIS2

In this example workflow, we demonstrate how to use DHIS2 Climate Tools to make sure that DHIS2 is continuously updated with the latest climate data. Specifically we show step by step how to download and import the latest available daily precipitation data from the ERA5-Land hourly data hosted at the Climate Data Store.

The notebook fetches and imports only data that is not yet present in DHIS2, making it safe to run on a recurring basis and ensuring that DHIS2 stays up to date with the latest available climate data.

By updating the input parameters at the top of the notebook, it’s possible to also run the import for other ERA5-Land climate variables.

If you’re only interested in downloading ERA5-Land data, see this detailed step-by-step guide.

Important: This notebook only aggregates to daily periods according the Gregorian calendar. Other calendar systems, like those used in Nepal or Ethiopia, are not yet supported.

Prerequisites¶

Before proceeding with the notebook, make sure the following are in place:

1. CDS API access¶

Make sure you have followed these instructions to authenticate and allow API access the CDS portal.

2. Required DHIS2 data element¶

Your DHIS2 instance must contain a data element that can receive the imported data.

For daily precipitation, the data element must have:

• valueType = NUMBER

• aggregationType = SUM

• It must belong to a data set with periodType = DAILY

If this data element does not already exist, you have two options:

• Create the data element manually in DHIS2.

• Or create the data element using the Python client.

Once the data element exists, copy its UID and set it as DHIS2_DATA_ELEMENT_ID in the Input Parameters section further down.

Library imports¶

We start by importing the necessary libraries:

Input parameters¶

Let’s first define all the input parameters so they are clearly stated at the top of the notebook.

For this example we will connect to a public DHIS2 instance, so it’s important that you create the precipitation data element (as described previously) and update the DHIS2_DATA_ELEMENT_ID below. Note also that the public instance resets every night, so this process will have to be repeated for each new day.

Since we are importing from hourly to daily data, setting the DHIS2_TIMEZONE_OFFSET parameter is needed to aggregate the hourly data to the correct days in the local timezone.

Note that IMPORT_FROM_UNITS and IMPORT_TO_UNITS define the units of the source data and the desired units for import, and should match the IMPORT_VARIABLE being imported. For example, to convert temperature data from Kelvin to degrees Celsius, set these to K and degC. For other unit conversions, see the MetPy documentation for working with units.

Connect to DHIS2¶

First, we connect the python-client to the DHIS2 instance we want to import into. You can point this to your own instance, but for the purposes of this example we will use one of the public access DHIS2 instances, since these are continuously reset:

Get the DHIS2 organisation units¶

In order to download and aggregate the data to our DHIS2 organisation units, we also use the python-client to get the level 2 organisation units from our DHIS2 instance:

Check when the data was last imported¶

Since we want to run this script on a regular interval, we want to avoid importing data that has already been imported. We therefore first want to check the last date for which data was imported for the data element we want to import into. This can be done using the convenience function analytics_latest_period_for_level() provided by the python-client:

Let’s extract and report the last imported month:

We then use this information to define when we will start the data download, and ensure that we only download data after the last_imported_string:

Download the necessary data¶

In the next step we download all the requested data to the local file system, using convenience functionality from the dhis2eo.data.cds.era5_land module.

Running this step may take some time depending on how many months of data are requested.

Note that after the initial data download, subsequent runs of this notebook will re-use the previously imported files to avoid repeated downloads of the same data.

For more details on this step, see our guide for Downloading ERA5-Land data.

Open the downloaded data¶

Once the data has been downloaded, we can then pass the list of files to xr.open_mfdataset(). This allows us to open and work with the data as if it were a single xarray dataset:

Handle cumulative variables¶

Some variables in ERA5-Land, such as total precipitation, are stored as cumulative (running total) values. These must be de-accumulated prior to aggregation, so that each hour only represents the precipitation that occured during the preceding hour:

Aggregate from hours to days¶

In this example we want to import daily data to DHIS2. We therefore aggregate the data from hourly to daily using earthkit.transforms. We also pass the DHIS2_TIMEZONE_OFFSET input to correctly match the hours with what should be considered a day in the local timezone of your country:

Aggregate to organisation units¶

Once the data have been aggregated to the correct daily level, we can then aggregate the gridded data to the organisation units from your DHIS2 instance:

Post-processing¶

After aggregating the precipitation data to the desired organizational units, we convert the xarray Dataset to a Pandas DataFrame. This makes it easier to inspect the data and prepare it for subsequent post-processing:

As seen above, the ERA5-Land precipitation is reported in meters, but the desired import unit is millimeters. In this step, to support converting between user-defined source and target units, we leverage MetPy’s units module to automatically convert between units before importing the data:

Create DHIS2 payload¶

At this point we have the final data that we want to import into DHIS2. In order to submit the data to DHIS2 we first have to convert the data a standardized JSON format, which can be done with the help of the dhis2eo library:

Import to DHIS2¶

We have now successfully completed a full workflow for downloading, postprocessing, aggegating, and importing daily ERA5-Land precipitation data into DHIS2.