2. Common DataStore functions¶

Examples of how to do some of the more commonly used functions:

mean, min, max, std
Selecting
Selecting by index
Downsample (time dimension)
Upsample / Interpolation (length and time dimension)

[1]:

import os

from dtscalibration import read_silixa_files

First we load the raw measurements into a DataStore object, as we learned from the previous notebook.

[2]:

filepath = os.path.join("..", "..", "tests", "data", "single_ended")

ds = read_silixa_files(directory=filepath, timezone_netcdf="UTC", file_ext="*.xml")

3 files were found, each representing a single timestep
4 recorded vars were found: LAF, ST, AST, TMP
Recorded at 1461 points along the cable
The measurement is single ended
Reading the data from disk

/home/docs/checkouts/readthedocs.org/user_builds/python-dts-calibration/envs/latest/lib/python3.9/site-packages/dtscalibration/datastore.py:81: UserWarning: Converting non-nanosecond precision timedelta values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  super().__init__(*args, **kwargs)
/home/docs/checkouts/readthedocs.org/user_builds/python-dts-calibration/envs/latest/lib/python3.9/site-packages/dtscalibration/datastore.py:81: UserWarning: Converting non-nanosecond precision timedelta values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  super().__init__(*args, **kwargs)

1 mean, min, max¶

The first argument is the dimension. The function is taken along that dimension. dim can be any dimension (e.g., time, x). The returned DataStore does not contain that dimension anymore.

Normally, you would like to keep the attributes (the informative texts from the loaded files), so set keep_attrs to True. They don’t take any space compared to your Stokes data, so keep them.

Note that also the sections are stored as attribute. If you delete the attributes, you would have to redefine the sections.

[5]:

ds_min = ds.mean(
    dim="time", keep_attrs=True
)  # take the minimum of all data variables (e.g., Stokes, Temperature) along the time dimension

[6]:

ds_max = ds.max(
    dim="x", keep_attrs=True
)  # Take the maximum of all data variables (e.g., Stokes, Temperature) along the x dimension

[7]:

ds_std = ds.std(
    dim="time", keep_attrs=True
)  # Calculate the standard deviation along the time dimension

2 Selecting¶

What if you would like to get the maximum temperature between \(x >= 20\) m and \(x < 35\) m over time? We first have to select a section along the cable.

[8]:

section = slice(20.0, 35.0)
section_of_interest = ds.sel(x=section)

[9]:

section_of_interest_max = section_of_interest.max(dim="x")

What if you would like to have the measurement at approximately \(x=20\) m?

[10]:

point_of_interest = ds.sel(x=20.0, method="nearest")

3 Selecting by index¶

What if you would like to see what the values on the first timestep are? We can use isel (index select)

[11]:

section_of_interest = ds.isel(time=slice(0, 2))  # The first two time steps

[12]:

section_of_interest = ds.isel(x=0)

4 Downsample (time dimension)¶

We currently have measurements at 3 time steps, with 30.001 seconds inbetween. For our next exercise we would like to down sample the measurements to 2 time steps with 47 seconds inbetween. The calculated variances are not valid anymore. We use the function resample from xarray.

[13]:

# We use the logic from xarray to resample. However, it returns an xarray dataset type. Therefore we convert it back to the dtscalibration Datastore type.
from dtscalibration import DataStore

ds_resampled = DataStore(ds.resample(time="47S").mean())

5 Upsample / Interpolation (length and time dimension)¶

So we have measurements every 0.12 cm starting at \(x=0\) m. What if we would like to change our coordinate system to have a value every 12 cm starting at \(x=0.05\) m. We use (linear) interpolation, extrapolation is not supported. The calculated variances are not valid anymore.

[14]:

x_old = ds.x.data
x_new = x_old[:-1] + 0.05  # no extrapolation
ds_xinterped = ds.interp(coords={"x": x_new})

We can do the same in the time dimension

[15]:

import numpy as np

time_old = ds.time.data
time_new = time_old + np.timedelta64(10, "s")
ds_tinterped = ds.interp(coords={"time": time_new})

[ ]:

2. Common DataStore functions¶

0 Access the data¶

1 mean, min, max¶

2 Selecting¶

3 Selecting by index¶

4 Downsample (time dimension)¶

5 Upsample / Interpolation (length and time dimension)¶

Table of Contents

This Page