A4. Loading sensortran files

This example loads sensortran files. Only single-ended measurements are currently supported. Sensortran files are in binary format. The library requires the *BinaryRawDTS.dat and *BinaryTemp.dat files.

import os
import glob
import matplotlib.pyplot as plt
from pandas.plotting import register_matplotlib_converters

register_matplotlib_converters()

from dtscalibration import read_sensortran_files

The example data files are located in ./python-dts-calibration/tests/data.

filepath = os.path.join("..", "..", "tests", "data", "sensortran_binary")
print(filepath)

../../tests/data/sensortran_binary

filepathlist = sorted(glob.glob(os.path.join(filepath, "*.dat")))
filenamelist = [os.path.basename(path) for path in filepathlist]

for fn in filenamelist:
    print(fn)

15_56_47_BinaryRawDTS.dat
15_56_47_BinaryTemp.dat
16_11_31_BinaryRawDTS.dat
16_11_31_BinaryTemp.dat
16_29_23_BinaryRawDTS.dat
16_29_23_BinaryTemp.dat

We will simply load in the binary files

ds = read_sensortran_files(directory=filepath)

3 files were found, each representing a single timestep
Recorded at 11582 points along the cable
The measurement is single ended

/home/docs/checkouts/readthedocs.org/user_builds/python-dts-calibration/envs/latest/lib/python3.9/site-packages/dtscalibration/datastore.py:81: UserWarning: Converting non-nanosecond precision timedelta values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  super().__init__(*args, **kwargs)
/home/docs/checkouts/readthedocs.org/user_builds/python-dts-calibration/envs/latest/lib/python3.9/site-packages/dtscalibration/datastore.py:81: UserWarning: Converting non-nanosecond precision timedelta values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  super().__init__(*args, **kwargs)

The object tries to gather as much metadata from the measurement files as possible (temporal and spatial coordinates, filenames, temperature probes measurements). All other configuration settings are loaded from the first files and stored as attributes of the DataStore. Sensortran’s data files contain less information than the other manufacturer’s devices, one being the acquisition time. The acquisition time is needed for estimating variances, and is set a constant 1s.

print(ds)

<dtscalibration.DataStore>
Sections:                  ()
Dimensions:                (x: 11582, time: 3, trans_att: 0)
Coordinates:
  * x                      (x) float32 -451.4 -450.9 ... 5.408e+03 5.409e+03
    filename               (time) <U25 '15_56_47_BinaryRawDTS.dat' ... '16_29...
    filename_temp          (time) <U23 '15_56_47_BinaryTemp.dat' ... '16_29_2...
    timestart              (time) datetime64[ns] 2009-09-23T22:56:46 ... 2009...
    timeend                (time) datetime64[ns] 2009-09-23T22:56:47 ... 2009...
  * time                   (time) datetime64[ns] 2009-09-23T22:56:47 ... 2009...
    acquisitiontimeFW      (time) timedelta64[ns] 00:00:01 00:00:01 00:00:01
  * trans_att              (trans_att) float64
Data variables:
    st                     (x, time) int32 39040680 39057147 ... 39071213
    ast                    (x, time) int32 39048646 39064414 ... 39407668
    tmp                    (x, time) float64 -273.1 -273.1 ... 82.41 82.71
    referenceTemperature   (time) float64 28.61 29.24 30.29
    st_zero                (time) float64 3.904e+07 3.906e+07 3.907e+07
    ast_zero               (time) float64 3.905e+07 3.907e+07 3.908e+07
    userAcquisitionTimeFW  (time) float64 1.0 1.0 1.0
Attributes: (12/16)
    survey_type:                 2
    hdr_version:                 3
    x_units:                     n/a
    y_units:                     counts
    num_points:                  12000
    num_pulses:                  25000
    ...                          ...
    hdr_size:                    176
    hw_config:                   84

.. and many more attributes. See: ds.attrs

The sensortran files differ from other manufacturers, in that they return the ‘counts’ of the Stokes and anti-Stokes signals. These are not corrected for offsets, which has to be done manually for proper calibration.

Based on the data available in the binary files, the library estimates a zero-count to correct the signals, but this is not perfectly accurate or constant over time. For proper calibration, the offsets would have to be incorporated into the calibration routine.

ds

<dtscalibration.DataStore>
Sections:                  ()
Dimensions:                (x: 11582, time: 3, trans_att: 0)
Coordinates:
  * x                      (x) float32 -451.4 -450.9 ... 5.408e+03 5.409e+03
    filename               (time) <U25 '15_56_47_BinaryRawDTS.dat' ... '16_29...
    filename_temp          (time) <U23 '15_56_47_BinaryTemp.dat' ... '16_29_2...
    timestart              (time) datetime64[ns] 2009-09-23T22:56:46 ... 2009...
    timeend                (time) datetime64[ns] 2009-09-23T22:56:47 ... 2009...
  * time                   (time) datetime64[ns] 2009-09-23T22:56:47 ... 2009...
    acquisitiontimeFW      (time) timedelta64[ns] 00:00:01 00:00:01 00:00:01
  * trans_att              (trans_att) float64
Data variables:
    st                     (x, time) int32 39040680 39057147 ... 39071213
    ast                    (x, time) int32 39048646 39064414 ... 39407668
    tmp                    (x, time) float64 -273.1 -273.1 ... 82.41 82.71
    referenceTemperature   (time) float64 28.61 29.24 30.29
    st_zero                (time) float64 3.904e+07 3.906e+07 3.907e+07
    ast_zero               (time) float64 3.905e+07 3.907e+07 3.908e+07
    userAcquisitionTimeFW  (time) float64 1.0 1.0 1.0
Attributes: (12/16)
    survey_type:                 2
    hdr_version:                 3
    x_units:                     n/a
    y_units:                     counts
    num_points:                  12000
    num_pulses:                  25000
    ...                          ...
    hdr_size:                    176
    hw_config:                   84

.. and many more attributes. See: ds.attrs

xarray.DataStore

• Dimensions:
  • x: 11582
  • time: 3
  • trans_att: 0
• Coordinates: (8)
  • x
    (x)
    float32
    -451.4 -450.9 ... 5.409e+03

    name :

    distance

    description :

    Length along fiber

    long_description :

    Starting at connector of forward channel

    units :

    m

    array([-451.37958, -450.87354, -450.3675 , ..., 5407.952  , 5408.4585 ,
           5408.9644 ], dtype=float32)

  • filename
    (time)
    <U25
    '15_56_47_BinaryRawDTS.dat' ... ...

    array(['15_56_47_BinaryRawDTS.dat', '16_11_31_BinaryRawDTS.dat',
           '16_29_23_BinaryRawDTS.dat'], dtype='<U25')

  • filename_temp
    (time)
    <U23
    '15_56_47_BinaryTemp.dat' ... '1...

    array(['15_56_47_BinaryTemp.dat', '16_11_31_BinaryTemp.dat',
           '16_29_23_BinaryTemp.dat'], dtype='<U23')

  • timestart
    (time)
    datetime64[ns]
    2009-09-23T22:56:46 ... 2009-09-...

    description :

    time start of the measurement

    timezone :

    UTC

    array(['2009-09-23T22:56:46.000000000', '2009-09-23T23:11:30.000000000',
           '2009-09-23T23:29:22.000000000'], dtype='datetime64[ns]')

  • timeend
    (time)
    datetime64[ns]
    2009-09-23T22:56:47 ... 2009-09-...

    description :

    time end of the measurement

    timezone :

    UTC

    array(['2009-09-23T22:56:47.000000000', '2009-09-23T23:11:31.000000000',
           '2009-09-23T23:29:23.000000000'], dtype='datetime64[ns]')

  • time
    (time)
    datetime64[ns]
    2009-09-23T22:56:47 ... 2009-09-...

    description :

    time halfway the measurement

    timezone :

    UTC

    array(['2009-09-23T22:56:47.000000000', '2009-09-23T23:11:31.000000000',
           '2009-09-23T23:29:23.000000000'], dtype='datetime64[ns]')

  • acquisitiontimeFW
    (time)
    timedelta64[ns]
    00:00:01 00:00:01 00:00:01

    description :

    Acquisition time of the forward measurement

    array([1000000000, 1000000000, 1000000000], dtype='timedelta64[ns]')

  • trans_att
    (trans_att)
    float64

    name :

    Locations introducing transient directional differential attenuation

    description :

    Locations along the x-dimension introducing transient directional differential attenuation

    long_description :

    Connectors introduce additional differential attenuation that is different for the forward and backward direction, and varies over time.

    units :

    m

    array([], dtype=float64)

• Data variables: (7)
  • st
    (x, time)
    int32
    39040680 39057147 ... 39071213

    name :

    st

    description :

    Stokes intensity

    units :

    -

    array([[39040680, 39057147, 39067220],
           [39038580, 39053177, 39063543],
           [39038768, 39054349, 39064780],
           ...,
           [39155768, 39179638, 39196217],
           [39046316, 39063478, 39073966],
           [39046948, 39061160, 39071213]], dtype=int32)

  • ast
    (x, time)
    int32
    39048646 39064414 ... 39407668

    name :

    ast

    description :

    anti-Stokes intensity

    units :

    -

    array([[39048646, 39064414, 39074033],
           [39046719, 39060574, 39071003],
           [39046655, 39061723, 39072593],
           ...,
           [39304136, 39313172, 39321329],
           [39461032, 39474405, 39483689],
           [39362443, 39388893, 39407668]], dtype=int32)

  • tmp
    (x, time)
    float64
    -273.1 -273.1 ... 82.41 82.71

    name :

    tmp

    description :

    Temperature calibrated by device

    units :

    degC

    array([[-273.1499939 , -273.1499939 , -273.1499939 ],
           [-273.1499939 , -273.1499939 , -273.1499939 ],
           [-273.1499939 , -273.1499939 , -273.1499939 ],
           ...,
           [  51.81999969,   49.75999832,   48.02000046],
           [  74.16000366,   73.44999695,   72.34999847],
           [  80.91000366,   82.41000366,   82.70999908]])

  • referenceTemperature
    (time)
    float64
    28.61 29.24 30.29

    name :

    reference temperature

    description :

    Internal reference temperature

    units :

    $^\circ$C

    array([28.61147461, 29.23735962, 30.29247437])

  • st_zero
    (time)
    float64
    3.904e+07 3.906e+07 3.907e+07

    name :

    ST_zero

    description :

    Stokes zero count

    units :

    counts

    array([39042026.18660287, 39057430.34449761, 39067731.48325359])

  • ast_zero
    (time)
    float64
    3.905e+07 3.907e+07 3.908e+07

    name :

    AST_zero

    description :

    anit-Stokes zero count

    units :

    counts

    array([39050438.62200957, 39065503.        , 39075698.97607656])

  • userAcquisitionTimeFW
    (time)
    float64
    1.0 1.0 1.0

    name :

    userAcquisitionTimeFW

    description :

    Measurement duration of forward channel

    long_description :

    Desired measurement duration of forward channel

    units :

    seconds

    array([1., 1., 1.])

• Indexes: (3)
  • x
    PandasIndex

    PandasIndex(Index([-451.37957763671875,    -450.87353515625, -450.36749267578125,
            -449.8614807128906,  -449.3554382324219,  -448.8493957519531,
            -448.3433837890625, -447.83734130859375,  -447.3313293457031,
            -446.8252868652344,
           ...
                 5404.41015625,      5404.916015625,         5405.421875,
              5405.92822265625,    5406.43408203125,     5406.9404296875,
               5407.4462890625,     5407.9521484375,    5408.45849609375,
              5408.96435546875],
          dtype='float32', name='x', length=11582))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2009-09-23 22:56:47', '2009-09-23 23:11:31',
                   '2009-09-23 23:29:23'],
                  dtype='datetime64[ns]', name='time', freq=None))

  • trans_att
    PandasIndex

    PandasIndex(Index([], dtype='float64', name='trans_att'))

• Attributes: (16)

  survey_type :

  2

  hdr_version :

  3

  x_units :

  n/a

  y_units :

  counts

  num_points :

  12000

  num_pulses :

  25000

  channel_id :

  1

  num_subtraces :

  354

  num_skipped :

  0

  probe_name :

  walla1

  hdr_size :

  176

  hw_config :

  84

  isDoubleEnded :

  0

  forwardMeasurementChannel :

  0

  backwardMeasurementChannel :

  N/A

  _sections :

  null ...

ds0 = ds.isel(time=0)

plt.figure()
ds0.st.plot(label="Stokes signal")
plt.axhline(ds0.st_zero.values, c="r", label="'zero' measurement")
plt.legend()
plt.title("")
plt.axhline(c="k")

<matplotlib.lines.Line2D at 0x7f9c2bbc6910>

After a correction and rescaling (for human readability) the data will look more like other manufacturer’s devices

ds["st"] = (ds.st - ds.st_zero) / 1e4
ds["ast"] = (ds.ast - ds.ast_zero) / 1e4

ds.isel(time=0).st.plot(label="Stokes intensity")
ds.isel(time=0).ast.plot(label="anti-Stokes intensity")
plt.legend()
plt.axhline(c="k", lw=1)
plt.xlabel("")
plt.title("")
plt.ylim([-50, 500])

(-50.0, 500.0)