{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9. Import a time series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we are adding a timeseries to the DataStore object. This might be useful if the temperature in one of the calibration baths was measured with an external device. It requires three steps to add the measurement files to the DataStore object:\n",
    "1. Load the measurement files (e.g., csv, txt) with pandas into a pandas.Series object\n",
    "2. Add the pandas.Series object to the DataStore\n",
    "3. Align the time to that of the DTS measurement (required for calibration)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-20T09:16:48.371460Z",
     "iopub.status.busy": "2023-10-20T09:16:48.371128Z",
     "iopub.status.idle": "2023-10-20T09:16:49.350990Z",
     "shell.execute_reply": "2023-10-20T09:16:49.350668Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "from dtscalibration import read_silixa_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: load the measurement files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-20T09:16:49.353033Z",
     "iopub.status.busy": "2023-10-20T09:16:49.352815Z",
     "iopub.status.idle": "2023-10-20T09:16:49.355625Z",
     "shell.execute_reply": "2023-10-20T09:16:49.355358Z"
    }
   },
   "outputs": [],
   "source": [
    "filepath = os.path.join(\n",
    "    \"..\",\n",
    "    \"..\",\n",
    "    \"tests\",\n",
    "    \"data\",\n",
    "    \"external_temperature_timeseries\",\n",
    "    \"Loodswaternet2018-03-28 02h.csv\",\n",
    ")\n",
    "\n",
    "# Bonus:\n",
    "print(filepath, \"\\n\")\n",
    "with open(filepath, \"r\") as f:\n",
    "    head = [next(f) for _ in range(5)]\n",
    "print(\" \".join(head))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-20T09:16:49.375672Z",
     "iopub.status.busy": "2023-10-20T09:16:49.375532Z",
     "iopub.status.idle": "2023-10-20T09:16:49.399183Z",
     "shell.execute_reply": "2023-10-20T09:16:49.398877Z"
    }
   },
   "outputs": [],
   "source": [
    "ts = pd.read_csv(filepath, sep=\",\", index_col=0, parse_dates=True)[\n",
    "    \"Pt100 2\"\n",
    "]  # See pandas' read_csv documentation for more options\n",
    "ts = ts.tz_localize(\"Europe/Amsterdam\")  # Set the timezone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-20T09:16:49.401036Z",
     "iopub.status.busy": "2023-10-20T09:16:49.400906Z",
     "iopub.status.idle": "2023-10-20T09:16:49.405335Z",
     "shell.execute_reply": "2023-10-20T09:16:49.405054Z"
    }
   },
   "outputs": [],
   "source": [
    "ts.head()  # Double check the timezone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we quickly create a DataStore from xml-files with Stokes measurements to add the external timeseries to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-20T09:16:49.407001Z",
     "iopub.status.busy": "2023-10-20T09:16:49.406895Z",
     "iopub.status.idle": "2023-10-20T09:16:49.587342Z",
     "shell.execute_reply": "2023-10-20T09:16:49.587037Z"
    }
   },
   "outputs": [],
   "source": [
    "filepath_ds = os.path.join(\"..\", \"..\", \"tests\", \"data\", \"double_ended2\")\n",
    "ds = read_silixa_files(directory=filepath_ds, timezone_netcdf=\"UTC\", file_ext=\"*.xml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Add the temperature measurements of the external probe to the DataStore.\n",
    "\n",
    "First add the coordinates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-20T09:16:49.589047Z",
     "iopub.status.busy": "2023-10-20T09:16:49.588934Z",
     "iopub.status.idle": "2023-10-20T09:16:49.591317Z",
     "shell.execute_reply": "2023-10-20T09:16:49.591061Z"
    }
   },
   "outputs": [],
   "source": [
    "ds.coords[\"time_external\"] = ts.index.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second we add the measured values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-20T09:16:49.592925Z",
     "iopub.status.busy": "2023-10-20T09:16:49.592798Z",
     "iopub.status.idle": "2023-10-20T09:16:49.594944Z",
     "shell.execute_reply": "2023-10-20T09:16:49.594703Z"
    }
   },
   "outputs": [],
   "source": [
    "ds[\"external_probe\"] = ((\"time_external\",), ts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Align the time of the external measurements to the Stokes measurement times\n",
    "We linearly interpolate the measurements of the external sensor to the times we have DTS measurements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-20T09:16:49.596483Z",
     "iopub.status.busy": "2023-10-20T09:16:49.596379Z",
     "iopub.status.idle": "2023-10-20T09:16:49.716179Z",
     "shell.execute_reply": "2023-10-20T09:16:49.715795Z"
    }
   },
   "outputs": [],
   "source": [
    "ds[\"external_probe_dts\"] = ds[\"external_probe\"].interp(time_external=ds.time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-20T09:16:49.718112Z",
     "iopub.status.busy": "2023-10-20T09:16:49.717968Z",
     "iopub.status.idle": "2023-10-20T09:16:49.722423Z",
     "shell.execute_reply": "2023-10-20T09:16:49.721693Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(ds.data_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use `external_probe_dts` when we define sections and use it for calibration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}