328 lines
13 KiB
ReStructuredText
328 lines
13 KiB
ReStructuredText
TimescaleDB for Time-Series Data
|
||
================================
|
||
|
||
`TimescaleDB <https://www.timescale.com?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=tutorial>`_ is an open-source database designed to make SQL scalable for time-series data. It is engineered up from PostgreSQL, providing automatic partitioning across time and space, while retaining the standard PostgreSQL interface.
|
||
|
||
PostgREST turns your PostgreSQL database directly into a RESTful API, since TimescaleDB is packaged as a PostgreSQL extension it works with PostgREST as well.
|
||
|
||
In this tutorial we'll explore some of TimescaleDB features through PostgREST.
|
||
|
||
Install Docker
|
||
--------------
|
||
|
||
For an easier setup we're going to use `Docker <https://www.docker.com/get-started>`_, make sure you have it installed.
|
||
|
||
Run TimescaleDB
|
||
---------------
|
||
|
||
First, let’s pull and start the `TimescaleDB container image <http://bit.ly/2SpxiYJ>`_:
|
||
|
||
.. code-block:: bash
|
||
|
||
docker run --name tsdb_tut \
|
||
-e POSTGRES_PASSWORD=mysecretpassword \
|
||
-p 5433:5432 \
|
||
-d timescale/timescaledb:latest-pg11
|
||
|
||
This will run the container as a daemon and expose port ``5433`` to the host system so that it doesn't conflict with another PostgreSQL installation.
|
||
|
||
Set up TimescaleDB
|
||
------------------
|
||
|
||
Now, we'll create the ``timescaledb`` extension in our database.
|
||
|
||
Run ``psql`` in the container we created in the previous step.
|
||
|
||
.. code-block:: bash
|
||
|
||
docker exec -it tsdb_tut psql -U postgres
|
||
## Run all the following commands inside psql
|
||
|
||
And create the extension:
|
||
|
||
.. code-block:: postgres
|
||
|
||
create extension if not exists timescaledb cascade;
|
||
|
||
Create an Hypertable
|
||
--------------------
|
||
|
||
`Hypertables <https://docs.timescale.com/latest/using-timescaledb/hypertables?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=hypertables>`_ are the core abstraction TimescaleDB offers for dealing with time-series data.
|
||
|
||
To create an ``hypertable``, first we need to create standard PostgreSQL tables:
|
||
|
||
.. code-block:: postgres
|
||
|
||
create table if not exists locations(
|
||
device_id text primary key
|
||
, location text
|
||
, environment text
|
||
);
|
||
|
||
create table if not exists conditions(
|
||
time timestamp with time zone not null
|
||
, device_id text references locations(device_id)
|
||
, temperature numeric
|
||
, humidity numeric
|
||
);
|
||
|
||
Now, we'll convert ``conditions`` into an hypertable with `create_hypertable <http://docs.timescale.com/latest/api?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=create-hypertable#create_hypertable>`_:
|
||
|
||
.. code-block:: postgres
|
||
|
||
SELECT create_hypertable('conditions', 'time', chunk_time_interval => interval '1 day');
|
||
-- This also implicitly creates an index: CREATE INDEX ON "conditions"(time DESC);
|
||
|
||
-- Exit psql
|
||
exit
|
||
|
||
|
||
Load sample data
|
||
----------------
|
||
|
||
To have some data to play with, we'll download the ``weather_small`` data set from `TimescaleDB's sample datasets <https://docs.timescale.com/latest/tutorials/other-sample-datasets?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=datasets>`_.
|
||
|
||
.. code-block:: bash
|
||
|
||
## Run bash inside the database container
|
||
docker exec -it tsdb_tut bash
|
||
|
||
## Download and uncompress the data
|
||
wget -qO- https://timescaledata.blob.core.windows.net/datasets/weather_small.tar.gz | tar xvz
|
||
|
||
## Copy data into the database
|
||
psql -U postgres <<EOF
|
||
\COPY locations FROM weather_small_locations.csv CSV
|
||
\COPY conditions FROM weather_small_conditions.csv CSV
|
||
EOF
|
||
|
||
## Exit bash
|
||
exit
|
||
|
||
Run PostgREST
|
||
-------------
|
||
|
||
For the last step in the setup, pull and start the official `PostgREST image <https://hub.docker.com/r/postgrest/postgrest/>`_:
|
||
|
||
.. code-block:: bash
|
||
|
||
docker run --rm -p 3000:3000 \
|
||
--name tsdb_pgrst \
|
||
--link tsdb_tut \
|
||
-e PGRST_DB_URI="postgres://postgres:mysecretpassword@tsdb_tut/postgres" \
|
||
-e PGRST_DB_ANON_ROLE="postgres" \
|
||
-d postgrest/postgrest:latest
|
||
|
||
PostgREST on Hypertables
|
||
------------------------
|
||
|
||
We'll now see how to read data from hypertables through PostgREST.
|
||
|
||
Since hypertables can be queried using standard `SELECT statements <https://docs.timescale.com/v1.2/using-timescaledb/reading-data?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=reading-data>`_, we can query them through PostgREST normally.
|
||
|
||
Suppose we want to run this query on ``conditions``:
|
||
|
||
.. code-block:: postgres
|
||
|
||
select
|
||
time,
|
||
device_id,
|
||
humidity
|
||
from conditions
|
||
where
|
||
humidity > 90 and
|
||
time < '2016-11-16'
|
||
order by time desc
|
||
limit 10;
|
||
|
||
Using PostgREST :ref:`horizontal <h_filter>`/:ref:`vertical <v_filter>` filtering, this query can be expressed as:
|
||
|
||
.. code-block:: bash
|
||
|
||
curl -G "localhost:3000/conditions" \
|
||
-d select=time,device_id,humidity \
|
||
-d humidity=gt.90 \
|
||
-d time=lt.2016-11-16 \
|
||
-d order=time.desc \
|
||
-d limit=10
|
||
## This command is equivalent to:
|
||
## curl "localhost:3000/conditions?select=time,device_id,humidity&humidity=gt.90&time=lt.2016-11-16&order=time.desc&limit=10"
|
||
## Here we used -G and -d to make the command more readable
|
||
|
||
The response will be:
|
||
|
||
.. code-block:: json
|
||
|
||
[{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000982","humidity":90.90000000000006},
|
||
{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000968","humidity":92.3},
|
||
{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000963","humidity":96.29999999999993},
|
||
{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000951","humidity":94.39999999999998},
|
||
{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000950","humidity":93.69999999999982},
|
||
{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000915","humidity":94.69999999999997},
|
||
{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000911","humidity":93.2000000000001},
|
||
{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000910","humidity":91.30000000000017},
|
||
{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000901","humidity":92.30000000000005},
|
||
{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000895","humidity":91.00000000000014}]
|
||
|
||
JOINs with relational tables
|
||
----------------------------
|
||
|
||
Hypertables support all standard `PostgreSQL constraints <https://docs.timescale.com/latest/using-timescaledb/schema-management?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=constraints#constraints>`_ . We can make use of the foreign key defined on ``locations`` to make a JOIN through PostgREST. A query such as:
|
||
|
||
.. code-block:: postgres
|
||
|
||
select
|
||
c.time,
|
||
c.temperature,
|
||
l.location,
|
||
l.environment
|
||
from conditions c
|
||
left join locations l on
|
||
c.device_id = l.device_id
|
||
order by time desc
|
||
limit 10;
|
||
|
||
Can be expressed in PostgREST by using :ref:`resource_embedding`.
|
||
|
||
.. code-block:: bash
|
||
|
||
curl -G localhost:3000/conditions \
|
||
-d select="time,temperature,device:locations(location,environment)" \
|
||
-d order=time.desc \
|
||
-d limit=10
|
||
|
||
.. code-block:: json
|
||
|
||
[{"time":"2016-11-16T21:18:00+00:00","temperature":69.49999999999991,"device":{"location":"office-000202","environment":"inside"}},
|
||
{"time":"2016-11-16T21:18:00+00:00","temperature":90,"device":{"location":"field-000205","environment":"outside"}},
|
||
{"time":"2016-11-16T21:18:00+00:00","temperature":60.499999999999986,"device":{"location":"door-00085","environment":"doorway"}},
|
||
{"time":"2016-11-16T21:18:00+00:00","temperature":91,"device":{"location":"swamp-000188","environment":"outside"}},
|
||
{"time":"2016-11-16T21:18:00+00:00","temperature":42,"device":{"location":"arctic-000219","environment":"outside"}},
|
||
{"time":"2016-11-16T21:18:00+00:00","temperature":70.80000000000003,"device":{"location":"office-000201","environment":"inside"}},
|
||
{"time":"2016-11-16T21:18:00+00:00","temperature":62.699999999999974,"device":{"location":"door-00084","environment":"doorway"}},
|
||
{"time":"2016-11-16T21:18:00+00:00","temperature":85.49999999999918,"device":{"location":"field-000204","environment":"outside"}},
|
||
{"time":"2016-11-16T21:18:00+00:00","temperature":42,"device":{"location":"arctic-000218","environment":"outside"}},
|
||
{"time":"2016-11-16T21:18:00+00:00","temperature":42,"device":{"location":"arctic-000217","environment":"outside"}}]
|
||
|
||
Time-Oriented Analytics
|
||
-----------------------
|
||
|
||
TimescaleDB includes new aggregate functions for time-oriented `analytics <https://docs.timescale.com/latest/api?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=analytics#analytics>`_.
|
||
|
||
For using aggregate queries with PostgREST you must create VIEWs or :ref:`s_procs`. Here's an example for using `time_bucket <https://docs.timescale.com/latest/api?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=time-bucket#time_bucket>`_:
|
||
|
||
.. code-block:: postgres
|
||
|
||
-- Run psql in the database container
|
||
docker exec -it tsdb_tut psql -U postgres
|
||
|
||
-- Create the function
|
||
create or replace function temperature_summaries(gap interval default '1 hour', prefix text default 'field')
|
||
returns table(hour text, avg_temp numeric, min_temp numeric, max_temp numeric) as $$
|
||
select
|
||
time_bucket(gap, time)::text as hour,
|
||
trunc(avg(temperature), 2),
|
||
trunc(min(temperature), 2),
|
||
trunc(max(temperature), 2)
|
||
from conditions c
|
||
where c.device_id in (
|
||
select device_id from locations
|
||
where location like prefix || '-%')
|
||
group by hour
|
||
$$ language sql stable;
|
||
|
||
-- Exit psql
|
||
exit
|
||
|
||
Every time the schema is changed you must reload PostgREST :ref:`schema cache <schema_reloading>` so it can pick up the function parameters correctly. To reload, run:
|
||
|
||
.. code-block:: bash
|
||
|
||
docker kill --signal=USR1 tsdb_pgrst
|
||
|
||
|
||
Now, since the function is ``stable``, we can call it with ``GET`` as:
|
||
|
||
.. code-block:: bash
|
||
|
||
curl -G "localhost:3000/rpc/temperature_summaries" \
|
||
-d gap=2minutes \
|
||
-d order=hour.asc \
|
||
-d limit=10 \
|
||
-H "Accept: text/csv"
|
||
## time_bucket accepts an interval type as it's argument
|
||
## so you can pass gap=5minutes or gap=5hours
|
||
|
||
.. code-block:: sql
|
||
|
||
hour,avg_temp,min_temp,max_temp
|
||
"2016-11-15 12:00:00+00",72.97,68.00,78.00
|
||
"2016-11-15 12:02:00+00",73.01,68.00,78.00
|
||
"2016-11-15 12:04:00+00",73.05,68.00,78.10
|
||
"2016-11-15 12:06:00+00",73.07,68.00,78.10
|
||
"2016-11-15 12:08:00+00",73.11,68.00,78.10
|
||
"2016-11-15 12:10:00+00",73.14,68.00,78.10
|
||
"2016-11-15 12:12:00+00",73.17,68.00,78.19
|
||
"2016-11-15 12:14:00+00",73.21,68.10,78.19
|
||
"2016-11-15 12:16:00+00",73.24,68.10,78.29
|
||
"2016-11-15 12:18:00+00",73.27,68.10,78.39
|
||
|
||
Note you can use PostgREST standard filtering on function results. Here we also changed the :ref:`res_format` to CSV.
|
||
|
||
Fast Ingestion with Bulk Insert
|
||
-------------------------------
|
||
|
||
You can use PostgREST :ref:`bulk_insert` to leverage TimescaleDB `fast ingestion <https://docs.timescale.com/latest/introduction/timescaledb-vs-postgres?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=fast-ingest>`_.
|
||
|
||
Let's do an insert of three rows:
|
||
|
||
.. code-block:: bash
|
||
|
||
curl "localhost:3000/conditions" \
|
||
-H "Content-Type: application/json" \
|
||
-H "Prefer: return=representation" \
|
||
-d @- << EOF
|
||
[
|
||
{"time": "2019-02-21 01:00:01-05", "device_id": "weather-pro-000000", "temperature": 40.0, "humidity": 59.9},
|
||
{"time": "2019-02-21 01:00:02-05", "device_id": "weather-pro-000000", "temperature": 42.0, "humidity": 69.9},
|
||
{"time": "2019-02-21 01:00:03-05", "device_id": "weather-pro-000000", "temperature": 44.0, "humidity": 79.9}
|
||
]
|
||
EOF
|
||
|
||
By using the ``Prefer: return=representation`` header we can see the successfully inserted rows:
|
||
|
||
.. code-block:: json
|
||
|
||
[{"time":"2019-02-21T06:00:01+00:00","device_id":"weather-pro-000000","temperature":40.0,"humidity":59.9},
|
||
{"time":"2019-02-21T06:00:02+00:00","device_id":"weather-pro-000000","temperature":42.0,"humidity":69.9},
|
||
{"time":"2019-02-21T06:00:03+00:00","device_id":"weather-pro-000000","temperature":44.0,"humidity":79.9}]
|
||
|
||
Let's now insert a thousand rows, we'll use `jq <https://stedolan.github.io/jq/>`_ for constructing the array.
|
||
|
||
.. code-block:: bash
|
||
|
||
yes "{\"time\": \"$(date +'%F %T')\", \"device_id\": \"weather-pro-000001\", \"temperature\": 50, \"humidity\": 60}" | \
|
||
head -n 1000 | jq -s '.' | \
|
||
curl -i -d @- "http://localhost:3000/conditions" \
|
||
-H "Content-Type: application/json" \
|
||
-H "Prefer: count=exact"
|
||
|
||
With ``Prefer: count=exact`` we can know how many rows were inserted. Check out the response:
|
||
|
||
.. code-block:: haskell
|
||
|
||
HTTP/1.1 201 Created
|
||
Transfer-Encoding: chunked
|
||
Date: Fri, 22 Feb 2019 16:47:05 GMT
|
||
Server: postgrest/5.2.0 (9969262)
|
||
Content-Range: */1000
|
||
|
||
You can see in ``Content-Range`` that the total number of inserted rows is ``1000``.
|
||
|
||
Summing it up
|
||
-------------
|
||
|
||
There you have it, with PostgREST you can get an instant and performant RESTful API for a TimescaleDB database.
|
||
|
||
For a more in depth exploration of TimescaleDB capabilities, check their `docs <https://docs.timescale.com?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=docs-tutorial>`_.
|