Add TimescaleDB tutorial (#201)

* Add TimescaleDB sponsorship
2019-03-01 16:18:20 -05:00
parent 44b825b939
commit 1c357179fd
4 changed files with 352 additions and 0 deletions
@@ -366,6 +366,8 @@ Note that the larger the table the slower this query runs in the database. The s
  Range-Unit: items
  Content-Range: 0-24/3573458

+.. _res_format:
+
 Response Format
 ---------------

@@ -577,6 +579,9 @@ Embedded resources can be aliased and filters can be applied on these aliases:

  GET /films?select=*,90_comps:competitions(name),91_comps:competitions(name)&90_comps.year=eq.1990&91_comps.year=eq.1991 HTTP/1.1

+
+.. _custom_queries:
+
 Custom Queries
 ==============

@@ -587,6 +592,8 @@ The PostgREST URL grammar limits the kinds of queries clients can perform. It pr
 * Geo-spatial queries that require an argument, like "points near (lat,lon)"
 * More sophisticated full-text search than a simple use of the :sql:`fts` filter

+.. _s_procs:
+
 Stored Procedures
 =================

@@ -942,6 +949,8 @@ Updates also support :code:`Prefer: return=representation` plus :ref:`v_filter`.

  Beware of accidentally updating every row in a table. To learn to prevent that see :ref:`block_fulltable`.

+.. _bulk_insert:
+
 Bulk Insert
 -----------

@@ -23,6 +23,16 @@
 |
 PostgREST is a standalone web server that turns your PostgreSQL database directly into a RESTful API. The structural constraints and permissions in the database determine the API endpoints and operations.

+Sponsors
+--------
+
+.. image:: _static/timescaledb.png
+  :target: https://www.timescale.com?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=docs
+  :width:  222px
+  :align:  center
+
+`TimescaleDB <https://www.timescale.com?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=docs>`_ is an scalable time-series database packaged as a PostgreSQL extension. See our tutorial for using `TimescaleDB with PostgREST <integrations/timescaledb.html>`_.
+
 Motivation
 ----------

@@ -89,6 +99,12 @@ Translations
   tutorials/tut0.rst
   tutorials/tut1.rst

+.. toctree::
+   :caption: Integrations
+   :titlesonly:
+
+   integrations/timescaledb.rst
+
 .. toctree::
   :caption: Installation
   :titlesonly:
@@ -0,0 +1,327 @@
+TimescaleDB for Time-Series Data
+================================
+
+`TimescaleDB <https://www.timescale.com?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=tutorial>`_ is an open-source database designed to make SQL scalable for time-series data. It is engineered up from PostgreSQL, providing automatic partitioning across time and space, while retaining the standard PostgreSQL interface.
+
+PostgREST turns your PostgreSQL database directly into a RESTful API, since TimescaleDB is packaged as a PostgreSQL extension it works with PostgREST as well.
+
+In this tutorial we'll explore some of TimescaleDB features through PostgREST.
+
+Install Docker
+--------------
+
+For an easier setup we're going to use `Docker <https://www.docker.com/get-started>`_, make sure you have it installed.
+
+Run TimescaleDB
+---------------
+
+First, let’s pull and start the `TimescaleDB container image <http://bit.ly/2SpxiYJ>`_:
+
+.. code-block:: bash
+
+  docker run --name tsdb_tut \
+    -e POSTGRES_PASSWORD=mysecretpassword \
+    -p 5433:5432 \
+    -d timescale/timescaledb:latest-pg11
+
+This will run the container as a daemon and expose port ``5433`` to the host system so that it doesn't conflict with another PostgreSQL installation.
+
+Set up TimescaleDB
+------------------
+
+Now, we'll create the ``timescaledb`` extension in our database.
+
+Run ``psql`` in the container we created in the previous step.
+
+.. code-block:: bash
+
+  docker exec -it tsdb_tut psql -U postgres
+  ## Run all the following commands inside psql
+
+And create the extension:
+
+.. code-block:: postgres
+
+  create extension if not exists timescaledb cascade;
+
+Create an Hypertable
+--------------------
+
+`Hypertables <https://docs.timescale.com/latest/using-timescaledb/hypertables?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=hypertables>`_ are the core abstraction TimescaleDB offers for dealing with time-series data.
+
+To create an ``hypertable``, first we need to create standard PostgreSQL tables:
+
+.. code-block:: postgres
+
+  create table if not exists locations(
+    device_id    text  primary key
+  , location     text
+  , environment  text
+  );
+
+  create table if not exists conditions(
+    time         timestamp with time zone  not null
+  , device_id    text                      references locations(device_id)
+  , temperature  numeric
+  , humidity     numeric
+  );
+
+Now, we'll convert ``conditions`` into an hypertable with `create_hypertable <http://docs.timescale.com/latest/api?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=create-hypertable#create_hypertable>`_:
+
+.. code-block:: postgres
+
+  SELECT create_hypertable('conditions', 'time', chunk_time_interval => interval '1 day');
+  -- This also implicitly creates an index: CREATE INDEX ON "conditions"(time DESC);
+
+  -- Exit psql
+  exit
+
+
+Load sample data
+----------------
+
+To have some data to play with, we'll download the ``weather_small`` data set from `TimescaleDB's sample datasets <https://docs.timescale.com/latest/tutorials/other-sample-datasets?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=datasets>`_.
+
+.. code-block:: bash
+
+  ## Run bash inside the database container
+  docker exec -it tsdb_tut bash
+
+  ## Download and uncompress the data
+  wget -qO- https://timescaledata.blob.core.windows.net/datasets/weather_small.tar.gz | tar xvz
+
+  ## Copy data into the database
+  psql -U postgres <<EOF
+    \COPY locations  FROM weather_small_locations.csv  CSV
+    \COPY conditions FROM weather_small_conditions.csv CSV
+  EOF
+
+  ## Exit bash
+  exit
+
+Run PostgREST
+-------------
+
+For the last step in the setup, pull and start the official `PostgREST image <https://hub.docker.com/r/postgrest/postgrest/>`_:
+
+.. code-block:: bash
+
+  docker run --rm -p 3000:3000 \
+    --name tsdb_pgrst \
+    --link tsdb_tut \
+    -e PGRST_DB_URI="postgres://postgres:mysecretpassword@tsdb_tut/postgres" \
+    -e PGRST_DB_ANON_ROLE="postgres" \
+    -d postgrest/postgrest:latest
+
+PostgREST on Hypertables
+------------------------
+
+We'll now see how to read data from hypertables through PostgREST.
+
+Since hypertables can be queried using standard `SELECT statements <https://docs.timescale.com/v1.2/using-timescaledb/reading-data?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=reading-data>`_, we can query them through PostgREST normally.
+
+Suppose we want to run this query on ``conditions``:
+
+.. code-block:: postgres
+
+  select
+    time,
+    device_id,
+    humidity
+  from conditions
+  where
+    humidity > 90           and
+    time     < '2016-11-16'
+  order by time desc
+  limit 10;
+
+Using PostgREST :ref:`horizontal <h_filter>`/:ref:`vertical <v_filter>` filtering, this query can be expressed as:
+
+.. code-block:: bash
+
+  curl -G "localhost:3000/conditions" \
+    -d select=time,device_id,humidity \
+    -d humidity=gt.90 \
+    -d time=lt.2016-11-16 \
+    -d order=time.desc \
+    -d limit=10
+  ## This command is equivalent to:
+  ## curl "localhost:3000/conditions?select=time,device_id,humidity&humidity=gt.90&time=lt.2016-11-16&order=time.desc&limit=10"
+  ## Here we used -G and -d to make the command more readable
+
+The response will be:
+
+.. code-block:: json
+
+  [{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000982","humidity":90.90000000000006},
+   {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000968","humidity":92.3},
+   {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000963","humidity":96.29999999999993},
+   {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000951","humidity":94.39999999999998},
+   {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000950","humidity":93.69999999999982},
+   {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000915","humidity":94.69999999999997},
+   {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000911","humidity":93.2000000000001},
+   {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000910","humidity":91.30000000000017},
+   {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000901","humidity":92.30000000000005},
+   {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000895","humidity":91.00000000000014}]
+
+JOINs with relational tables
+----------------------------
+
+Hypertables support all standard `PostgreSQL constraints <https://docs.timescale.com/latest/using-timescaledb/schema-management?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=constraints#constraints>`_ . We can make use of the foreign key defined on ``locations`` to make a JOIN through PostgREST. A query such as:
+
+.. code-block:: postgres
+
+  select
+    c.time,
+    c.temperature,
+    l.location,
+    l.environment
+  from conditions c
+  left join locations l on
+    c.device_id = l.device_id
+  order by time desc
+  limit 10;
+
+Can be expressed in PostgREST by using :ref:`resource_embedding`.
+
+.. code-block:: bash
+
+  curl -G localhost:3000/conditions \
+    -d select="time,temperature,device:locations(location,environment)" \
+    -d order=time.desc \
+    -d limit=10
+
+.. code-block:: json
+
+  [{"time":"2016-11-16T21:18:00+00:00","temperature":69.49999999999991,"device":{"location":"office-000202","environment":"inside"}},
+   {"time":"2016-11-16T21:18:00+00:00","temperature":90,"device":{"location":"field-000205","environment":"outside"}},
+   {"time":"2016-11-16T21:18:00+00:00","temperature":60.499999999999986,"device":{"location":"door-00085","environment":"doorway"}},
+   {"time":"2016-11-16T21:18:00+00:00","temperature":91,"device":{"location":"swamp-000188","environment":"outside"}},
+   {"time":"2016-11-16T21:18:00+00:00","temperature":42,"device":{"location":"arctic-000219","environment":"outside"}},
+   {"time":"2016-11-16T21:18:00+00:00","temperature":70.80000000000003,"device":{"location":"office-000201","environment":"inside"}},
+   {"time":"2016-11-16T21:18:00+00:00","temperature":62.699999999999974,"device":{"location":"door-00084","environment":"doorway"}},
+   {"time":"2016-11-16T21:18:00+00:00","temperature":85.49999999999918,"device":{"location":"field-000204","environment":"outside"}},
+   {"time":"2016-11-16T21:18:00+00:00","temperature":42,"device":{"location":"arctic-000218","environment":"outside"}},
+   {"time":"2016-11-16T21:18:00+00:00","temperature":42,"device":{"location":"arctic-000217","environment":"outside"}}]
+
+Time-Oriented Analytics
+-----------------------
+
+TimescaleDB includes new aggregate functions for time-oriented `analytics <https://docs.timescale.com/latest/api?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=analytics#analytics>`_.
+
+For using aggregate queries with PostgREST you must create VIEWs or :ref:`s_procs`. Here's an example for using `time_bucket <https://docs.timescale.com/latest/api?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=time-bucket#time_bucket>`_:
+
+.. code-block:: postgres
+
+  -- Run psql in the database container
+  docker exec -it tsdb_tut psql -U postgres
+
+  -- Create the function
+  create or replace function temperature_summaries(gap interval default '1 hour', prefix text default 'field')
+  returns table(hour text, avg_temp numeric, min_temp numeric, max_temp numeric) as $$
+    select
+      time_bucket(gap, time)::text as hour,
+      trunc(avg(temperature), 2),
+      trunc(min(temperature), 2),
+      trunc(max(temperature), 2)
+    from conditions c
+    where c.device_id in (
+      select device_id from locations
+      where location like prefix || '-%')
+    group by hour
+  $$ language sql stable;
+
+  -- Exit psql
+  exit
+
+Every time the schema is changed you must reload PostgREST :ref:`schema cache <schema_reloading>` so it can pick up the function parameters correctly. To reload, run:
+
+.. code-block:: bash
+
+  docker kill --signal=USR1 tsdb_pgrst
+
+
+Now, since the function is ``stable``, we can call it with ``GET`` as:
+
+.. code-block:: bash
+
+  curl -G "localhost:3000/rpc/temperature_summaries" \
+    -d gap=2minutes \
+    -d order=hour.asc \
+    -d limit=10 \
+    -H "Accept: text/csv"
+  ## time_bucket accepts an interval type as it's argument
+  ## so you can pass gap=5minutes or gap=5hours
+
+.. code-block:: sql
+
+  hour,avg_temp,min_temp,max_temp
+  "2016-11-15 12:00:00+00",72.97,68.00,78.00
+  "2016-11-15 12:02:00+00",73.01,68.00,78.00
+  "2016-11-15 12:04:00+00",73.05,68.00,78.10
+  "2016-11-15 12:06:00+00",73.07,68.00,78.10
+  "2016-11-15 12:08:00+00",73.11,68.00,78.10
+  "2016-11-15 12:10:00+00",73.14,68.00,78.10
+  "2016-11-15 12:12:00+00",73.17,68.00,78.19
+  "2016-11-15 12:14:00+00",73.21,68.10,78.19
+  "2016-11-15 12:16:00+00",73.24,68.10,78.29
+  "2016-11-15 12:18:00+00",73.27,68.10,78.39
+
+Note you can use PostgREST standard filtering on function results. Here we also changed the :ref:`res_format` to CSV.
+
+Fast Ingestion with Bulk Insert
+-------------------------------
+
+You can use PostgREST :ref:`bulk_insert` to leverage TimescaleDB `fast ingestion <https://docs.timescale.com/latest/introduction/timescaledb-vs-postgres?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=fast-ingest>`_.
+
+Let's do an insert of three rows:
+
+.. code-block:: bash
+
+  curl "localhost:3000/conditions" \
+    -H "Content-Type: application/json" \
+    -H "Prefer: return=representation" \
+    -d @- << EOF
+    [
+      {"time": "2019-02-21 01:00:01-05", "device_id": "weather-pro-000000", "temperature": 40.0, "humidity": 59.9},
+      {"time": "2019-02-21 01:00:02-05", "device_id": "weather-pro-000000", "temperature": 42.0, "humidity": 69.9},
+      {"time": "2019-02-21 01:00:03-05", "device_id": "weather-pro-000000", "temperature": 44.0, "humidity": 79.9}
+    ]
+  EOF
+
+By using the ``Prefer: return=representation`` header we can see the successfully inserted rows:
+
+.. code-block:: json
+
+  [{"time":"2019-02-21T06:00:01+00:00","device_id":"weather-pro-000000","temperature":40.0,"humidity":59.9},
+   {"time":"2019-02-21T06:00:02+00:00","device_id":"weather-pro-000000","temperature":42.0,"humidity":69.9},
+   {"time":"2019-02-21T06:00:03+00:00","device_id":"weather-pro-000000","temperature":44.0,"humidity":79.9}]
+
+Let's now insert a thousand rows, we'll use `jq <https://stedolan.github.io/jq/>`_ for constructing the array.
+
+.. code-block:: bash
+
+  yes "{\"time\": \"$(date +'%F %T')\", \"device_id\": \"weather-pro-000001\", \"temperature\": 50, \"humidity\": 60}" | \
+  head -n 1000 | jq -s '.' | \
+  curl -i -d @- "http://localhost:3000/conditions" \
+    -H "Content-Type: application/json" \
+    -H "Prefer: count=exact"
+
+With ``Prefer: count=exact`` we can know how many rows were inserted. Check out the response:
+
+.. code-block:: haskell
+
+  HTTP/1.1 201 Created
+  Transfer-Encoding: chunked
+  Date: Fri, 22 Feb 2019 16:47:05 GMT
+  Server: postgrest/5.2.0 (9969262)
+  Content-Range: */1000
+
+You can see in ``Content-Range`` that the total number of inserted rows is ``1000``.
+
+Summing it up
+-------------
+
+There you have it, with PostgREST you can get an instant and performant RESTful API for a TimescaleDB database.
+
+For a more in depth exploration of TimescaleDB capabilities, check their `docs <https://docs.timescale.com?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=docs-tutorial>`_.