diff --git a/_static/timescaledb.png b/_static/timescaledb.png new file mode 100644 index 000000000..d6403efa4 Binary files /dev/null and b/_static/timescaledb.png differ diff --git a/api.rst b/api.rst index 722c79000..398c37b96 100644 --- a/api.rst +++ b/api.rst @@ -366,6 +366,8 @@ Note that the larger the table the slower this query runs in the database. The s Range-Unit: items Content-Range: 0-24/3573458 +.. _res_format: + Response Format --------------- @@ -577,6 +579,9 @@ Embedded resources can be aliased and filters can be applied on these aliases: GET /films?select=*,90_comps:competitions(name),91_comps:competitions(name)&90_comps.year=eq.1990&91_comps.year=eq.1991 HTTP/1.1 + +.. _custom_queries: + Custom Queries ============== @@ -587,6 +592,8 @@ The PostgREST URL grammar limits the kinds of queries clients can perform. It pr * Geo-spatial queries that require an argument, like "points near (lat,lon)" * More sophisticated full-text search than a simple use of the :sql:`fts` filter +.. _s_procs: + Stored Procedures ================= @@ -942,6 +949,8 @@ Updates also support :code:`Prefer: return=representation` plus :ref:`v_filter`. Beware of accidentally updating every row in a table. To learn to prevent that see :ref:`block_fulltable`. +.. _bulk_insert: + Bulk Insert ----------- diff --git a/index.rst b/index.rst index faec2ae8f..ee17eb075 100644 --- a/index.rst +++ b/index.rst @@ -23,6 +23,16 @@ | PostgREST is a standalone web server that turns your PostgreSQL database directly into a RESTful API. The structural constraints and permissions in the database determine the API endpoints and operations. +Sponsors +-------- + +.. image:: _static/timescaledb.png + :target: https://www.timescale.com?utm_campaign=postgrest&utm_source=sponsor&utm_medium=referral&utm_content=docs + :width: 222px + :align: center + +`TimescaleDB `_ is an scalable time-series database packaged as a PostgreSQL extension. See our tutorial for using `TimescaleDB with PostgREST `_. + Motivation ---------- @@ -89,6 +99,12 @@ Translations tutorials/tut0.rst tutorials/tut1.rst +.. toctree:: + :caption: Integrations + :titlesonly: + + integrations/timescaledb.rst + .. toctree:: :caption: Installation :titlesonly: diff --git a/integrations/timescaledb.rst b/integrations/timescaledb.rst new file mode 100644 index 000000000..ea372a507 --- /dev/null +++ b/integrations/timescaledb.rst @@ -0,0 +1,327 @@ +TimescaleDB for Time-Series Data +================================ + +`TimescaleDB `_ is an open-source database designed to make SQL scalable for time-series data. It is engineered up from PostgreSQL, providing automatic partitioning across time and space, while retaining the standard PostgreSQL interface. + +PostgREST turns your PostgreSQL database directly into a RESTful API, since TimescaleDB is packaged as a PostgreSQL extension it works with PostgREST as well. + +In this tutorial we'll explore some of TimescaleDB features through PostgREST. + +Install Docker +-------------- + +For an easier setup we're going to use `Docker `_, make sure you have it installed. + +Run TimescaleDB +--------------- + +First, let’s pull and start the `TimescaleDB container image `_: + +.. code-block:: bash + + docker run --name tsdb_tut \ + -e POSTGRES_PASSWORD=mysecretpassword \ + -p 5433:5432 \ + -d timescale/timescaledb:latest-pg11 + +This will run the container as a daemon and expose port ``5433`` to the host system so that it doesn't conflict with another PostgreSQL installation. + +Set up TimescaleDB +------------------ + +Now, we'll create the ``timescaledb`` extension in our database. + +Run ``psql`` in the container we created in the previous step. + +.. code-block:: bash + + docker exec -it tsdb_tut psql -U postgres + ## Run all the following commands inside psql + +And create the extension: + +.. code-block:: postgres + + create extension if not exists timescaledb cascade; + +Create an Hypertable +-------------------- + +`Hypertables `_ are the core abstraction TimescaleDB offers for dealing with time-series data. + +To create an ``hypertable``, first we need to create standard PostgreSQL tables: + +.. code-block:: postgres + + create table if not exists locations( + device_id text primary key + , location text + , environment text + ); + + create table if not exists conditions( + time timestamp with time zone not null + , device_id text references locations(device_id) + , temperature numeric + , humidity numeric + ); + +Now, we'll convert ``conditions`` into an hypertable with `create_hypertable `_: + +.. code-block:: postgres + + SELECT create_hypertable('conditions', 'time', chunk_time_interval => interval '1 day'); + -- This also implicitly creates an index: CREATE INDEX ON "conditions"(time DESC); + + -- Exit psql + exit + + +Load sample data +---------------- + +To have some data to play with, we'll download the ``weather_small`` data set from `TimescaleDB's sample datasets `_. + +.. code-block:: bash + + ## Run bash inside the database container + docker exec -it tsdb_tut bash + + ## Download and uncompress the data + wget -qO- https://timescaledata.blob.core.windows.net/datasets/weather_small.tar.gz | tar xvz + + ## Copy data into the database + psql -U postgres <`_: + +.. code-block:: bash + + docker run --rm -p 3000:3000 \ + --name tsdb_pgrst \ + --link tsdb_tut \ + -e PGRST_DB_URI="postgres://postgres:mysecretpassword@tsdb_tut/postgres" \ + -e PGRST_DB_ANON_ROLE="postgres" \ + -d postgrest/postgrest:latest + +PostgREST on Hypertables +------------------------ + +We'll now see how to read data from hypertables through PostgREST. + +Since hypertables can be queried using standard `SELECT statements `_, we can query them through PostgREST normally. + +Suppose we want to run this query on ``conditions``: + +.. code-block:: postgres + + select + time, + device_id, + humidity + from conditions + where + humidity > 90 and + time < '2016-11-16' + order by time desc + limit 10; + +Using PostgREST :ref:`horizontal `/:ref:`vertical ` filtering, this query can be expressed as: + +.. code-block:: bash + + curl -G "localhost:3000/conditions" \ + -d select=time,device_id,humidity \ + -d humidity=gt.90 \ + -d time=lt.2016-11-16 \ + -d order=time.desc \ + -d limit=10 + ## This command is equivalent to: + ## curl "localhost:3000/conditions?select=time,device_id,humidity&humidity=gt.90&time=lt.2016-11-16&order=time.desc&limit=10" + ## Here we used -G and -d to make the command more readable + +The response will be: + +.. code-block:: json + + [{"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000982","humidity":90.90000000000006}, + {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000968","humidity":92.3}, + {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000963","humidity":96.29999999999993}, + {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000951","humidity":94.39999999999998}, + {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000950","humidity":93.69999999999982}, + {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000915","humidity":94.69999999999997}, + {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000911","humidity":93.2000000000001}, + {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000910","humidity":91.30000000000017}, + {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000901","humidity":92.30000000000005}, + {"time":"2016-11-15T23:58:00+00:00","device_id":"weather-pro-000895","humidity":91.00000000000014}] + +JOINs with relational tables +---------------------------- + +Hypertables support all standard `PostgreSQL constraints `_ . We can make use of the foreign key defined on ``locations`` to make a JOIN through PostgREST. A query such as: + +.. code-block:: postgres + + select + c.time, + c.temperature, + l.location, + l.environment + from conditions c + left join locations l on + c.device_id = l.device_id + order by time desc + limit 10; + +Can be expressed in PostgREST by using :ref:`resource_embedding`. + +.. code-block:: bash + + curl -G localhost:3000/conditions \ + -d select="time,temperature,device:locations(location,environment)" \ + -d order=time.desc \ + -d limit=10 + +.. code-block:: json + + [{"time":"2016-11-16T21:18:00+00:00","temperature":69.49999999999991,"device":{"location":"office-000202","environment":"inside"}}, + {"time":"2016-11-16T21:18:00+00:00","temperature":90,"device":{"location":"field-000205","environment":"outside"}}, + {"time":"2016-11-16T21:18:00+00:00","temperature":60.499999999999986,"device":{"location":"door-00085","environment":"doorway"}}, + {"time":"2016-11-16T21:18:00+00:00","temperature":91,"device":{"location":"swamp-000188","environment":"outside"}}, + {"time":"2016-11-16T21:18:00+00:00","temperature":42,"device":{"location":"arctic-000219","environment":"outside"}}, + {"time":"2016-11-16T21:18:00+00:00","temperature":70.80000000000003,"device":{"location":"office-000201","environment":"inside"}}, + {"time":"2016-11-16T21:18:00+00:00","temperature":62.699999999999974,"device":{"location":"door-00084","environment":"doorway"}}, + {"time":"2016-11-16T21:18:00+00:00","temperature":85.49999999999918,"device":{"location":"field-000204","environment":"outside"}}, + {"time":"2016-11-16T21:18:00+00:00","temperature":42,"device":{"location":"arctic-000218","environment":"outside"}}, + {"time":"2016-11-16T21:18:00+00:00","temperature":42,"device":{"location":"arctic-000217","environment":"outside"}}] + +Time-Oriented Analytics +----------------------- + +TimescaleDB includes new aggregate functions for time-oriented `analytics `_. + +For using aggregate queries with PostgREST you must create VIEWs or :ref:`s_procs`. Here's an example for using `time_bucket `_: + +.. code-block:: postgres + + -- Run psql in the database container + docker exec -it tsdb_tut psql -U postgres + + -- Create the function + create or replace function temperature_summaries(gap interval default '1 hour', prefix text default 'field') + returns table(hour text, avg_temp numeric, min_temp numeric, max_temp numeric) as $$ + select + time_bucket(gap, time)::text as hour, + trunc(avg(temperature), 2), + trunc(min(temperature), 2), + trunc(max(temperature), 2) + from conditions c + where c.device_id in ( + select device_id from locations + where location like prefix || '-%') + group by hour + $$ language sql stable; + + -- Exit psql + exit + +Every time the schema is changed you must reload PostgREST :ref:`schema cache ` so it can pick up the function parameters correctly. To reload, run: + +.. code-block:: bash + + docker kill --signal=USR1 tsdb_pgrst + + +Now, since the function is ``stable``, we can call it with ``GET`` as: + +.. code-block:: bash + + curl -G "localhost:3000/rpc/temperature_summaries" \ + -d gap=2minutes \ + -d order=hour.asc \ + -d limit=10 \ + -H "Accept: text/csv" + ## time_bucket accepts an interval type as it's argument + ## so you can pass gap=5minutes or gap=5hours + +.. code-block:: sql + + hour,avg_temp,min_temp,max_temp + "2016-11-15 12:00:00+00",72.97,68.00,78.00 + "2016-11-15 12:02:00+00",73.01,68.00,78.00 + "2016-11-15 12:04:00+00",73.05,68.00,78.10 + "2016-11-15 12:06:00+00",73.07,68.00,78.10 + "2016-11-15 12:08:00+00",73.11,68.00,78.10 + "2016-11-15 12:10:00+00",73.14,68.00,78.10 + "2016-11-15 12:12:00+00",73.17,68.00,78.19 + "2016-11-15 12:14:00+00",73.21,68.10,78.19 + "2016-11-15 12:16:00+00",73.24,68.10,78.29 + "2016-11-15 12:18:00+00",73.27,68.10,78.39 + +Note you can use PostgREST standard filtering on function results. Here we also changed the :ref:`res_format` to CSV. + +Fast Ingestion with Bulk Insert +------------------------------- + +You can use PostgREST :ref:`bulk_insert` to leverage TimescaleDB `fast ingestion `_. + +Let's do an insert of three rows: + +.. code-block:: bash + + curl "localhost:3000/conditions" \ + -H "Content-Type: application/json" \ + -H "Prefer: return=representation" \ + -d @- << EOF + [ + {"time": "2019-02-21 01:00:01-05", "device_id": "weather-pro-000000", "temperature": 40.0, "humidity": 59.9}, + {"time": "2019-02-21 01:00:02-05", "device_id": "weather-pro-000000", "temperature": 42.0, "humidity": 69.9}, + {"time": "2019-02-21 01:00:03-05", "device_id": "weather-pro-000000", "temperature": 44.0, "humidity": 79.9} + ] + EOF + +By using the ``Prefer: return=representation`` header we can see the successfully inserted rows: + +.. code-block:: json + + [{"time":"2019-02-21T06:00:01+00:00","device_id":"weather-pro-000000","temperature":40.0,"humidity":59.9}, + {"time":"2019-02-21T06:00:02+00:00","device_id":"weather-pro-000000","temperature":42.0,"humidity":69.9}, + {"time":"2019-02-21T06:00:03+00:00","device_id":"weather-pro-000000","temperature":44.0,"humidity":79.9}] + +Let's now insert a thousand rows, we'll use `jq `_ for constructing the array. + +.. code-block:: bash + + yes "{\"time\": \"$(date +'%F %T')\", \"device_id\": \"weather-pro-000001\", \"temperature\": 50, \"humidity\": 60}" | \ + head -n 1000 | jq -s '.' | \ + curl -i -d @- "http://localhost:3000/conditions" \ + -H "Content-Type: application/json" \ + -H "Prefer: count=exact" + +With ``Prefer: count=exact`` we can know how many rows were inserted. Check out the response: + +.. code-block:: haskell + + HTTP/1.1 201 Created + Transfer-Encoding: chunked + Date: Fri, 22 Feb 2019 16:47:05 GMT + Server: postgrest/5.2.0 (9969262) + Content-Range: */1000 + +You can see in ``Content-Range`` that the total number of inserted rows is ``1000``. + +Summing it up +------------- + +There you have it, with PostgREST you can get an instant and performant RESTful API for a TimescaleDB database. + +For a more in depth exploration of TimescaleDB capabilities, check their `docs `_.