- Create separate test suite for observability tests
- Create wrapper script `postgrest-test-observability`
- Add to CI and `postgrest-check`
- Move JWT cache tests under observability tests
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Refactoring: State validation helpers used in JwtCacheSpec moved to SpecHelper
to make them available in other Spec modules.
(cherry picked from commit 85a313a8cc)
Fixes#4646. Using the repro on #4646, this now produces the log:
```
11/Feb/2026:09:40:08 -0500: Warp server error: stack overflow
```
When:
```
$ curl localhost:3000/
curl: (52) Empty reply from server
```
(cherry picked from commit e95e815483)
retryingListen function potentially leaks database connections. This patch ensures the connections are released in case of listen/notify errors.
(cherry picked from commit 00c7cb1a22)
* Calculation of hint message when requested relation is not present in schema cache requires creation of a FuzzySet (to use fuzzy search to find candidate tables). For schemas with many tables it is costly.
This patch introduces dbTablesFuzzyIndex in SchemaCache to memoize the FuzzySet creation.
* Additionally, because of FuzzySet large memory requirements, this patch introduces a limit of 500 relations per schema, above which FuzzySet is not created and hint calculation disabled.
(cherry picked from commit e592d568c6)
The `Ord` instance definition for type `Routine` had a logical
error when comparing two routines. The error did not affect any
end users. However, for correctness and completeness reasons, this
commit fixes the error.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Co-authored-by: Joel Jacobson <joel@compiler.org>
The hasSingleUnnamedParam function was only checking the parameter type
but not whether the parameter actually had no name. This caused functions
with a single NAMED parameter (e.g., `foo(data json)`) to incorrectly
match the single-param fallback mode.
The result was a confusing PostgreSQL error 42883 "function does not exist"
instead of a clean PGRST202 error explaining that no matching function
was found.
Added ppName == mempty check so functions with named parameters don't
incorrectly match the single-param fallback.
(cherry picked from commit fd6a3bdccf)
Centralizes handling of preferences that are used to
create `Preference-Applied` header which is returned on
responses.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 29c2aa015f)
Postgrest fails on unsupported pg versions. However before killing
the thread, it continues to print a few more log messages which
were misleading. This commit fixes this by making sure that the
no log message should be printed after the unsupported pg version
observation and kill the thread immediately.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 0fa2f569a7)
When running postgres from nix-shell, nix creates a directory
structure like `postgrest/postgrest-with-pg-17-XXX` in the `/tmp`
directory. This commit removes the extra `postgrest/` prefix to
shorten length of absolute path length of filenames.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 802cce9a28)
The `test_io.py` module is too bloated (2100+ lines). To
logically group related tests, as a first step, this commit
separates authentication related IO tests into `test_auth.py`
module.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit c561a3749c)
Removes a test related to jwt cache which is stale
since #4084.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 379eaec8e0)
Renames these scripts to `postgrest-with-pg-xx`. The renaming
helps reduce the length of temporary filenames. This is needed
to ensure that socket file names remain under the maximum
allowed length of 107 chars.
Closes#4461.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 462798dc2d)
The docker-compose example did not work with multiple containers
when PGRST_SERVER_HOST is set to `localhost`. This updates the
value to `0.0.0.0` allowing other containers to connect.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 08c6deba12)
The x86 GitHub runner will not be available anymore, soon.
We might be able to re-introduce this, once we can build a static
executable via Nix on darwin, too.
- Adds fixtures to `test/io/fixtures.sql` to test resource
embedding related queries.
- Moves the resource embedding related tests that no longer
require big schema from `test_big_schema.py` to `test_io.py`.
Closes#4417.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 91abcd49e1)
This should reduce setup time for build process.
- cache: introduced in #2928, defunct since #4084
- clock: introduced in #2928, defunct since #4084
- heredoc: introduced in #714, defunct since #4390
- iproute: introduced in #3560, defunct since #4288
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 99217433b1)
The changelog description mentioned that we follow semantic
versioning but from now on we don't. Hence updated the description
to reflect new versioning policy.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit eb908c696f)
Updates streaming-commons to version 0.2.3.1. This resolves#3202.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 58efc2680e)
When db-pre-config is accidentally set to a pg reserved word
like "true", it fails with a confusing error. The function
names should be properly quoted to avoid such errors. This commit
resolves this by quoting the pre-config function name.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit a688878236)
Moves the functions `escapeIdent` and `trimNullChars` to
SchemaCache/Identifiers.hs module.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 66f84c5903)
This change is an initial step to change JWT aud configuration to regular expression.
Exporting function
audMatchesCfg :: AppConfig -> Text -> Bool
from Config module allows changing the way how JWT aud is configured to be isolated and not affect code in Auth.JWT
* replaces v2-build -> v2-run (which should be v2-exec)
with one-shot v2-run
* fixes empty postgrest.prof file by sending SIGINT to the profiled
postgrest process to trigger writing it explicitly
To make schema cache loading wait tests robust it is necessary to provide three separate internal config variables:
* "internal-schema-cache-query-sleep" - introduces delay in schema queries execution
* "internal-schema-cache-load-sleep" - introduces delay between schema queries execution and processing their results
* "internal-schema-cache-relationship-load-sleep" - introduces delay in processing relationship query results
Thanks to these changes it is now possible to test various schema loading scenarios with the right granularity robustly (eg. make sure requests wait for schema loading but not for relationship loading).
We usually miss adding or changing the example config
file when we do changes to our configurations. Moving
the example config file from CLI module to Config module
would make it less easy to miss doing the associated
changes to the example config file.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
There are a few benefits for this:
- All fixtures in one module, so single source of truth.
- The fixtures are automatically imported and injected by pytest
so no explicit imports needed for these.
- Linters won't complain about redefinition of outer scope objects.
Co-authored-by: Jens Troeger <jens.troeger@light-speed.de>
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The `--ready` flag is a wrapper around the admin server
`/ready` request. This is done through using an http client
library in postgrest.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The CLI page had the config file parameter missing. This commit
adds that. Also adds the CLI usage output which shows all
available options. Closes#4300.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Moves the `set_statement_timeout` and `reset_statement_timeout`
function to postgrest.py. This cleans up test_io.py so it only
contains tests. Also makes these functions reusable in other
other tests like test_cli.py.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
This splits off the transaction concerns into a MainTx module
It's just moving some functions inside the Query module to the MainTx module.
Some types names were also changed so they don't have the "Query" prefix, to avoid confusion.
Sets hpctixfile to be unique for every postgrest process that
is run.
Previously, this was based on the test name, but issues arise
when two postgrest processes are run under the same test, which
generates two files where one gets overwritten by the other.
Consequently, coverage data used to get lost, which is now fixed
with this commit.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Closes https://github.com/PostgREST/postgrest/issues/4226.
This requires moving query generation to the top App.hs module.
At this point is also simple to log the transaction variables + the
pre-request function call but this is not done here to reduce scope.
Towards fixing #4316. The config error tests were not
correctly tested. This commit allows capturing stderr
output and then checks the error messages properly.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Replaces the "API server listening on unix socket" with simpler
"API server listening on " observation. This allows refactoring
redundant code.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The IPv6 address logged at the startup like `::1:80` was
wrong because the port isn't clearly separated. This commit
corrects it, now logging as `[::1]:80`.
This is done in accordance to RFC 3986. In short, we did this
have a clear separation between the port and host because
the components of an IPv6 are separated with the ':' character.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The prometheus metrics text format requires `Content-Type` header
for correct scraping which fails otherwise. Closes#4271.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The `fromJsonBodyF` function was very unreadable and making
changes were difficult. This commit improves the readability
for easier future changes and improvements.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Closes https://github.com/PostgREST/postgrest/issues/4245.
It adds a query on schema cache construction, but doesn't add a new
attribute to the SchemaCache type. We only need the information to be
passed to the `initialMediaHandlers` function to build the builtin media
handlers.
Changes:
1. Refactoring and some cleanup of JWT handling code:
* Instead of caching AuthResult cache decoded claims (which signature was verified). Validating claims and determining role is done after cache lookup
* Cleaned up API so that usage of it is simplified: lookupJwtCache cache key >>= parseClaims configJwtAud time
* Handling of JwtCacheState initialization and updates of configuration is encapsulated in Auth.JwtCache module
2. Generic high performance (hopefully) scalable, dynamically resizeable cache implementation based on stm, stm-hamt and sieve algorithm. It also integrates with PostgREST measurements infrastructure providing usage stats (ie. hit ratio, evictions count)
This way we benefit from NixOS' binary cache to deliver GHC for us and
don't need to cache it ourselves.
This will become relevant once we do that for more platforms.
We have fixed the static build of PostgreSQL upstream, so we don't need
our separate overlay anymore. One more step towards building a static
executable on other platforms.
Instead of rolling our own, we can use some tooling from nix / nixpkgs.
We drop the "statically linked" check, because our goal is to compile
mostly-static executables to darwin, too. However, those will never be
fully static, because they always link to the platform's libc.
The dependency on it was removed in #3608 already, but we forgot to
remove it from postgrest.cabal, which caused it to still be built.
We didn't realize because all references of it were stripped away by
dead code elimination anyway.
The OpenAPI specification was incorrectly exposing GET methods for
VOLATILE functions, even though such functions properly reject GET
requests at runtime with "405 Method Not Allowed". This created
a mismatch between the advertised API specification and the actual
runtime behavior.
VOLATILE functions should only be callable via POST since they may
have side effects, while STABLE and IMMUTABLE functions can safely
be called via GET since they don't modify database state.
Fix by checking the pdVolatility field in makeProcPathItem() and
only including GET methods in the OpenAPI PathItem for non-volatile
functions.
The runtime behavior was already correct; this fixes only the
OpenAPI documentation generation.
Completes the jwt loadtests, now we have non-cached, cached + worst case
for the cache.
- jwt-hs (edited): now has the cache disabled
- jwt-hs-cache: cache enabled for jwt-hs
- jwt-hs-cache-worst: worst case of the jwt-hs cache
- jwt-rsa (edited): now has the cache disabled
- jwt-rsa-cache: cache enabled for jwt-rsa
- jwt-rsa-cache-worst: worst case of the jwt-rsa cache
Also deletes `nix/tools/generate_targets_rsa.py` and uses a single
python script.
Should prove what's mentioned on
https://github.com/PostgREST/postgrest/pull/4084#issuecomment-2998170423
Previously, the loadtest would always compare against main and the
latest tag. This meant a loadtest running on the v13 release branch,
would compare to a *future* version of both. This is not useful, and
also not supported by `postgrest-loadtest-against`, which recently
introduced a change on `main`, that now breaks the tests on the back
branches.
With this change, the loadtest will always run against the current
target branch of a PR, so against the v13 branch for a PR against v13,
for example. Also, it will compare against the latest released tag *for
that branch*.
Thus, when running this for v12, it will test against the v12 branch and
the v12.2.12 tag.
Splitting the executable in a separate output avoids distributing all of
the libraries and documentation, which are not needed when just running
PostgREST.
Reduces closure size from 4.3G to 73.9M for the flake exported packages.
Resolves#4149
The `commitlint` script disrupts our release workflow if failed
on push event. We also don't need it on push because once pushed,
git commit history can't be amended.
Closes https://github.com/PostgREST/postgrest/issues/4107.
Adds two python scripts:
- monitor_pid.py: monitors the postgrest process each second
until it exits, then outputs a csv with the results. The nix wrappers
use the `loadtest` dir for the output.
- merge_monitor_result.py: receives a list of csvs and merges them into
a single markdown table. The nix wrappers use the `loadtest/*.csv`
files for the input.
The nix `postgrest-with-pgrst` and `postgrest-loadtest-report` commands
use these scripts to add monitoring for `postgrest-loadtest` and
`postgrest-loadtest-against`.
This is so the process monitoring results are more meaningful.
* remove the amount of total targets from the loadtest kind name
since the targets can change.
* clarifies the top comment on the generate_targets python script
* increase TOTAL_TARGETS on jwt-hs loadtest
The `jwt-aud` config was not validated when containing ':'
character according to RFC 3986. This fix validates it and
fails at startup if it is invalid.
Until now we had a load test with 50k unique JWTs signed with symmetric key.
This commit adds a new load test with 10k JWTs signed with RSA 4096.
Existing -k jwt parameter was changed to -k jwt-hs-50k.
New test is run with -k jwt-rsa-10k parameter.
Additionally a new parameter --jwtcache=off was added to turn off JWT caching in the above load tests.
This change introduces a PostgREST.Auth.Jwt module containing JWT validation functions.
The reason to extract them from Auth is to enable JwtCache module to reuse them without introducing module dependency cycle.
Moat has been bought by Oracle. The advertising subpage redirects to
oracle.com, but pointing at that would be a bit misleading.
No need to keep failing links either.
BREAKING CHANGE
The endpoint was at risk of being left unprotected when exposing it.
The accompanying `admin-server-config-enabled` config was also dropped.
This loadtests the jwt decoding logic. For this it adds an optional
`-k`(kind) parameter to `postgrest-loadtest` and
`postgrest-loadtest-against`.
Old kind (default):
```
postgrest-loadtest -k mixed
postgrest-loadtest-against -k mixed
```
New kind:
```
postgrest-loadtest -k jwt
postgrest-loadtest-against -k jwt
```
Internally it uses a dynamically generated targets file using python
which looks like:
```
GET http://postgrest/authors_only
Authorization: Bearer <jwt>
GET http://postgrest/authors_only
Authorization: Bearer <another-jwt>
...
```
Then this is used to run vegeta with the `-lazy` option.
The old error design didn't allow us to reuse error `code`,
`message` etc. With this refactor, these parts of the error body
can be reused for other potential features.
This also removes the `ErrorCode` type and replace the types
with Text codes.
Our current ErrorCode type wasn't giving us a full picture
of how many different types of errors we are handling. This
PR makes this explicit by including the error code type for
all types of errors.
This also moves the pin for nixpkgs into flake.lock instead of our
custom file. Even for the classic interface via default.nix, the pin
will be loaded from flake.lock, thus everything stays in-sync.
This just exposes the PostgREST package, not more.
The goal is to avoid duplication, so we're re-using only exports from
default.nix.
Resolves#3026
Supersedes #3105
We never need dynamic haskell libraries, because even the dynamic builds
only dynamically link non-haskell dependencies, but always link haskell
dependencies statically.
Profiled libraries are only required when running the memory test, so
explicitly enable them for the profiled package.
This also means, that we don't need to hide the memory test behind a
feature flag anymore. The reason always was assumed to be the big number
of rebuilds required for it. I assume ever since we moved off of
static-haskell-nix and back to nixpkgs-based builds, we have been
building profiled libraries for all our dependencies anway.
I removed those exceptions earlier this week, because I got fooled by my
local test output. Those only return 403 in GitHub Actions - it seems
like those websites block access from there.
Apparently Cirrus removed the 14-1 image. When I firsted looked into
this some days ago, when the job started failing, the docs were not
updated, yet - so it wasn't clear. Now the docs mention freebsd-14-2
explicitly...
This manager was recently introduced to renovate and is now creating PRs
for upper version bounds of our haskell dependencies. We still need to
figure out how to deal with those in the best way, but some basic
configuration can already be done. Here, we:
- disallow any of those updates on the backbranches.
- group GHC-provided dependencies together.
- group packages from the hasql ecosystem together.
- the fuzzyset dependency must be restricted to <0.3 - we know that
already and did that on purpose.
Apparently the STACK_ROOT has been moved to C:\sr - for unknown reasons,
at least to me.
This should enable caching again and make the stack on windows builds
much faster than recently.
When macos-14 was rolled out libpq was not installed, but by now it is
by default. Thus, we don't need to do that, it only creates a warning
annotation right now.
Since we're currently on the unstable channel and will likely stay there
for a while, let's encode this in the update script.
Once we switch back to stable, if we do, we can still adjust it again.
Currently it's redundant and not easy to read.
```
{"message":"JSON object requested, multiple (or no) rows returned",
"details":"The result contains 2 rows"}
```
Now:
```
{"message":"Cannot coerce the result to a single JSON object",
"details":"The result contains an array of 0 objects"}
``
Also correct docs which had a wrong error code.
It used to be that this was only enabled with log-level=debug.
But the default log-level is misleading, for example:
```
$ PGRST_DB_SCHEMAS="apflora" postgrest-with-postgresql-16 -f test/io/big_schema.sql postgrest-run
...
13/Nov/2024:22:08:20 -0500: Config reloaded
13/Nov/2024:22:08:20 -0500: Schema cache queried in 36.3 milliseconds
13/Nov/2024:22:08:20 -0500: Schema cache loaded 326 Relations, 305 Relationships, 7 Functions, 0 Domain Representations, 4 Media Type Handlers, 1194 Timezones
```
The "Schema cache loaded" can take a while to appear, yet the 22:08:20
time is the same. If we reveal the load time this is clarified:
```
13/Nov/2024:22:08:37 -0500: Schema cache loaded in 16770.1 milliseconds
```
This should fix CI which is failing lately like this:
https://cirrus-ci.com/task/4665005218463744
ld-elf.so.1: /lib/libc.so.7: version FBSD_1.8 required by
/usr/local/bin/stack not found
It's important for observability to have an historic trace of the pool
size. Currently we expose it on the metrics endpoint, but not all
deployments use it.
This logs the pool size after the successful connection log to make it
more visible:
<timestamp>: Connection Pool initialized with a maximum size of 4 connections
It's not immediately clear on which port the API server is listening.
Also it's not clear that the "pgrst" channel is for database
notifications.
Goes from:
<timestamp>: Admin server listening on 0.0.0.0:3001
<timestamp>: Listening on 0.0.0.0:3000
<timestamp>: Listening for notifications on the "pgrst" channel
To:
<timestamp>: Admin server listening on 0.0.0.0:3001
<timestamp>: API server listening on 0.0.0.0:3000
<timestamp>: Listening for database notifications on the "pgrst" channel
This change adds an explanation of how to handle an array of JSON objects within an RPC call. To pass multiple objects, an array of JSON objects must be the JSON value, with the key being the json or jsonb variable name of the Postgres function.
For people who want to perform multiple tasks/inserts/updates within a single API call, this is a needed explanation for that use-case.
- Fixed "column reference <col> is ambiguous" error when selecting "?select=...table(col,count())"
- Fixed "column <json_aggregate>.<alias> does not exist" error when selecting "?select=...table(aias:count())"
Fixes#3660. Load the config after getting the pg version but before loading the schema.
The regression happened on f09655b.
Also remove schema cache load wrapper and separate db queries in different functions.
Co-authored-by: Laurence Isla <lau.isla.c@gmail.com>
Fixes https://github.com/PostgREST/postgrest/issues/3523.
Now if there's a failure when obtaining the pg version OR schema cache,
we do the same retrying process. This way we don't add two retries.
Refactors and renames the "connectionWorker" to "schemaCacheLoader".
This makes more sense since what we really want is the schema cache,
the version is the pre-requisite for ensuring our
schema cache queries work.
Additionally, we no longer log ` Attempting to connect to the database...`
at startup unnecessarily. This is only logged whenever there's a retry attempt.
Previously, we'd silently report "200 OK" on the root endpoint, but
would never return any endpoints from the schema cache.
Now the schema cache query fails because of the ::regnamespace cast.
The schema cache and OpenAPI output would currently list the first found
enum with the same name instead of the correct type. One other case
where this comes up is when a regular type and an enum type have the
same name. For example in the spec fixtures, we have an enum called
"bit". Every "bit" type, no matter whether it's that enum or the
built-in bit type, will show those enum options in the OpenApi output.
Not adding a test, because OpenAPI is supposed to go away in the future
anyway.
Previously they were passed as a JSON payload. This results in a LATERAL
join for the calling expression, which prevents LIMIT from being pushed
into the inlined function call, making some requests really slow.
Resolves#2858
* fix: forbid same server-port and admin-server-port
Forbids server-port and admin-server-port from being equal altogether,
despite they might not conflict at all in case admin and app are bound
to different addresses. Implemented as per the discussion at
https://github.com/PostgREST/postgrest/issues/3508#issuecomment-2125123633
This reduces our Template Haskell dependencies.
The commit hash never made it into the nix-based static executable
anyway. Since we'd like to move to produce more executables via nix in
the future, it will be hard to maintain the commit hash.
The cabal-provided Paths_ module allows us to use the version number
from postgrest.cabal. This can be done equally well with the GHC-defined
CPP macro "VERSION_postgrest".
By making this change we avoid the inclusion of the Paths_ module, which
also stores some paths related to the cabal configuration. Those are
problematic to go into the final executable, because for nix-based
builds those are paths to the /nix/store/... - which means that our
static executable then depends on those paths.. and we can't build a
minimal docker image anymore.
To counter this, we have been using dead code elimination when building
the static executable. This has been working well, but there is a
problem on aarch64-darwin, which we will hit once can finally make our
way there: GHC on aarch64-darwin (or darwin in general?) can't do dead
code elimination - and thus it'd be impossible to create those minimal
docker images for those platforms. More information upstream in nixpkgs:
https://github.com/NixOS/nixpkgs/issues/318013
Since we changed our release workflow, we have adjusted:
- the docs to use postgrest.org/en/v12/ -style URLs, i.e. only using the
major component.
- the pre-release / devel versions to contain only two instead of four
version parts, i.e. currently 12.3.
Timing dependent tests in the IO tests don't work too well when the next
commit increases the JWT parsing performance.
The remaining IO tests are for coverage and basic breakage. Loadtests
are adapted so that performance regressions for JWT caching would be
detected that way.
The former depends on th-orphans which does not cross-compile well,
because of template haskell usage.
neat-interpolation is also much better maintained.
This also potentially helps with packaging for Debian/Ubuntu in #2273.
Contravariant.Extras uses Template Haskell, which is hard to
cross-compile. Reducing usage of Template Haskell with the ultimate goal
of solving all cross compilation challenges.
This seems to happen on nix 2.22 only, v2.21 in CI and v2.20 locally work fine. The error is:
vendor folder is empty, please set 'vendorHash = null;' in your expression
For full logs, run 'nix-store -l /nix/store/kxnnr344n7gsxzc6kycj19hs19rvddjj-slocat-go-modules.drv'.
error: 1 dependencies of derivation '/nix/store/x9w480k36a11i99m6zp12d5cjijsn3lm-slocat.drv' failed to build
Since the slocat module doesn't actually have any dependencies, this shouldn't matter much.
Those create annoying "cancelled" notifications which looks like CI on
main was failing. It's not, though.
By just disabling the cancel-in-progress setting, but keeping the group
intact, this should queue multiple tag jobs / tag workflows behind each
other and still avoids the underlying problem which occurs when running
them in parallel.
This action makes sure to always have the correct GHC and/or stack
version installed in all environments. This solves problem where ghc or
stack might not be available on newer macos images anymore or where
ghcup is not available by default on our new custom github runner on
arm.
This is a first step to split up the cabal and stack caches in separate
pieces. Here we split the work folder, which just contains the
postgrest-specific build artifacts, into a separate cache.
More fine-grained caching should give us better cache hits and much
fewer upload size in the regular case, improving CI performance.
Since the work file caches are very small (about 30-40 MB) they are
cached for PRs, too. This will allow the majority of PRs, which only
change source code files, but no dependencies, to still have cached
their build files for additional commits.
This restores caches on all branches and pull requests, but only stores
them on the main branch and release branches. This prevents those caches
from being evicted early when we hit the 10 GB limit quickly.
Update hasql-notifications to include the fix on
https://github.com/diogob/hasql-notifications/issues/24.
Which now reveals the following error:
```
$ postgrest-with-postgresql-16 --replica -f test/spec/fixtures/load.sql postgrest-run
17/May/2024:18:35:38 -0500: Successfully connected to PostgreSQL 16.2 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.0, 64-bit
17/May/2024:18:35:38 -0500: Could not listen for notifications on the "pgrst" channel. ERROR: cannot execute LISTEN during recovery
17/May/2024:18:35:38 -0500: Retrying listening for notifications...
```
This is still not good because the LISTEN channel will be retried
forever without a backoff.
When using `postgrest-with-postgresql-* --replica`, the PGRST_DB_URI
will set the replica host as preference. This to be able to run
quick manual tests with postgrest running on a replica.
The fatal logic is now inside `usePool`. It centralizes the
logic which is better for Locality of Behavior.
Removes:
- The need to do checkIsFatal on other parts of the code
- SCFatalFail/ConnFatalFail states which are no longer needed.
```
$ postgrest-gen-secret
uMd97XSQzNkA1CWhMZ7u88Pj0RNyhrpo
$ postgrest-gen-jwt postgrest_test_author
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoicG9zdGdyZXN0X3Rlc3RfYXV0aG9yIn0.Xod-F15qsGL0WhdOCr2j3DdKuTw9QJERVgoFD3vGaWA
```
Also modifies postgrest-run to include a default PGRST_JWT_SECRET for
quicker manual tests.
This doesn't make sense, because each tag is only pushed when those
pipelines have already passed. Thus, we can save time and avoid wasting
resources and don't run those again.
This happens a commit is pushed to main while the pipelines have not
finished for the previous commit. In this case the devel-tag pipelines
will run concurrently, leading to unpredictable results for the devel
release.
It's very unlikely, but it can (and did) happen that both the server and
admin ports have the same number returned from freeport(). This then
leads to a situation where PostgREST will accept the same port in both
cases, because the host "localhost" will allow binding to ipv4 or ipv6
respectively. This will make the IO tests fail.
This change makes sure that the admin port will never be the same as the
server port and thus avoids this problem.
The $subject commit broke internal_schema_cache_sleep for other tests.
This reverts the order change, but keeps the scaling by x1000 to ms and
thus changes other users of this setting to the new scale.
Moving values defined with `let` to the function arguments (with
defaults) means other consumers of `default.nix` can customize these
values.
One example is a `flake.nix`, which can then supply the `nixpkgs` input.
This changes the postgrest-release tool to work with our new workflow.
It can be run on main and the v* release branches. When on a release
branch, it will bump a patch version and push to that branch only.
When on main, it will bump a minor version by default. To bump a major
version, pass --major. This first bump will be force-pushed to the
v<major> branch. A second bump to the current development version will
then be pushed to the main branch.
The tool will not tag commits anymore - this happens in CI
automatically.
References #3113Resolves#3082
By increasing the delays in this test by factor 400x, postgrest will not
swamp pg with connection retries after the failed schema cache anymore.
This would happen because there is no backoff included after fatal
errors. Once it does, the io tests hang indefinitely in CI.
The statement timeout needs to be cleaned up after each test that
modifies it instead of before the test. Otherwise the changed timeout
leaks into other tests.
Defaulting to max-jobs = auto should improve build times by using more
cores.
Setting always-allow-substitutes to true should cause all nix
derivations to be cached on cachix, which should improve performance of
the MacOS job dramatically, when no rebuilds need to happen.
The release / tag job has logic to decide whether to push a new tag on
stable branches, which depends on the all the tags being fetched. The
checkout action doesn't do that by default, so enable that.
This work by automatically pushing a new tag on main and release
branches after each commit. The tag will be "devel" on main and the
version from postgrest.cabal for release branches. The release
workflow then runs as a tag pipeline, making the actual release.
For release branches, the tag will only be created if a tag for this
version doesn't exist, yet. This means to actually make a new patch
release, we still need to bump the version in postgrest.cabal. We
can automate this later as part of our backport-bot.
Resolves#2006Resolves#2997
This now behaves similar to other CI build jobs, which always run on the
main branches, but only conditionally on PRs, depending on which files
changed.
* new --replica option to `postgrest-with-postgresql-*`
* new command `postgrest-test-replica`
* new sanity tests on test_replica.py
* add postgrest-test-replica to postgrest-check and postgrest-coverage
Using the -latest tag is potentially prone to errors, because an update of the tag
could break our CI. This recently happend with macos-latest, which we downgraded
to macos-12 earlier.
Using an explicit version reference makes this problem much less likely - in fact
renovate will pick up new versions once they exist and will suggest updates for it.
Thus, we will see the failures in a related PR instead of randomly everywhere.
f9ee1f7e introduced the hoisting of function settings as transaction-scoped
settings. However, this currently doesn't work with memory units, which are
case-sensitive according to the docs [1]. This removes the lowercasing of
values to make them work.
This is not added to the CHANGELOG, because this feature was not released, yet.
[1]: https://www.postgresql.org/docs/current/config-setting.html#CONFIG-SETTING-NAMES-VALUES
With this:
- Is no longer necessary to pass observer as an argument
to every function that needs observations.
- We can invoke the observer on every function that uses AppConfig.
However it'd be better to just call the observer in the upper modules
(like on App.hs).
A previous commit allowed to select each tool separately on the toolbox.
This commit makes use of that for CI to possibly speed up loading from cachix
a little bit. It will also cause fewer cache misses when nix code is changed.
Resolves#3183
This is to reduce storage requirements for GitHub Actions cache. We already build with GHC 9.4.x
via Nix on Linux x64, via stack on FreeBSD, MacOS and Windows and via Cabal on Linux ARM. That
should cover 9.4.x enough.
The Lint & Style job needs to run on all PRs, not only when something "test" related
changes. Otherwise not all workflow, nix or other files are style-checked and linted.
The enable-executable-static flag is set by default, so doesn't make a difference.
The pkg-config improvement for libpq was merged upstream, so we can use the same
here already.
This documents supported GHC versions. GHC 9.8.1 is currently commented
out to reflect the fact that PostgREST can't currently be built with it
straight from hackage - we still require some overrides in cabal.project
for that.
Splitting the output into separate files and adding top-level newlines makes this
much better to read and understand when looking at diffs.
Inspired by #1699
All those DROP IF EXISTS and CREATE IF NOT EXISTS etc. just give a lot more text to
read and understand. If in fact a user creates the same thing twice, they should be
able to understand the error message from postgres.
The jwt_token type was not created consistently in all examples, which can
be confusing when following those. To return an object with a single key
named token, it's enough to have an OUT parameter to the function.
Resolves https://github.com/PostgREST/postgrest-docs/issues/280
Those were left out of the schema dump when the features were introduced, probably
because ByteString doesn't have a toJSON instance. Changing the type to Text solves
this easily.
Resolves#3237
The obviously wrong statement is, that PostgREST does not support asymmetric keys, while it
does. Extending on this type of attack is not necessary, because it is in fact covered by
the paragraph before - reading the algorithm from the JWT header is the problem in that case,
too. We don't do that.
This leaves us with the sentence about how the chosen library is the most important part. While
that is correct, the hint about high quality libraries for use on the *client* side is mis-
leading: The important part here is the library we choose to implement PostgREST with, not the
client-side lib. Thus, removing the whole paragraph is the best thing to do here.
Resolves https://github.com/PostgREST/postgrest-docs/issues/123
Some users connect PostgREST to the wrong PostgreSQL instance - likely because they are
not even aware that another instance is running. By using the standard port 5432 instead
of 5433, we avoid this problem. The user will be made aware very early that they have
another postgresql instance running - and can solve the problem at this stage. If they
decide to change the port, they are much more likely to remember that in the later stages
of the tutorial, too.
Resolves https://github.com/PostgREST/postgrest-docs/issues/304
Commit 85fbb233 accidentally changed the default socket location in which libpq is
looking for postgresql unix sockets. This is changed in nixpkgs via patch. By imp-
orting the default patches, this is changed back to what it was before. Without
those patches it was changed from /run/postgresql to /tmp.
Not a bugfix, because it was not released, yet.
This moves the _build folder into the repo root, to avoid postgrest-watch ending in an
infinite loop of restarting the build.
Also, for repeated use during development, running linkcheck is not a good idea, this
will quickly result in rate-limiting requests from various servers.
This allows more flexible control over the working directory. Values for workingDir must always start
with a / and will then be relative to the repo root.
Going forward, an uneven minor version will be a development version, while an even minor will be
considered a stable version to be released. This is similar to what GHC does and was discussed in
#3113.
This bump should have happened after branching off v12.0.0, but obviously we didn't know about it
back then. This will happen immediately after branching off a new release from now on.
Previously using a generic mimetype handler failed when any kind of select= was given, because
we tried to cast the select-result to the original table type. With this change, this cast is
only applied when select=* is given implicitly or explicitly. This is the only case where this
makes sense, because this guarantees that correct columns are selected in the correct order for
this cast to succeed.
Resolves#3160
The upside is that postgrest-with-postgresql-xxx postgrest-test-io works as expected
now. The downside is, that postgrest-with-postgresql-xxx psql now starts without
any schema. This now needs an explicit postgrest-with-postgresql-xxx -f path/to.sql
to do anything useful.
Resolves#2864
Removing the condition when migrating the loadtest workflow was not helpufl, this
triggers the report job a few times per pipeline. The goal was to always download
the report, even when the overall pipeline fails because of some other jobs.
Explicitly checking for both success and failure should be enough.
When updating stack.yaml, we need to make sure to update stack.yaml.lock, too.
This check prevents them from getting out of sync by failing CI in this case.
This improves cachability.
The reason why those workflows were split in the first place was just to obtain loadtest
results quicker, because the in the separated workflow, only the single loadtest job
needs to finish before the artifacts can be downloaded.
However, the disadvantage of this approach was, that the results were not as easily
accessible as they could be in a single workflow. Additionally, it's possible to depend
on the "prepopulate nix" job for efficiency if the loadtest runs in the main workflow.
The two different PR and Merge jobs were introduced to be able to test the main branch
against the latest release. However, this is now included in the PR job, too, so no need
for the two separate jobs anymore.
The nix actions cache currently leads to repeated "no space left on
devices" errors for jobs in CI.
The prepopulate job is useless without the nix actions cache, so it
will go away at the same time.
The cache key for nix now depends on default.nix and shell.nix in the root folder and all
.patch files in the nix folder. Those may change the output of our nix derivations, so
must be included. At the same time, there is no reason to include the actions/setup-nix
folder. This would only lead to new caches being created every time we update one of the
dependent actions in this file. Finally, we never restore caches with a different id any-
more. There is no point in having the style job fall back to the static cache for example.
The cache keys for cabal can be more explicit: We only have one postgrest.cabal and one
relevant cabal.project file. We were missing the cabal.project.freeze file, though, which
affects the dependencies used, too.
It used to be possible to spin up a nix environment via docker container this way,
but the upstream nixos/nix image has changed and the docker build doesn't succeed
anymore. Since nobody complained about that, we can assume it is not being used
anyway.
cachix-action v14 added a new daemon mode, which pushes new derivations to the store
as soon as they have been built. This replaces the seed cachix workflow nicely by just
pushing from all jobs directly.
This works around https://github.com/NixOS/nixpkgs/issues/286285 to use -split-sections
in a cross-compiling scenario. This will reduce the size of the static executable and also
remove the remaining references to /nix/store/.. reducing closure size dramatically.
This also fixes the docker image blowing up in size since we switched to pkgsStatic.
By passing -split-sections to all dependencies, GHC will link only the
modules we actually use and not the full package for each dependency.
This does neither work on MacOS nor Windows, thus we don't do it for
stack right now.
Stripping unused symbols in CI will further decrease the size of those files.
This makes the static build fail in case any references to the nix store
are left over. Those will increase the closure size of the nix derivation
massively and lead to a huge docker image.
At the same time, those references will not be functional on non-nix systems,
to which the static executable is distributed, anyway.
PR #2358 added a bulk insert to the loadtest. However this broke the regular insert test,
which just returned 400 Bad Request because of a missing PK value since. Adding the new
id column in the payload to the ?columns= argument fixes that.
The recent nixpkgs update gave us a new version of vegeta. This version includes a new
DNS cache features - which unfortunately doesn't play well with unix sockets. Disabling
the DNS cache makes requests succeed again.
stock curl doesn't make it easy to make many parallel requests to a
single endpoint. The endpoint has to be repeated N times.
```
curl --parallel https://example.comhttps://example.com ..
```
This provides a wrapper for parallel curl. It's useful for testing
scenarios like pool timeouts.
```
parallel-curl N https://example.com
```
Previously this command upgraded to the latest unstable version of nixpkgs,
but this was often broken. Taking the latest stable branch should give
better results.
If your database schema has changed while the PostgREST server is running,
send the server a SIGUSR1 signal or restart it (http://postgrest.org/en/stable/admin.html#schema-reloading) to ensure the schema cache is not stale. This sometimes fixes apparent bugs.
-->
### Environment
* PostgreSQL version: (if using docker, specify the image)
* PostgREST version: (if using docker, specify the image)
* Operating system:
### Description of issue
Describe the behavior you expected vs the actual behavior. Include:
- A minimal SQL definition.
- How you make the request to PostgREST (curl command preferred).
On the highest level, PostgREST processes an HTTP request, if it's accepted it builds a SQL statement for it, executes it, and produces a response.
## Code Map
This section talks briefly about various important modules.
The starting point of the program is `main/Main.hs`, which calls `src/PostgREST/CLI.hs` which then calls `src/PostgREST/App.hs`.
`App.hs` is then in charge of composing the different modules.
### ApiRequest.hs
PostgREST operates over two types of resources: database relations(tables or views) and database functions; providing different representations(depending on the media type)
for them.
This module is in charge of representing the operation over an `ApiRequest` type. It parses the URL querystring following PostgREST syntax, the request headers, and the request body
(if possible it avoids parsing the body and sends it directly to the db).
A request might be rejected at this level if it's invalid, e.g. providing an unknown media type to PostgREST or using an unknown HTTP method.
### Plan.hs
Using the Schema Cache, this module enables more complex functionality(like resource embedding) by enriching the ApiRequest. It generates Plan types(`ReadPlan`, `MutatePlan`)
that then will be used to generate a SQL statement.
A request might be rejected at this level if it's invalid, e.g. by doing resource embedding on a nonexistent resource.
An OPTIONS request doesn't require a plan to be generated.
### Query.hs
This module constructs single SQL statements that can be parametrized and prepared. Only at this stage a PostgreSQL connection from the pool is used.
A query might fail(and be rollbacked) at this level if it doesn't comply to certain conditions, e.g. by not returning a single row when a ``Accept: application/vnd.pgrst.object`` header is specified.
An OPTIONS request doesn't require a query to be executed.
### Response.hs
This module constructs the HTTP response body with the right headers.
It builds the OpenAPI response using the schema cache.
### Auth.hs
This module provides functions to deal with JWT authorization.
### SchemaCache.hs
This queries the PostgreSQL system catalogs and caches the metadata into a SchemaCache type,
### AppState.hs
The state of the App which is kept across requests.
This spawns threads which are used to execute concurrent jobs.
Jobs include connection recover and a listener for the PostgreSQL LISTEN command.
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).
All notable changes to this project will be documented in this file. From version `14.0` onwards PostgREST follows a `MAJOR.PATCH` two-part versioning. Only even-numbered MAJOR versions will be released, reserving odd-numbered MAJOR versions for development.
## Unreleased
## [14.7] - 2026-03-20
### Fixed
- Fix not logging SIGTERM and SIGINT by @steve-chavez in #4728
## [14.6] - 2026-03-06
### Fixed
- Fix leaking table and function names when calculating error hint by @taimoorzaeem in #4675
## [14.5] - 2026-02-12
### Fixed
- Don't hide async exceptions in logs by @stevechavez in #4646
## [14.4] - 2026-01-29
### Fixed
- Ensure Listener connections are released by @mkleczek in #4614
- Fix incorrectly filtering the returned representation for PATCH requests when using `or/and` filters by @laurenceisla in #3707
- Fix listener running with exception masked after first failure by @mkleczek#4615
## [14.3] - 2026-01-03
### Fixed
- Fix performance and high memory usage of relation hint calculation by @mkleczek in #4462, #4463
## [14.2] - 2025-12-18
### Fixed
- Fix `hasSingleUnnamedParam` incorrectly matching functions with named parameters by @joelonsql in #4553
+ Functions with a single named parameter (e.g., `foo(data json)`) no longer incorrectly match the single-param fallback, returning a clean `PGRST202` error instead of a confusing PostgreSQL `42883` error.
- Fix misleading logs on unsupported PostgreSQL versions by @taimoorzaeem in #4519
- Fix regression where the `PGRST103` error response was truncated by @laurenceisla in #4455
+ Happened when an `offset` was greater than the rows requested and `Prefer: count=exact` was sent.
- Fix not returning `Content-Length` on empty HTTP `201` responses by @laurenceisla in #4518
- Fix inaccurate Server-Timing header durations by @steve-chavez in #4522
- Fix inaccurate "Schema cache queried" logs by @steve-chavez in #4522
## [14.1] - 2025-11-05
## Fixed
- Fix `db-pre-config` function failing when function names are pg reserved words by @taimoorzaeem in #4380
- Fix `server-host=!6` incorrectly binds to IPv4 address by @taimoorzaeem in #3202
## [14.0] - 2025-10-24
### Added
- Bounded JWT cache using the SIEVE algorithm by @mkleczek in #4084
+ It now uses a fixed size cache instead of arbitrary sized cache.
- Add `--ready` flag for postgrest healthcheck by @taimoorzaeem in #4239
### Fixed
- Fix not logging OpenAPI queries when `log-query=main-query` is enabled by @steve-chavez in #4226
- Fix not logging explain query when `log-query=main-query` is enabled by @steve-chavez in #4319
- Fix not logging transaction variables and db-pre-request function when `log-query=main-query` is enabled by @steve-chavez in #3934
- Fix not logging the JSON message to stderr on a `PGRST002` error by @laurenceisla in #4129
- Fix reloading the Schema Cache unnecessarily on a `PGRST002` error by @laurenceisla in #4367
- Fix schema cache loading taking a long time for large schemas by @mkleczek in #4360, #3704
### Changed
- Drop support for PostgreSQL EOL version 12 by @wolfgangwalther in #3865
- From now on PostgREST will follow a `MAJOR.PATCH` two-part versioning. Only even-numbered MAJOR versions will be released, reserving odd-numbered MAJOR versions for development.
- Replaced `jwt-cache-max-lifetime` config with `jwt-cache-max-entries` by @mkleczek in #4084
-`log-query` config now takes a boolean instead of a string value by @steve-chavez in #3934
## [13.0.8] - 2025-10-24
### Fixed
- Fix loading utf-8 config files with `ASCII` locale set by @taimoorzaeem in #4386
## [13.0.7] - 2025-09-14
### Added
- Improve the `PGRST106` error when the requested schema is invalid by @laurenceisla in #4089
+ It now shows the invalid schema in the `message` field.
+ The exposed schemas are now listed in the `hint` instead of the `message` field.
- Improve error details of `PGRST301` error by @taimoorzaeem in #4051
## [13.0.6] - 2025-08-30
### Fixed
- Fix logging the Haskell type instead of the listener error message directly by @laurenceisla in #3588
- Fix format of `IPv6` address logged at PostgREST startup by @taimoorzaeem in #4291
- Fix empty enum in `preferParams` OpenAPI parameter by @laurenceisla in #4292
## [13.0.5] - 2025-08-24
### Fixed
- Fix OpenAPI broken docs link by @taimoorzaeem in #4080
- Fix OpenAPI specification incorrectly exposing GET methods for volatile functions by @joelonsql in #4174
- Fix empty spread embeddings return unexpected SQL error by @taimoorzaeem in #3887
- Fix `/metrics` endpoint not responding with `Content-Type` header by @taimoorzaeem in #4271
## [13.0.4] - 2025-06-17
### Fixed
- Fix regression that makes full-text search not work on domain types based on `tsvector` by @laurenceisla in #4135
- Fix `jwt-aud` config not failing when set to an invalid URI by @taimoorzaeem in #4132
## [13.0.3] - 2025-06-16
- Fix `max-affected` preference not failing with RPC when `handling=strict` by @taimoorzaeem in #4100
- Fix a property definition's type in OpenAPI not showing the correct base type of a recursive domain by @laurenceisla in #4136
### Fixed
## [13.0.2] - 2025-06-02
### Fixed
- Fix regression that makes `ORDER BY` with nulls-order not work alongside limits by @laurenceisla in #4109
## [13.0.1] - 2025-06-01
### Fixed
- Fix jwt error returning HTTP status `400` for invalid role by @taimoorzaeem in #3601
- Fix `db-extra-search-path` cannot be set to nothing by @taimoorzaeem in #4074
+ It can now be disabled by setting it to empty string.
+ Schema Cache load error is now logged including `db-schemas` and `db-extra-search-path` config values.
## [13.0.0] - 2025-05-08
### Added
-#3558, Add the `admin-server-host` config to set the host for the admin server - @develop7
-#3607, Log to stderr when the JWT secret is less than 32 characters long - @laurenceisla
-#2858, Performance improvements when calling RPCs via GET using indexes in more cases - @wolfgangwalther
-#3560, Log resolved host in "Listening on ..." messages - @develop7
-#3727, Log maximum pool size - @steve-chavez
-#1536, Add string comparison feature for jwt-role-claim-key - @taimoorzaeem
-#3747, Allow `not_null` value for the `is` operator - @taimoorzaeem
-#2255, Apply `to_tsvector()` explicitly to the full-text search filtered column (excluding `tsvector` types) - @laurenceisla
-#1578, Log the main SQL query to stderr at the current `log-level` when `log-query=main-query` - @laurenceisla
-#3903, Log connection pool borrows on `log-level=debug` - @taimoorzaeem
-#3041, Allow spreading one-to-many and many-to-many embedded resources - @laurenceisla
+ The selected columns in the embedded resources are aggregated into arrays
+ Aggregates are not supported
-#2967, Add `Proxy-Status` header for better error response - @taimoorzaeem
-#3697, #3602, Handle queries on non-existing table gracefully - @taimoorzaeem
-#3600, #3926, Improve JWT errors - @taimoorzaeem
-#3013, Fix `order=` with POST, PATCH, PUT and DELETE requests - @taimoorzaeem
-#3965, Fix filter on unselected columns in a table-valued function - @taimoorzaeem
-#4052, Fix schema cache load duplicate objects with different object type but same oid - @taimoorzaeem
### Changed
-#2052, Dropped support for PostgreSQL 9.6 - @wolfgangwalther
-#2052, Dropped support for PostgreSQL 10 - @wolfgangwalther
-#2052, Dropped support for PostgreSQL 11 - @wolfgangwalther
-#3508, PostgREST now fails to start when `server-port` and `admin-server-port` config options are the same - @develop7
-#3607, PostgREST now fails to start when the JWT secret is less than 32 characters long - @laurenceisla
-#3644, Fail schema cache lookup with invalid `db-schemas` or `db-extra-search-path` config - @wolfgangwalther
- Previously, this would silently return 200 - OK on the root endpoint, but don't provide any usable endpoints.
- Note: This also applies when deleting the `public` schema - both config options default to that.
-#3757, Remove support for `Prefer: params=single-object` - @joelonsql
+ This preference was deprecated in favor of Functions with an array of JSON objects
-#3013, Drop support for Limited updates/deletes
+ The feature was complicated and largely unused.
-#3956, Drop `/config` endpoint of admin server - @steve-chavez
+ The endpoint was at risk of being left unprotected when exposing it.
+ The accompanying `admin-server-config-enabled` config was also dropped.
-#3598, PostgREST now validates the `kid` parameter of the JWT - @wolfgangwalther
+ If the JWT contains a ``kid`` parameter, then PostgREST will look for the JSON Web Key in the `jwt-secret`.
+ If the JWT doesn't contain a `kid`, the behavior should be backwards compatible. PostgREST will try each key in the `jwt-secret` one by one until it finds one that works.
- #3697, #3602, Querying non-existent table now returns `PGRST205` error instead of empty json - @taimoorzaeem
- #3210, Dump schema cache through admin API - @taimoorzaeem
- #2676, Performance improvement on bulk json inserts, around 10% increase on requests per second by removing `json_typeof` from write queries - @steve-chavez
- #3435, Add log-level=debug, for development purposes - @steve-chavez
- #1526, Add `/metrics` endpoint on admin server - @steve-chavez
- Exposes connection pool metrics, schema cache metrics
- #3404, Show the failed MESSAGE or DETAIL in the `details` field of the `PGRST121` (could not parse RAISE 'PGRST') error - @laurenceisla
- #3404, Show extra information in the `PGRST121` (could not parse RAISE 'PGRST') error - @laurenceisla
+ Shows the failed MESSAGE or DETAIL in the `details` field
+ Shows the correct JSON format in the `hints` field
- #3340, Log when the LISTEN channel gets a notification - @steve-chavez
- #3184, Log full pg version to stderr on connection - @steve-chavez
- #3242. Add config `db-hoisted-tx-settings` to apply only hoisted function settings - @taimoorzaeem
- #3214, #3229 Log connection pool events on log-level=debug - @steve-chavez, @laurenceisla
### Fixed
- #3237, Dump media handlers and timezones with --dump-schema - @wolfgangwalther
+ If the LISTEN connection fails, it's retried with exponential backoff
- #3414, Force listener to connect to read-write instances using `target_session_attrs` - @steve-chavez
- #3255, Fix incorrect `413 Request Entity Too Large` on pg errors `54*` - @taimoorzaeem
- #3549, Remove verbosity from error logs starting with "An error occurred..." and replacing it with "Failed to..." - @laurenceisla
### Deprecated
- Support for PostgreSQL versions 9.6, 10 and 11 is deprecated. From this on version onwards, PostgREST will only support non-end-of-life PostgreSQL versions. See https://www.postgresql.org/support/versioning/.
- `Prefer: params=single-object` is deprecated. Use [a function with a single unnamed JSON parameter](https://postgrest.org/en/latest/references/api/functions.html#function-single-json) instead. - @steve-chavez
### Documentation
- #3289, Add dark mode. Can be toggled by a button in the bottom right corner. - @laurenceisla
- #3384, Add architecture diagram and documentation - @steve-chavez
## [12.0.3] - 2024-05-09
### Fixed
@@ -72,6 +406,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
- Removed `application/octet-stream`, `text/plain`, `text/xml` [builtin support for scalar results](https://postgrest.org/en/v11.1/references/api/resource_representation.html#scalar-function-response-format) - @steve-chavez
- Removed default `application/openapi+json` media type for [db-root-spec](https://postgrest.org/en/v11.1/references/configuration.html#db-root-spec) - @steve-chavez
+ All PostgreSQL versions now use GUCs in JSON format for [Headers, Cookies and JWT claims](https://postgrest.org/en/v12/references/transactions.html#request-headers-cookies-and-jwt-claims).
## [11.2.2] - 2023-10-25
@@ -375,6 +710,11 @@ This project adheres to [Semantic Versioning](http://semver.org/).
- #2312, Using `Prefer: return=representation` no longer returns a `Location` header - @laurenceisla
- #1984, For the cases where one to one relationships are detected, json objects will be returned instead of json arrays of length 1
+ If you wish to override this behavior, you can use computed relationships to return arrays again
+ You can get the newly detected one-to-one relationships by using the `--dump-schema` option and filtering with [jq](https://github.com/jqlang/jq).
[](https://gitter.im/begriffs/postgrest)
@@ -59,8 +52,8 @@ Big thanks to our sponsors! You can join them by supporting PostgREST on [Patreo
## Usage
1.Download the binary ([latest release](https://github.com/PostgREST/postgrest/releases/latest))
for your platform.
1.See the docs for [how to install PostgREST on your platform](https://docs.postgrest.org/en/stable/explanations/install.html). You can also [use Docker](https://docs.postgrest.org/en/stable/explanations/install.html#docker).
2. Invoke for help:
```bash
@@ -68,7 +61,7 @@ Big thanks to our sponsors! You can join them by supporting PostgREST on [Patreo
```
## [Documentation](http://postgrest.org)
Latest documentation is at [postgrest.org](http://postgrest.org). You can contribute to the docs in [PostgREST/postgrest-docs](https://github.com/PostgREST/postgrest-docs).
Latest documentation is at [postgrest.org](http://postgrest.org). You can contribute to the docs in [PostgREST/postgrest/docs](https://github.com/PostgREST/postgrest/tree/main/docs).
## Performance
@@ -145,14 +138,14 @@ and the [API guide](http://postgrest.org/en/stable/api.html).
## Supporting development
You can help PostgREST ongoing maintenance and development by:
- Making a regular donation through Patreon https://www.patreon.com/postgrest
- Alternatively, you can make a one-time donation via Paypal https://www.paypal.me/postgrest
You can help PostgREST ongoing maintenance and development by making a regular donation through Patreon https://www.patreon.com/postgrest
Every donation will be spent on making PostgREST better for the whole community.
## Contributing
Contributions are always welcome and appreciated. Please see the [Contributing guidelines](https://github.com/PostgREST/postgrest/blob/main/CONTRIBUTING.md).
[](https://gitter.im/begriffs/postgrest)
PostgREST docs use the reStructuredText format, check this [cheatsheet](https://github.com/ralsina/rst-cheatsheet/blob/master/rst-cheatsheet.rst) to get acquainted with it.
To build the docs locally, use [nix](https://nixos.org/nix/):
```bash
nix-shell
```
Once in the nix-shell you have the following commands available:
-`postgrest-docs-build`: Build the docs.
-`postgrest-docs-serve`: Build the docs and start a livereload server on `http://localhost:5500`.
-`postgrest-docs-spellcheck`: Run aspell.
To build the docs locally, see [the Nix development readme](/nix/README.md#documentation).
## Documentation structure
This documentation is structured according to tutorials-howtos-topics-references. For more details on the rationale of this structure,
see https://www.divio.com/blog/documentation.
## Translating
To create `.po` files for translation into a new language pass the language code as the first argument to `postgrest-docs-build`.
Example to add German/de:
```
postgrest-docs-build de
```
The livereload server also supports a language/locale argument to show the translated docs during translation:
```
postgrest-docs-serve de
```
Spellcheck is currently only available for the default language.
*`PostgREST + Auth0: Create REST API in mintutes, and add social login using Auth0 <https://samkhawase.com/blog/postgrest/>`_ - A step-by-step tutorial to show how to dockerize and integrate Auth0 to PostgREST service.
*`PostgREST + PostGIS API tutorial in 5 minutes <https://gis-ops.com/postgrest-postgis-api-tutorial-geospatial-api-in-5-minutes/>`_ -
In this tutorial, GIS • OPS shows how to perform PostGIS calculations through PostgREST :ref:`s_procs` interface.
*`"CodeLess" backend using postgres, postgrest and oauth2 authentication with keycloak <https://www.mathieupassenaud.fr/codeless_backend/>`_ -
A step-by-step tutorial for using PostgREST with KeyCloak(hosted on a managed service).
@@ -20,7 +17,9 @@ Community Tutorials
*`Build data-driven applications with Workers and PostgreSQL <https://developers.cloudflare.com/workers/tutorials/postgres/>`_ - A tutorial on how to integrate with PostgREST and PostgreSQL using Cloudflare Workers.
.. * `A poor man's API <https://blog.frankel.ch/poor-man-api>`_ - Shows how to integrate PostgREST with Apache APISIX as an alternative to Nginx.
*`A poor man's API <https://blog.frankel.ch/poor-man-api>`_ - Shows how to integrate PostgREST with Apache APISIX as an alternative to Nginx.
.. * `Accessing a PostgreSQL database in Godot 4 via PostgREST <https://peterkingsbury.com/2022/08/16/godot-postgresql-postgrest/>`_
*`delibrium-postgrest <https://gitlab.com/delibrium/delibrium-postgrest/>`_ - example school API and front-end in Vue.js
*`ETH-transactions-storage <https://github.com/Adamant-im/ETH-transactions-storage>`_ - indexer for Ethereum to get transaction list by ETH address
*`general <https://github.com/PierreRochard/general>`_ - example auth back-end
@@ -49,17 +49,18 @@ DevOps
*`cloudgov-demo-postgrest <https://github.com/GSA/cloudgov-demo-postgrest>`_ - demo for a federally-compliant REST API on cloud.gov
*`cloudstark/helm-charts <https://github.com/cloudstark/helm-charts/tree/master/postgrest>`_ - helm chart to deploy PostgREST to a Kubernetes cluster via a Deployment and Service
*`jbkarle/postgrest <https://github.com/jbkarle/postgrest>`_ - helm chart with a demo database for development and test purposes
*`Limezest/postgrest-cloud-run <https://github.com/Limezest/postgrest-cloud-run>`_ - expose a PostgreSQL database on Cloud SQL using Cloud Run
*`cyril-sabourault/postgrest-cloud-run <https://github.com/cyril-sabourault/postgrest-cloud-run>`_ - expose a PostgreSQL database on Cloud SQL using Cloud Run
*`eyberg/postgrest <https://repo.ops.city/v2/packages/eyberg/postgrest/10.1.1/x86_64/show>`_ - run PostgREST as a Nanos unikernel
*`jbkarle/postgrest <https://github.com/jbkarle/postgrest>`_ - helm chart with a demo database for development and test purposes
.._eco_external_notification:
External Notification
---------------------
These are PostgreSQL bridges that propagate LISTEN/NOTIFY to external queues for further processing. This allows stored procedures to initiate actions outside the database such as sending emails.
These are PostgreSQL bridges that propagate LISTEN/NOTIFY to external queues for further processing. This allows functions to initiate actions outside the database such as sending emails.
*`pg-notify-stdout <https://github.com/mkleczek/pg-notify-stdout>`_ - writes notifications to standard output (use in shell scripts etc.)
*`pg-notify-webhook <https://github.com/vbalasu/pg-notify-webhook>`_ - trigger webhooks from PostgreSQL's LISTEN/NOTIFY
This section talks briefly about various important modules.
Main
----
The starting point of the program is `Main.hs <https://github.com/PostgREST/postgrest/blob/main/main/Main.hs>`_.
CLI
---
Main then calls `CLI.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/CLI.hs>`_, which is in charge of :ref:`cli`.
App
---
`App.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/App.hs>`_ is then in charge of composing the different modules.
Auth
----
`Auth.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Auth.hs>`_ is in charge of :ref:`authn`.
Api Request
-----------
`ApiRequest.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/ApiRequest.hs>`_ is in charge of parsing the URL query string (following PostgREST syntax), the request headers, and the request body.
A request might be rejected at this level if it's invalid. For example when providing an unknown media type to PostgREST or using an unknown HTTP method.
Plan
----
Using the Schema Cache, `Plan.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Plan.hs>`_ generates an internal AST, filling out-of-band SQL details (like an ``ON CONFLICT (pk)`` clause) required to complete the user request.
A request might be rejected at this level if it's invalid. For example when doing resource embedding on a nonexistent resource.
Query
-----
`Query.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Query.hs>`_ generates the SQL queries (parametrized and prepared) required to satisfy the user request.
Only at this stage a connection from the pool might be used.
Schema Cache
------------
`SchemaCache.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/SchemaCache.hs>`_ is in charge of :ref:`schema_cache`.
Config
------
`Config.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Config.hs>`_ is in charge of :ref:`configuration`.
Admin
-----
`Admin.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Admin.hs>`_ is in charge of the :ref:`admin_server`.
HTTP
----
The HTTP server is provided by `Warp <https://aosabook.org/en/posa/warp.html>`_.
Listener
--------
`Listener.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Listener.hs>`_ is in charge of the :ref:`listener`.
@@ -13,9 +13,9 @@ A role can be thought of as either a database user, or a group of database users
Roles for Each Web User
-----------------------
PostgREST can accommodate either viewpoint. If you treat a role as a single user then the :ref:`jwt_impersonation` does most of what you need. When an authenticated user makes a request PostgREST will switch into the database role for that user, which in addition to restricting queries, is available to SQL through the :code:`current_user` variable.
PostgREST can accommodate either viewpoint. If you treat a role as a single user then :ref:`user_impersonation` does most of what you need. When an authenticated user makes a request PostgREST will switch into the database role for that user, which in addition to restricting queries, is available to SQL through the :code:`current_user` variable.
You can use row-level security to flexibly restrict visibility and access for the current user. Here is an `example <https://www.2ndquadrant.com/en/blog/application-users-vs-row-level-security/>`_ from Tomas Vondra, a chat table storing messages sent between users. Users can insert rows into it to send messages to other users, and query it to see messages sent to them by other users.
You can use row-level security to flexibly restrict visibility and access for the current user. Here is an `example <https://www.enterprisedb.com:443/blog/application-users-vs-row-level-security>`_ from Tomas Vondra, a chat table storing messages sent between users. Users can insert rows into it to send messages to other users, and query it to see messages sent to them by other users.
..code-block::postgres
@@ -156,7 +156,7 @@ You can also grant execute on all functions in a schema to a higher privileged r
Security definer
----------------
A function is executed with the privileges of the user who calls it. This means that the user has to have all permissions to do the operations the procedure performs.
A function is executed with the privileges of the user who calls it. This means that the user has to have all permissions to do the operations the function performs.
If the function accesses private database objects, your :ref:`API roles <roles>` won't be able to successfully execute the function.
Another option is to define the function with the :code:`SECURITY DEFINER` option. Then only one permission check will take place, the permission to call the function, and the operations in the function will have the authority of the user who owns the function itself.
@@ -166,7 +166,7 @@ Another option is to define the function with the :code:`SECURITY DEFINER` optio
-- login as a user wich has privileges on the private schemas
@@ -180,7 +180,7 @@ Note the ``SECURITY DEFINER`` keywords at the end of the function. See `PostgreS
Views
=====
Views are invoked with the privileges of the view owner, much like stored procedures with the ``SECURITY DEFINER`` option. When created by a SUPERUSER role, all `row-level security <https://www.postgresql.org/docs/current/ddl-rowsecurity.html>`_ policies will be bypassed.
Views are invoked with the privileges of the view owner, much like functions with the ``SECURITY DEFINER`` option. When created by a SUPERUSER role, all `row-level security <https://www.postgresql.org/docs/current/ddl-rowsecurity.html>`_ policies will be bypassed.
If you're on PostgreSQL >= 15, this behavior can be changed by specifying the ``security_invoker`` option.
An external service like `Auth0 <https://auth0.com/>`_ can do the hard work transforming OAuth from Github, Twitter, Google etc into a JWT suitable for PostgREST. Auth0 can also handle email signup and password reset flows.
To use Auth0, create `an application <https://auth0.com/docs/get-started/applications>`_ for your app and `an API <https://auth0.com/docs/get-started/apis>`_ for your PostgREST server. Auth0 supports both HS256 and RS256 scheme for the issued tokens for APIs. For simplicity, you may first try HS256 scheme while creating your API on Auth0. Your application should use your PostgREST API's `API identifier <https://auth0.com/docs/get-started/apis/api-settings>`_ by setting it with the `audience parameter <https://auth0.com/docs/secure/tokens/access-tokens/get-access-tokens#control-access-token-audience>`_ during the authorization request. This will ensure that Auth0 will issue an access token for your PostgREST API. For PostgREST to verify the access token, you will need to set ``jwt-secret`` on PostgREST config file with your API's signing secret.
@@ -163,14 +169,15 @@ If you want to have a visual overview of your API in your browser you can add sw
..code-block::yaml
swagger:
image:swaggerapi/swagger-ui
ports:
- "8080:8080"
expose:
- "8080"
environment:
API_URL:http://localhost:3000/
# in services:
swagger:
image:swaggerapi/swagger-ui
ports:
- "8080:8080"
expose:
- "8080"
environment:
API_URL:http://localhost:3000/
With this you can see the swagger-ui in your browser on port 8080.
@@ -181,13 +188,9 @@ Building from Source
When a pre-built binary does not exist for your system you can build the project from source.
..note::
We discourage building and using PostgREST on **Alpine Linux** because of a reported GHC memory leak on that platform.
You can build PostgREST from source with `Stack <https://github.com/commercialhaskell/stack>`_. It will install any necessary Haskell dependencies on your system.
*`Install Stack <https://docs.haskellstack.org/en/stable/README/#how-to-install-stack>`_ for your platform
*`Install Stack <https://docs.haskellstack.org/en/stable/#how-to-install-stack>`_ for your platform
@@ -42,7 +42,7 @@ The first step is to create an Nginx configuration file that proxies requests to
HTTPS
-----
PostgREST aims to do one thing well: add an HTTP interface to a PostgreSQL database. To keep the code small and focused we do not implement HTTPS. Use a reverse proxy such as NGINX to add this, `here's how <https://nginx.org/en/docs/http/configuring_https_servers.html>`_. Note that some Platforms as a Service like Heroku also add SSL automatically in their load balancer.
PostgREST aims to do one thing well: add an HTTP interface to a PostgreSQL database. To keep the code small and focused we do not implement HTTPS. Use a reverse proxy such as NGINX to add this, `here's how <https://nginx.org/en/docs/http/configuring_https_servers.html>`_.
This creates a shared memory zone called "login" to store a log of IP addresses that access the rate limited urls. The space reserved, 10 MB (:code:`10m`) will give us enough space to store a history of 160k requests. We have chosen to allow only allow one request per second (:code:`1r/s`).
Next we apply the zone to certain routes, like a hypothetical stored procedure called :code:`login`.
Next we apply the zone to certain routes, like a hypothetical function called :code:`login`.
..code-block::nginx
@@ -73,17 +73,10 @@ Alternate URL Structure
As discussed in :ref:`singular_plural`, there are no special URL forms for singular resources in PostgREST, only operators for filtering. Thus there are no URLs like :code:`/people/1`. It would be specified instead as
..tabs::
..code-block::bash
..code-tab:: http
GET /people?id=eq.1 HTTP/1.1
Accept: application/vnd.pgrst.object+json
..code-tab:: bash Curl
curl "http://localhost:3000/people?id=eq.1" \
-H "Accept: application/vnd.pgrst.object+json"
curl "http://localhost:3000/people?id=eq.1"\
-H "Accept: application/vnd.pgrst.object+json"
This allows compound primary keys and makes the intent for singular response independent of a URL convention.
A PostgREST instance exposes all the tables, views, and stored procedures of a single `PostgreSQL schema <https://www.postgresql.org/docs/current/ddl-schemas.html>`_ (a namespace of database objects). This means private data or implementation details can go inside different private schemas and be invisible to HTTP clients.
A PostgREST instance exposes all the tables, views, and functions of a single `PostgreSQL schema <https://www.postgresql.org/docs/current/ddl-schemas.html>`_ (a namespace of database objects). This means private data or implementation details can go inside different private schemas and be invisible to HTTP clients.
It is recommended that you don't expose tables on your API schema. Instead expose views and stored procedures which insulate the internal details from the outside world.
It is recommended that you don't expose tables on your API schema. Instead expose views and functions which insulate the internal details from the outside world.
This allows you to change the internals of your schema and maintain backwards compatibility. It also keeps your code easier to refactor, and provides a natural way to do API versioning.
..image::../_static/db.png
..container::svg-container-md
..container:: img-dark
.. See https://github.com/sphinx-doc/sphinx/issues/2240#issuecomment-187366626
-:ref:`custom_media`. See :ref:`any_handler`, if you need to support an ``application/soap+xml`` media type or if you want to respond with XML without sending a media type.
This uses the :code:`byteamagic_mime()` function from the `pg_byteamagic extension <https://github.com/nmandery/pg_byteamagic>`_ to automatically generate the type in the :code:`files` table. To guess the type of a file, it's generally enough to look at the beginning of the file, which is more efficient.
Next, we set modify the function to set the content type and filename.
We use this opportunity to configure some basic, client-side caching.
For production, you probably want to configure additional caches, e.g. on the :ref:`reverse proxy <admin>`.
For production, you probably want to configure additional caches, e.g. on the :ref:`reverse proxy <nginx>`.
@@ -31,7 +31,7 @@ As in :ref:`sql_user_management`, we create a :code:`basic_auth` schema:
-- We put things inside the basic_auth schema to hide
-- them from public view. Certain public procs/views will
-- refer to helpers and tables inside.
CREATESCHEMAIFNOTEXISTSbasic_auth;
CREATESCHEMAbasic_auth;
As in :ref:`sql_user_management`, we create the :code:`pgcrypto` and :code:`pgjwt` extensions. Here we prefer to put the extensions in its own schemas:
@@ -40,21 +40,21 @@ As in :ref:`sql_user_management`, we create the :code:`pgcrypto` and :code:`pgjw
In order to be able to work with postgres' SCRAM-SHA-256 password hashes, we also need the PBKDF2 key derivation function. Luckily there is `a PL/pgSQL implementation on stackoverflow <https://stackoverflow.com/q/47162200/2337147>`_:
In order to be able to work with postgres' SCRAM-SHA-256 password hashes, we also need the PBKDF2 key derivation function. Luckily there is `a PL/pgSQL implementation on stackoverflow <https://stackoverflow.com/a/72805848>`_:
Analogous to :ref:`sql_user_management` creates the function :code:`basic_auth.user_role`, we create a helper function to check the user's password, here with another name and signature (since we want the username, not an email address).
Analogous to how :ref:`sql_user_management` creates the function :code:`basic_auth.user_role`, we create a helper function to check the user's password, here with another name and signature (since we want the username, not an email address).
But contrary to :ref:`sql_user_management`, this function does not use a dedicated :code:`users` table with passwords, but instead utilizes the built-in table `pg_catalog.pg_authid <https://www.postgresql.org/docs/current/catalog-pg-authid.html>`_:
@@ -157,25 +157,20 @@ Here we use the username instead of the email address to identify a user.
Logins
~~~~~~
As described in :ref:`client_auth`, we'll create a JWT token inside our login function. Note that you'll need to adjust the secret key which is hard-coded in this example to a secure (at least thirty-two character) secret of your choosing.
As described in :ref:`jwt-from-sql`, we'll create a JWT token inside our login function. Note that you'll need to adjust the secret key which is hard-coded in this example to a secure (at least thirty-two character) secret of your choosing.
..code-block::plpgsql
CREATETYPEbasic_auth.jwt_tokenAS(
tokentext
);
..code-block::postgres
-- if you are not using psql, you need to replace :DBNAME with the current database's name.
SELECT basic_auth.check_user_pass(username, password) INTO _role;
@@ -190,8 +185,7 @@ As described in :ref:`client_auth`, we'll create a JWT token inside our login fu
SELECT login.username as role,
extract(epoch FROM now())::integer + 60*60 AS exp
) r
INTOresult;
RETURNresult;
INTO token;
END;
$$;
@@ -259,19 +253,11 @@ Test at the REST level
~~~~~~~~~~~~~~~~~~~~~~
An API request to call this function would look like:
..tabs::
..code-block::bash
..code-tab:: http
POST /rpc/login HTTP/1.1
{ "username": "foo", "password": "bar" }
..code-tab:: bash Curl
curl "http://localhost:3000/rpc/login" \
-X POST -H "Content-Type: application/json" \
-d '{ "username": "foo", "password": "bar" }'
curl "http://localhost:3000/rpc/login"\
-X POST -H "Content-Type: application/json"\
-d '{ "username": "foo", "password": "bar" }'
The response would look like the snippet below. Try decoding the token at `jwt.io <https://jwt.io/>`_. (It was encoded with a secret of :code:`reallyreallyreallyreallyverysafe` as specified in the SQL code above. You'll want to change this secret in your app!)
@@ -296,31 +282,18 @@ Let's add a table, intended for the :code:`foo` user:
Now try to get the table's contents with:
..tabs::
..code-block::bash
..code-tab:: http
GET /foobar HTTP/1.1
..code-tab:: bash Curl
curl "http://localhost:3000/foobar"
curl "http://localhost:3000/foobar"
This should fail --- of course, we haven't specified the user, thus PostgREST falls back to the :code:`anon` user and denies access.
Add an :code:`Authorization` header. Please use the token value from the login function call above instead of the one provided below.
This will fail again --- we get :code:`Permission denied to set role`. We forgot to allow the authenticator role to switch into this user by executing:
As mentioned on :ref:`jwt_generation`, an external service can provide user management and coordinate with the PostgREST server using JWT. It’s also possible to support logins entirely through SQL. It’s a fair bit of work, so get ready.
As mentioned on :ref:`jwt_generation`, an external service can provide user management and coordinate with the PostgREST server using JWT. It's also possible to support logins entirely through SQL. It's a fair bit of work, so get ready.
Storing Users and Passwords
---------------------------
@@ -17,9 +17,8 @@ First we'll need a table to keep track of our users:
-- We put things inside the basic_auth schema to hide
-- them from public view. Certain public procs/views will
-- refer to helpers and tables inside.
createschemaifnotexistsbasic_auth;
createtableifnotexists
createtable
basic_auth.users(
emailtextprimarykeycheck(email~*'^.+@.+\..+$'),
passtextnotnullcheck(length(pass)<512),
@@ -28,9 +27,9 @@ First we'll need a table to keep track of our users:
We would like the role to be a foreign key to actual database roles, however PostgreSQL does not support these constraints against the :code:`pg_roles` table. We'll use a trigger to manually enforce it.
@@ -72,9 +69,9 @@ Next we'll use the pgcrypto extension and a trigger to keep passwords safe in th
With the table in place we can make a helper to check a password against the encrypted column. It returns the database role for a user if the email and password are correct.
@@ -113,20 +110,18 @@ Then, add ``db-anon-role`` to the configuration file to allow anonymous requests
db-anon-role="anon"
.._jwt-from-sql:
JWT from SQL
~~~~~~~~~~~~
You can create JWT tokens in SQL using the `pgjwt extension <https://github.com/michelp/pgjwt>`_. It's simple and requires only pgcrypto. If you're on an environment like Amazon RDS which doesn't support installing new extensions, you can still manually run the `SQL inside pgjwt <https://github.com/michelp/pgjwt/blob/master/pgjwt--0.1.1.sql>`_ (you'll need to replace ``@extschema@`` with another schema or just delete it) which creates the functions you will need.
Next write a stored procedure that returns the token. The one below returns a token with a hard-coded role, which expires five minutes after it was issued. Note this function has a hard-coded secret as well.
Next write a function that returns the token. The one below returns a token with a hard-coded role, which expires five minutes after it was issued. Note this function has a hard-coded secret as well.
@@ -186,8 +175,7 @@ As described in `JWT from SQL`_, we'll create a JWT inside our login function. N
select_roleasrole,login.emailasemail,
extract(epochfromnow())::integer+60*60asexp
)r
intoresult;
returnresult;
intotoken;
end;
$$languageplpgsqlsecuritydefiner;
@@ -199,19 +187,11 @@ the anonymous user :code:`anon` doesn't need permission to read the :code:`basic
An API request to call this function would look like:
..tabs::
..code-block::bash
..code-tab:: http
POST /rpc/login HTTP/1.1
{ "email": "foo@bar.com", "pass": "foobar" }
..code-tab:: bash Curl
curl "http://localhost:3000/rpc/login" \
-X POST -H "Content-Type: application/json" \
-d '{ "email": "foo@bar.com", "pass": "foobar" }'
curl "http://localhost:3000/rpc/login"\
-X POST -H "Content-Type: application/json"\
-d '{ "email": "foo@bar.com", "pass": "foobar" }'
The response would look like the snippet below. Try decoding the token at `jwt.io <https://jwt.io/>`_. (It was encoded with a secret of :code:`reallyreallyreallyreallyverysafe` as specified in the SQL code above. You'll want to change this secret in your app!)
PostgREST makes use of PostgreSQL string representations to work with data types. Thanks to this, you can use special values, such as ``now`` for timestamps, ``yes`` for booleans or time values including the time zones. This page describes how you can take advantage of these string representations to perform operations on different PostgreSQL data types.
PostgREST makes use of PostgreSQL string representations to work with data types. Thanks to this, you can use special values, such as ``now`` for timestamps, ``yes`` for booleans or time values including the time zones. This page describes how you can take advantage of these string representations and some alternatives to perform operations on different PostgreSQL data types.
..contents::
:local:
:depth:1
Timestamps
----------
You can use the **time zone** to filter or send data if needed.
..code-block::postgres
createtablereports(
idintprimarykey
,due_datetimestamptz
);
Suppose you are located in Sydney and want create a report with the date in the local time zone. Your request should look like this:
The response has the date in the time zone configured by the server: ``UTC -05:00`` (see :ref:`prefer_timezone`).
You can use other comparative filters and also all the `PostgreSQL special date/time input values <https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-DATETIME-SPECIAL-TABLE>`_ as illustrated in this example:
..tabs::
..code-tab:: http
GET /reports?or=(and(due_date.gte.today,due_date.lte.tomorrow),and(due_date.gt.-infinity,due_date.lte.epoch)) HTTP/1.1
Then, for example, to query the auditoriums that are located in the first cinema (position 0 in the array) and on the second floor (position 1 in the next inner array), we can use the arrow operators this way:
..tabs::
..code-block::bash
..code-tab:: http
GET /movies?select=title,auditorium:cinema_floor_auditorium->0->1&id=eq.1 HTTP/1.1
You can use range :ref:`operators <operators>` to filter the data. But, in this case, requesting a filter like ``events?duration=cs.2023-01-01`` will return an error, because PostgreSQL needs an explicit cast from string to timestamp. A workaround is to use a range starting and ending in the same date:
..tabs::
..code-tab:: http
GET /events?duration=cs.[2023-01-01,2023-01-01] HTTP/1.1
As you may have noticed, the ``tsrange`` value is returned as a string literal. To return it as a JSON value, first you need to create a function that will do the conversion from a ``tsrange`` type:
If you don't want to modify casts for built-in types, an option would be to `create a custom type <https://www.postgresql.org/docs/current/sql-createtype.html>`_
@@ -491,20 +121,11 @@ Let's download the PostgREST logo for our test.
Now, to send the file ``postgrest-logo.png`` we need to set the ``Content-Type: application/octet-stream`` header in the request:
..tabs::
..code-block::bash
..code-tab:: http
POST /rpc/upload_binary HTTP/1.1
Content-Type: application/octet-stream
postgrest-logo.png
..code-tab:: bash Curl
curl "http://localhost:3000/rpc/upload_binary" \
-X POST -H "Content-Type: application/octet-stream" \
--data-binary "@postgrest-logo.png"
curl "http://localhost:3000/rpc/upload_binary"\
-X POST -H "Content-Type: application/octet-stream"\
--data-binary "@postgrest-logo.png"
To get the image from the database, use :ref:`custom_media` like so:
@@ -516,17 +137,10 @@ To get the image from the database, use :ref:`custom_media` like so:
selectfilefromfileswhereid=$1;
$$languagesql;
..tabs::
..code-block::bash
..code-tab:: http
GET /get_image?id=1 HTTP/1.1
Accept: image/png
..code-tab:: bash Curl
curl "http://localhost:3000/get_image?id=1" \
-H "Accept: image/png"
curl "http://localhost:3000/get_image?id=1"\
-H "Accept: image/png"
See :ref:`providing_img` for a step-by-step example on how to handle images in HTML.
@@ -534,6 +148,104 @@ See :ref:`providing_img` for a step-by-step example on how to handle images in H
Be careful when saving binaries in the database, having a separate storage service for these is preferable in most cases. See `Storing Binary files in the Database <https://wiki.postgresql.org/wiki/BinaryFilesInDB>`_.
Composite Types
---------------
With PostgREST, you have two options to handle `composite type columns <https://www.postgresql.org/docs/current/rowtypes.html>`_.
..code-block::postgres
createtypedimensionas(
lengthdecimal(6,2),
widthdecimal(6,2),
heightdecimal(6,2),
unittext
);
createtableproducts(
idintprimarykey,
sizedimension
);
insertintoproducts(id,size)
values(1,'(5.0,5.0,10.0,"cm")');
On one hand you can insert values using string representation.
..code-block::bash
curl "http://localhost:3000/products"\
-X POST -H "Content-Type: application/json"\
-d @- << EOF
{ "id": 2, "size": "(0.7,0.5,1.8,\"m\")" }
EOF
Or you could insert the same data in JSON format.
..code-block::bash
curl "http://localhost:3000/products"\
-X POST -H "Content-Type: application/json"\
-d @- << EOF
{
"id": 2,
"size": {
"length": 0.7,
"width": 0.5,
"height": 1.8,
"unit": "m"
}
}
EOF
You can also query the data using arrow operators. See :ref:`composite_array_columns`.
Enums
-----
You can handle `Enumerated Types <https://www.postgresql.org/docs/current/datatype-enum.html>`_ using string representations:
..code-block::postgres
createtypeletter_sizeasenum('s','m','l','xl');
createtableproducts(
idintprimarykeygeneratedalwaysasidentity,
nametext,
sizeletter_size
);
To insert or update the value use a string:
..code-block::bash
curl -X POST "http://localhost:3000/products"\
-H "Content-Type: application/json"\
-d @- << EOF
{ "name": "t-shirt", "size": "l" }
EOF
You can then query and filter the enum using the compatible :ref:`operators <operators>`.
For example, to get all the products larger than `m` and ordering them by their size:
@@ -551,53 +263,67 @@ You can work with data types belonging to additional supplied modules such as `h
The ``name`` column will have the name of the country in different formats. You can insert values using the string representation for that data type:
..tabs::
..code-tab:: http
POST /countries HTTP/1.1
Content-Type: application/json
..code-block::bash
curl "http://localhost:3000/countries"\
-X POST -H "Content-Type: application/json"\
-d @- << EOF
[
{ "id": 1, "name": "common => Egypt, official => \"Arab Republic of Egypt\", native => مصر" },
{ "id": 2, "name": "common => Germany, official => \"Federal Republic of Germany\", native => Deutschland" }
]
..code-tab:: bash Curl
curl "http://localhost:3000/countries" \
-X POST -H "Content-Type: application/json" \
-d @- << EOF
[
{ "id": 1, "name": "common => Egypt, official => \"Arab Republic of Egypt\", native => مصر" },
{ "id": 2, "name": "common => Germany, official => \"Federal Republic of Germany\", native => Deutschland" }
]
EOF
EOF
Notice that the use of ``"`` in the value of the ``name`` column needs to be escaped using a backslash ``\``.
You can also query and filter the value of a ``hstore`` column using the arrow operators, as you would do for a :ref:`JSON column<json_columns>`. For example, if you want to get the native name of Egypt:
..tabs::
..code-block::bash
..code-tab:: http
GET /countries?select=name->>native&name->>common=like.Egypt HTTP/1.1
To work with a ``json`` type column, you can handle the value as a JSON object.
..code-block::postgres
createtableproducts(
idintprimarykey,
nametextunique,
extra_infojson
);
You can insert a new product using a JSON object for the ``extra_info`` column:
..code-block::bash
curl "http://localhost:3000/products"\
-X POST -H "Content-Type: application/json"\
-d @- << EOF
{
"id": 1,
"name": "Canned fish",
"extra_info": {
"expiry_date": "2025-12-31",
"exportable": true
}
}
EOF
To query and filter the data see :ref:`json_columns` for a complete reference.
.._ww_postgis:
PostGIS
-------
You can use the string representation for `PostGIS <https://postgis.net/>`_ data types such as ``geometry`` or ``geography`` (you need to `install PostGIS <https://postgis.net/install/>`_ first).
You can use the string representation for `PostGIS <https://postgis.net/>`_ data types such as ``geometry`` or ``geography`` (you need to `install PostGIS <https://postgis.net/documentation/getting_started/>`_ first).
..code-block::postgres
@@ -612,42 +338,23 @@ You can use the string representation for `PostGIS <https://postgis.net/>`_ data
To add areas in polygon format, you can use string representation:
Now, when you request the information, PostgREST will automatically cast the ``area`` column into a ``Polygon`` geometry type. Although this is useful, you may need the whole output to be in `GeoJSON <https://geojson.org/>`_ format out of the box, which can be done by including the ``Accept: application/geo+json`` in the request. This will work for PostGIS versions 3.0.0 and up and will return the output as a `FeatureCollection Object <https://www.rfc-editor.org/rfc/rfc7946#section-3.3>`_:
..tabs::
..code-block::bash
..code-tab:: http
GET /coverage HTTP/1.1
Accept: application/geo+json
..code-tab:: bash Curl
curl "http://localhost:3000/coverage" \
-H "Accept: application/geo+json"
curl "http://localhost:3000/coverage"\
-H "Accept: application/geo+json"
..code-block::json
@@ -711,15 +418,9 @@ In the case that you are using older PostGIS versions, then creating a function
You can use range :ref:`operators <operators>` to filter the data. But, in this case, requesting a filter like ``events?duration=cs.2023-01-01`` will return an error, because PostgreSQL needs an explicit cast from string to timestamp. A workaround is to use a range starting and ending in the same date:
As you may have noticed, the ``tsrange`` value is returned as a string literal. To return it as a JSON value, first you need to create a function that will do the conversion from a ``tsrange`` type:
If you don't want to modify casts for built-in types, an option would be to `create a custom type <https://www.postgresql.org/docs/current/sql-createtype.html>`_
The response has the date in the time zone configured by the server: ``UTC -05:00`` (see :ref:`prefer_timezone`).
You can use other comparative filters and also all the `PostgreSQL special date/time input values <https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-DATETIME-SPECIAL-TABLE>`_ as illustrated in this example:
PostgREST is a standalone web server that turns your PostgreSQL database directly into a RESTful API. The structural constraints and permissions in the database determine the API endpoints and operations.
.. The static/empty.png(created with `convert -size 320x95 xc:#fcfcfc empty.png`) is an ugly workaround
to create space and center the logos. It's not easy to layout with restructuredText.
.. .. image:: _static/empty.png
:target: #sponsors
:width: 13em
.. image:: _static/empty.png
:target: #sponsors
|
@@ -92,12 +111,19 @@ PostgREST has a focused scope. It works well with other tools like Nginx. This f
Getting Support
----------------
The project has a friendly and growing community. For discussions, use the Github `discussions page <https://github.com/PostgREST/postgrest/discussions>`_ or join our `chat room <https://gitter.im/begriffs/postgrest>`_. You can also report or search for bugs/features on the Github `issues <https://github.com/PostgREST/postgrest/issues>`_ page.
The project has a friendly and growing community. For discussions, use the Github `discussions page <https://github.com/PostgREST/postgrest/discussions>`_. You can also report or search for bugs/features on the Github `issues <https://github.com/PostgREST/postgrest/issues>`_ page.
Release Notes
-------------
Releases
--------
The release notes are published on `PostgREST's GitHub release page <https://github.com/PostgREST/postgrest/releases>`_.
3. Create a new Heroku PostgreSQL add-on attached to the app and keep notes of the assigned add-on name (e.g. :code:`postgresql-curly-58902`) referred later as ${HEROKU_PG_DB_NAME}
..code-block::bash
heroku addons:create heroku-postgresql:standard-0 -a ${YOUR_APP_NAME}
# wait until the add-on is available
heroku pg:wait -a ${YOUR_APP_NAME}
4. Create the necessary user roles according to the
Set the following environment variables on Heroku:
.. code-block:: bash
heroku config:set POSTGREST_VER=10.0.0
heroku config:set PGRST_DB_SCHEMA=api
heroku config:set PGRST_DB_ANON_ROLE=api_user
..
PGRST_DB_URI can be setif an external database is used or if it's different from the default Heroku DATABASE_URL. This latter is used if nothing is provided.
POSTGREST_VER is mandatory to select and build the required PostgREST release.
See https://postgrest.org/en/stable/configuration.html#environment-variables for the full list of environment variables.
7. Build and deploy your app:
..code-block::bash
git add Procfile
git commit -m "PostgREST on Heroku"
git push heroku master
..
Your Heroku app should be live at :code:`${YOUR_APP_NAME}.herokuapp.com`
8. Test your app
From a terminal display the application logs:
..code-block::bash
heroku logs -t
..
From a different terminal retrieve with curl the records previously created:
.. code-block:: bash
curl https://${YOUR_APP_NAME}.herokuapp.com/todos
..
and test that any attempt to modify the table via a read-only user is not allowed:
.. code-block:: bash
curl https://${YOUR_APP_NAME}.herokuapp.com/todos -X POST \
An external service like `Auth0 <https://auth0.com/>`_ can do the hard work transforming OAuth from Github, Twitter, Google etc into a JWT suitable for PostgREST. Auth0 can also handle email signup and password reset flows.
To use Auth0, create `an application <https://auth0.com/docs/get-started/applications>`_ for your app and `an API <https://auth0.com/docs/get-started/apis>`_ for your PostgREST server. Auth0 supports both HS256 and RS256 scheme for the issued tokens for APIs. For simplicity, you may first try HS256 scheme while creating your API on Auth0. Your application should use your PostgREST API's `API identifier <https://auth0.com/docs/get-started/apis/api-settings>`_ by setting it with the `audience parameter <https://auth0.com/docs/secure/tokens/access-tokens/get-access-tokens#control-access-token-audience>`_ during the authorization request. This will ensure that Auth0 will issue an access token for your PostgREST API. For PostgREST to verify the access token, you will need to set ``jwt-secret`` on PostgREST config file with your API's signing secret.
..note::
Our code requires a database role in the JWT. To add it you need to save the database role in Auth0 `app metadata <https://auth0.com/docs/manage-users/user-accounts/metadata/manage-metadata-rules>`_. Then, you will need to write `a rule <https://auth0.com/docs/customize/rules>`_ that will extract the role from the user's app_metadata and set it as a `custom claim <https://auth0.com/docs/get-started/apis/scopes/sample-use-cases-scopes-and-claims#add-custom-claims-to-a-token>`_ in the access token. Note that, you may use Auth0's `core authorization feature <https://auth0.com/docs/manage-users/access-control/rbac>`_ for more complex use cases. Metadata solution is mentioned here for simplicity.
If the :ref:`active role <user_impersonation>` can delete table rows then the DELETE verb is allowed for clients. Here's an API request to delete old rows from a hypothetical logs table:
Note that it's very easy to delete the **entire table** by omitting the query parameter!
..tabs::
..code-block::bash
..code-tab:: http
DELETE /logs HTTP/1.1
..code-tab:: bash Curl
curl "http://localhost:3000/logs" -X DELETE
curl "http://localhost:3000/logs" -X DELETE
This can happen accidentally such as by switching a request from a GET to a DELETE. To protect against accidental operations use the `pg-safeupdate <https://github.com/eradman/pg-safeupdate>`_ PostgreSQL extension. It raises an error if UPDATE or DELETE are executed without specifying conditions. To install it you can use the `PGXN <https://pgxn.org/>`_ network:
For Linux distributions that use **systemd** (Ubuntu, Debian, Archlinux) you can create a daemon in the following way.
For Linux distributions that use **systemd** (Ubuntu, Debian, Arch Linux) you can create a daemon in the following way.
First, create postgrest configuration in ``/etc/postgrest/config``
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.