Compute per-row editability and deletability from a table's row-level
security policies and return them as synthetic columns so clients can
hide edit/delete affordances for rows the user cannot change.
- Introspect pg_policies and relrowsecurity at schema-cache load and
combine the UPDATE/DELETE USING qualifiers per table (permissive OR,
restrictive AND).
- Store the combined qualifiers on Table and inject can_edit/can_delete
as computed select fields when expanding `select *`, only for
RLS-enabled tables with a matching policy (COALESCE'd to a boolean).
- Keep the computed columns out of the OpenAPI spec so they are not
rendered as regular fields.
- Add a cfExpression field to CoercibleField to carry raw SQL
expressions through the planner to SqlFragment.
Add unique constraint and many-to-many relationship metadata to the
generated OpenAPI spec so clients can render them.
- Store unique constraints on Table as tableUniqueCols (mirroring
tablePKCols) instead of denormalizing them onto each Column.
- Compute unique constraints via a per-table tbl_unique_cols CTE in
tablesSqlQuery.
- Annotate unique columns and composite unique constraints in property
descriptions, and emit m2m markers in table descriptions.
We had just 3 tests remaining in test_io.py. This commit moves them to
their modules. So we have:
* test_graceful_shutdown.py
* test_zero_downtime.py
* test_pg_internal.py
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
(cherry picked from commit 02d83d1c01)
Upgrade auto-update to 0.2.7 which contains a fix to a bug causing some threads not seeing updates to the cached values.
(cherry picked from commit a2a592a617)
This is to make it easier for users to understand the changes,
much like PostgreSQL does for its changelog: https://www.postgresql.org/docs/19/release-19.html#RELEASE-19-SERVER
(see how they classify "Optimizer", "General Performance", etc).
We classify features, fixes and deprecation in subsections.
Breaking changes are put inside a "Migration to v16" guide.
(cherry picked from commit e3206262d2)
Hiding this will also hide errors when they happen...
In my case, my nix-shell's TMPDIR was too long, so black was failing
silently because it couldn't create a unix socket for its forkserver.
`AppState.hs` is imported by many modules. Some of those modules contain
important functions that need to be imported back to `AppState.hs`, causing
circular dependency problem.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
This gives the import statements more space, essentially, it collectively
removes 90+ unneeded lines in the imports statements.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
When running `postgrest-test-io`, pytest raises a `InsecureKeyLengthWarning`
for a test. To clear that warning, this commit increases the key length to
more than 32 characters.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
`postgrest-build` now builds twice as fast as compared to previous
build time. This can be tested by running `rm -rf dist-newstyle/` and
then run `postgrest-build`.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
src/ now contains all source code - in subdirectories, according to the
.cabal component they belong to. This will allow us to put vendored
libraries in the same place - and later split our own code into multiple
components/libraries as well.
Runs the doctests much faster, which is potentially useful in
combination with postgrest-watch for local development.
This implies that doctests run on compiled code, not in a GHCi session,
which has some implications:
- Only exported functions can be tested.
- Imports need to be made explicit in doctests themselves.
On the flipside, this would allow us to potentially include doctest
results in code coverage, I believe.
This change is a requirement to vendor hasql, which otherwise breaks the
existing doctests: hasql contains a .hsc file, which *needs* to be
compiled - not interpreted - to make the tests work.
Adds the `url_use_legacy_target_names` config.
Enabled (default):
* It allows using the resource name in filters,
orders or limits when it has an alias, e.g.
`table?select=alias:target(*)&target.id=eq.1`
* Logs a WARNING with a hint to use the alias
* Returns a Warning header in the response
Disabled:
* It returns an error, only the alias is allowed
* No warnings returned
This feature is deprecated
This change introduces a new way to perform liveness check - instead of trying to open a TCP connection to main server, we check if main socket is listening and if main server accept loop thread is alive.
Opening a TCP connection in liveness check was problematic because:
* it used available file descriptors which might have been a problem under load
* made liveness check unreliable when multiple PostgREST instances are available on the same port (eg. using SO_REUSEPORT)
Right now code in App.run does not properly use bracket/finally to close sockets and clean-up mainSocketRef. This is not a big problem at the moment because the application is going to exit enyway but introducting proper resource handling will make future refactorings safer.
This change makes AppState and Listener modules independent from the way how application is terminated. It removes stateMainThreadId from AppState and introduces stateKillApp. It also removes exported function AppState.getMainThreadId replacing it with more general killApp.
This change makes Admin module independent from how liveness check is performed. It moves reachMainApp function to App module and makes Admin.runAdmin, instead of action to get the main socket, take an IO action checking liveness as a parameter.
Thanks to this, any change in how liveness check is performed and, more importantly, what data it uses, does not require changing multiple modules (Admin and App).
Issue: the example epoch date is wrapped in parentheses. This causes the server to return PGRST303: "The JWT 'exp' claim must be a number"
Fix: remove parentheses
Admin server was crashing silently and requests hang indefinitely.
With this fix, we are now logging the admin server exceptions and also
close the socket afterwards so admin request don't hang.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Instead of generating them at loadtest runtime, we generate these
dynamic targets once via Nix. Because we're using libfaketime, we're not
bound to doing it at the same time as running the tests.
This allows us to run each loadtest right after each other, with minimal
other stuff going on inbetween. It's also a tiny bit more reproducible,
because the loadtests against both branches run with exactly the same
set of targets, where they did not before.
Instead of building, running, building, running, ... we now build all
executables once ahead of time and then run all loadtests right after
each other. This can sometimes reduce noise when load on the GHA runner
varies over time.
Since this requires us to move building into the loadtest-against
script, it also allows to go back to have the regular postgrest-loadtest
command default to building with cabal for faster local iteration.
I'm not sure whether anyone is actually using them, but they never
worked really well, I believe. We should add something based on an
existing pre-commit hook manager, not that brittle custom thing we have.
Allows me to remove postgrest-with-git in the next commit instead of
duplicating it.
This is a heavily discouraged way of using these tools - they will
become outdated almost immediately and would need to be reinstalled on
every change to main.
This list is chronically outdated and hard to maintain with the two
column layout. Simplify and reduce, not important to keep up-to-date to
bring the point across.
We don't need to restrict us to run every request once. Instead we can
run as many as possible within 60s by removing the `-lazy` flag.
This also allows us to generate fewer targets, because vegeta will do
the repetition for us.
This test is not really functional (anymore?). It depends on some tokens
expiring - but they don't, really. This is because:
1) Before merging the RSA and HS variants earlier in this patch series,
they ran roughly 15s and 34s respectively in CI.
2) Since PostgREST takes a clock skew of up to 30 seconds into account,
there would be *at most* 4s of expiring tokens in a test run - and
*only* in the HS case, not for RSA.
While we didn't have much problem in this test, it's surely not a
problem either to make the test more reproducible with a fixed base
timestamp. This gives us a tiny bit less of a diff on the next change.
This test is not very interesting, really: It's hard to imagine to be
able to produce a regression that would slow down the no-cache case, but
would *not* show when a cache is used.
Let's use the same dependency when creating key material and when
creating tokens, no need to carry multiple different dependencies and
converting keys between them.
Instead of passing a path to the output target file, just pass the
directory in which to create the file. This allows the script to create
multiple files later.
When we changed the test structure to use libfaketime, we moved away
from generating targets as part of the chained command running
with-pg/with-pgrst. This function was left-over from that, the
genTargets script does not need to relay its remaining arguments
arguments anymore.
Instead of creating separate test suites for the key type, the PostgREST
instance now accepts both keys via a JWKSet and the targets are
generated 50/50 for both.
The results are still reported seperately by using a different URL,
which shows up as separate rows in the results.
The script currently depends on results from "head" and "main" to be
present to calculate the change ratio. Fallback nicely in case these are
not available, which allows using the same to report URL-by-URL stats
locally.
BREAKING CHANGE
Breaks the string comparison operators implemented in #3813. Those can
be replaced with regex searches using JSON Path `search()` function.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Instead of starting a separate process, which would make the pid we
store the one from `faketime` instead of `postgrest`, we can also just
start libfaketime directly via LD_PRELOAD.
For admin server, port server-setting was set to the same port as
app server port. This doesn't have any effect because it is overridden
by the already initialized admin server socket, but it's better to
explicitly correct it to avoid anything unexpected.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
There is no need to store the out-links to the various build artifacts
in the current working directory while running postgrest-with-pg, which
is used by postgrest-loadtest. Instead reference the nix store paths
directly.
Resolves#5028
Before 9.12.1 it was necessary to mark functions as INLINABLE or INLINE to make GHC consider cross-module specialization of polymorphic functions.
9.12 added a new -fexpose-overloaded-unfoldings flag that exposes optimized polymorphic functions in interface files. -fspecialise-aggressively then makes GHC apply aggresive specialization.
This change ensures PostgREST starts listening on a server socket only after it loaded the schema cache and is ready to handle requests. It is no longer going to return 503 errors during startup until the schema cache is loaded.
Right now stateNextListenerDelay is only used in Listener module.
This change is a small refactoring moving nextListenerDelay to Listener to increase cohesion and decrease coupling.
This is useful when multiple instances run on the same machine, for
example behind a proxy. Unix sockets for web and admin servers can then
be put in the same folder for each instance.
Can be helpful when writing tests as well.
The `if response` check a few rows down would not actually work without
initializing the variable first. It'd throw:
```
E UnboundLocalError: cannot access local variable 'response' where
it is not associated with a value
```
retryingSchemaCacheLoad should not clear existing schema cache upon failure - there is no reason to do that. If there is a communication issue with the database server or db is down, clients are going to get 502 anyway. If it was a glitch when loading the schema cache - the clients are going to use old (stale) schema cache for some time until next retry re-loads it successfully.
There was an oversight on d556cea, `timeout_authenticator` doesn't have
any privileges on the tables so it shouldn't be making requests to the
API server or it will end up with 401 responses.
ghcr does not support uppercase letters, which was not a problem when
testing in my fork. But the PostgREST org's name obviously has uppercase
letters in it, so we need to lowercase these for the ghcr namespace.
This reverts commit 840d3f99ed.
This was a nice idea in theory, but in practice this means that the
build jobs do not run at all on the default branch anymore. This means
the caches they push to won't be used by any other jobs, neither on the
tag pipelines, nor on PRs. This in turn makes all of these jobs really
slow.
Not helpful!
Replaces the `Maybe [Text]` with `[Text]`. The `Maybe` is unnecessary
because we handle `Just []` and `Nothing` the same way.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Previously this would only fail if *each* row had `:x:` in it, which was
never the case, because the header never has it. It is supposed to fail
when *at least one* row has an ❌.
This also requires enabling shared_preload_libraries=pg_stat_statements.
Really we only need compute_query_id=on, but pg_stat_statements is
useful for other tests and it already enables compute_query_id (since
it's `auto` by default).
The PGRST000 database connection error message was "Database connection
error. Retrying the connection.", but reconnection attempts are already
logged separately by the reconnection observation, and on fatal errors
(e.g. authentication failure) PostgREST does not retry at all. Drop the
"Retrying the connection." part, leaving "Database connection error.".
The config dump in Config.hs listed db-config, db-pre-config and the
db-pool-* settings out of alphabetical order, while the rest of the dump
was sorted. Now the whole dump is consistently alphabetical and update
the expected IO test configs to match.
This was brought over in the last commit, because the postgis was
re-using the same config as another test. Now it has a separate config,
so we can reduce it a bit.
Previously, information about each test-suite was repeated in 3 separate
places:
- as a label and as implicit knowledge in the test-suite itself,
- as a comment in Main.hs, and
- as a configuration in SpecHelper.hs.
With this change, there will be a single source of truth in the test
suite itself. This will allow a single test-suite to easily test
multiple different configurations.
Currently, authentication and response execution each unwrap ExceptT with separate runExceptT calls, which split the main request flow across nested pattern matching and Either handling. Control flow is complex and difficult to understand.
The goal of this change is to make request execution as sequential
monadic code with clear error handling.
To implement that, request handling is now run in ExceptT over WriterT (Last ByteString) IO monad stack. Auth role is written after authentication succeeds and further returned along the response. Thanks to it response observation generation is centralized at the end of request handling.
It was necessary to abstract monad stack in getAuthResult, lookupJwtCache, postgrestResponse, and withTiming to enable introduction of WriterT.
This reports the percentage change between the current head branch and
the main branch, which is exactly the number we'll want to make our
decisions on "success or fail" on.
CI failures will initially be reported for regressions of 5% or more on
an individual number.
Because we separate loadtest results per URL now, we can move the error
tests into the regular mixed bag of loadtests - we will be able to tell
from the misspelled URLs when we hit a regression in that area.
We should be able to do similar things for JWT tests, but we'll need
more infrastructure here.
We previously used "rate", i.e. number of requests per second, as the
primary metric to judge loadtest results. However, this has always been
varying from run to run quite a bit, especially in CI where other jobs
possibly run on the same VM host.
The run-to-run variance has massively increased after splitting the
results up per request. Example run in CI with rate on the PR
introducing this change (on which we would expect no change at all):
| rate [1/s] | main | head | Δ |
|:-----------------------------------|-------:|-------:|-----:|
| / | 9.4 | 9.5 | 1% |
| /actors | 870.4 | 1023.0 | 18% |
| /actors?actor=eq.1 | 188.5 | 198.6 | 5% |
| /actors?actor=eq.1&columns=name | 197.3 | 167.1 | -15% |
| /actors?select=*,roles(*,films(*)) | 153.9 | 144.9 | -6% |
| /films?columns=id,title | 157.9 | 182.6 | 16% |
| /films?columns=id,title,year,... | 87.0 | 87.1 | 0% |
| /roles | 204.5 | 267.3 | 31% |
| /rpc/call_me | 231.3 | 208.8 | -10% |
| /rpc/call_me?name=John | 212.2 | 201.7 | -5% |
From the data we can easily tell that the very reason that rate as a
paramter has only worked, so far, because the data was *heavily*
dominated by the requests on the root endpoint for OpenAPI. The longer
duration makes the request much less vulnerable for concurrent activity.
For all other requests its essentially not possible to judge the effect
of a PR this way.
One way to counter this would be to massively increase the time the
loadtest runs. More samples will result in a smoother average. However,
that's not practical for usability of CI. In the original PR #1812 I
already evaluated using the *minimum latency* as the most reliable
criteriumi, but this has never really caught on. The theory behind this
is: The variation in timings between requests is happening because of
concurrent activity, priority chosen by the scheduler, availability of
resources and such - all factors *outside* our control, and *irrelevant*
to the Haskell code we're writing.
Using the minimum latency is an estimation of how fast the code can run
*in the best case*. This might not be a number relevant for production,
but it's much more directly related to the code we write.
Here's to show how variation becomes *much* smaller with minimum latency
as the parameter:
| min latency [μs] | main | head | Δ |
|:-----------------------------------|---------:|-------:|-----:|
| / | 1275.3 | 1263.6 | -1% |
| /actors | 10.0 | 9.9 | -1% |
| /actors?actor=eq.1 | 50.7 | 48.3 | -5% |
| /actors?actor=eq.1&columns=name | 54.1 | 54.0 | 0% |
| /actors?select=*,roles(*,films(*)) | 63.2 | 61.9 | -2% |
| /films?columns=id,title | 51.1 | 50.7 | -1% |
| /films?columns=id,title,year,... | 121.9 | 121.8 | 0% |
| /roles | 42.9 | 42.6 | -1% |
| /rpc/call_me | 45.6 | 45.4 | 0% |
| /rpc/call_me?name=John | 44.4 | 44.2 | 0% |
Since we're separating results per request now, we can only sensibly
focus on *one* parameter - otherwise this would get really clunky
UI-wise. Especially for automated CI failures, minimum latency is the
logical choice.
This commit starts using minimum latency, i.e. P0, but any percentile
should be an improvement over the status quo. A later commit will change
to a different P-value.
Different requests hit different code paths and perform very
differently. By looking at each request type separately, we should be
able to get a much better idea of what kind of change in performance
we're looking at and where the root cause might be.
It will hopefully also allow us to migrate some of the other test-cases
into the main loadtest.
Ultimately, we only look at the `rate` column, so we can just as well
remove all other columns.
This makes the next step, when we split results by request type, much
less noisy.
This change makes the API surface between MainTx and App smaller.
Currently, App reconstructs a database transaction by unpacking the isolation
level, transaction mode, DbHandler, and transaction runner returned by MainTx.
That exposes MainTx internals at the call site even though MainTx already owns
query setup, execution, decoding, and rollback behavior.
The goal is to keep transaction assembly in MainTx while App remains responsible
for pool execution, database error mapping, and response orchestration. DbTx now
carries the assembled SQL session, and App passes that session directly to the
connection pool.
Schema cache query timings are only needed immediately after a schema-cache reload to emit SchemaCacheQueriedObs. Storing them inside SchemaCache makes the cache carry transient observability data that is not part of the cached schema state and is never used by request handling.
This change makes querySchemaCache to return query timings in a tuple in parallel to SchemaCache and removes dbQueryTimings field.
Config variables are tested already via reading the config files in
the `configs/` directory.
If more are to be tested, it should be done via adding a file in
`configs/` and compare it with its associated file in `configs/expected/`.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
This change gets rid of unnecessary explicit bindRandomPortTCP in initServerSocket. Returned port value was ignored in removed code anyway as assigned port retrieval from an open socket is handled elsewhere.
Remove internal schema cache load and relationship load sleep settings plus
the delay wrappers they enabled. Drop IO tests that depended on the removed
settings.
Add an IO test that drops a table while schema cache reload is delayed. It verifies the stale cache path returns PostgreSQL 42P01 and the refreshed cache returns PGRST205.
Instead of taking wild guesses at the runtime of the target generation
itself, we're just making sure to reset the system time to a fixed value
when we ultimately start PostgREST. This allows us to create the right
JWT expiry values ahead of time.
We decided against doing this in #4913, therefore removing it. With that,
also mentioning that control flow never reaches there to avoid confusion.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
No need to do this in as many jobs. Splitting all the github stuff from
docker stuff into two jobs is enough. This still allows to conditionally
enable docker jobs in contributors repos, depending on whether the
relevant docker credentials are provided - but avoids using too many
concurrent runners.
Right now metrics observation handler does not track database connections but updates a single Gauge based on HasqlPoolObs events. This is problematic because Hasql pool reports various connection events in multiple phases. The connection state machine is not simple and to precisely report the number of connections in various states, it is necessary to track their lifecycles.
This change adds a ConnTrack data structure and logic to track database connections lifecycles. At the moment it supports "connected" and "inUse" connection counts precisely. The "pgrst_db_pool_available" metric is implemented on top of ConnTrack instead of a simple Gauge.
This was required for v9 and earlier, but these don't build with the
current nix invocation anymore anyway. Even loadtesting against v10 does
not work, because `--version` is used in one of the wait scripts and
this was only added in v11.2.
So no need to pretend we'd support comparing against older versions.
The functions `drain_stdout` and `match_log` should be in `util.py`
so they can be reused in other modules.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
No need to spin up full VM runners for small automation tasks, when we
can use single-CPU runners in containers instead.
https://docs.github.com/en/actions/reference/runners/github-hosted-runners#single-cpu-runners
(some of this will potentially not work, because dependencies in the
slim image might not be available - however, it makes no sense to create
this as a PR, because all jobs touched here run on branches only. Thus
pushing directly to main)
Disabling the autovacuum daemon should also help reproducibility in
theory, although I don't know of any cases where we hit a problem with
that.
VACUUM changes the order of rows that PostgreSQL returns for some table
without explicit ordering, thus doing the latter to make it consistently
reproducible.
After ANALYZE estimates are 100% exact for the moment, so some requests
which returned 206 Partial Response now return 200 instead. The fact
that PostgREST returns 206 on an unfiltered endpoint can probably be
considered a bug.
Proves the failure on https://github.com/PostgREST/postgrest/issues/4622.
This doesn't require additional test infra, only nginx. Taking advantage
of the `stream {}` context which is also compatible with unix socket
besides TCP.
The most likely offender to fail early is jwt-rsa-cache-worst - which
then cancels *all* loadtests at once. This is unnecessary churn for CI.
Instead we can re-run only that one test later.
The scheduled linkcheck job fails without notification to maintainers.
To catch the failure, a github workflow step has been added which
comments on issue #4106 so that the maintainers get notified.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
These fail with orioledb, because it randomly returns a different order.
Of course, that's allowed - ordering is not guaranteed unless specified.
We should write our tests with more explicit ordering in general, even
if multiple major versions show the same behavior.
This commit removes auth middleware for it hides
side effects and obscures logic. The auth operations
are now done in its own stage in the request-response
cycle.
It also removes the logging middleware because now
we instead use observation module to log the response.
This adds a CI job to run our latest postgrest version against the NixOS
VM test currently available in Nixpkgs. Now, this will not always be in
sync, so has the potential to be failing. However, since the Nixpkgs VM
test is really simple, this should only happen when we introduce a
breaking change on a very fundamental level. The failing test will then
be resolved once the new version is available in Nixpkgs and we have
updated our lock file.
This is essentially just a sanity check to make sure we're not breaking
something fundamentally - and if we do, it's a head up for me to adjust
the Nixpkgs tests accordingly. Those might otherwise break unnoticed
since Nixpkgs does not have a good notification system for such
breakages in place.
We do use the chance to run the static executable in this test, which
was previously not tested at all.
It also gives us a first test whether NixOS VM tests work well in GitHub
Actions.
So we can do `PGRST_DB_ANON_ROLE="" postgrest-run` since empty string has
meaning. argbash conflates both unset and empty string so this requires
a workaround.
Renames the "Functions" in the logs to "RPCs". This clarifies that
we log number of callable functions and not the number of SQL objects.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
I doubt that anyone would use cabal2nix or update-nix-fetchgit manually
- and at the same time everyone surely has git installed through their
OS anyway, otherwise they can't even get that far.
There is no point to actually realize the derivations to disk, when they
are in cache. `nix-build-uncached` checks the cache and stops before
downloading dependencies if they are available. It will however build as
normal on a cache-miss. Should speed up CI when no Nix dependencies
change a fair bit.
We can only do this for the MacOS job, which only tests the builds. We
can not do this for other jobs which require the build results to be
present before uploading parts of them as artifacts.
PostgREST failed when querying role settings where current
role name contained uppercase letters. This commit resolves
it by quoting the CURRENT_USER.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
These should have been updated by postgrest-nixpkgs-upgrade, when I
upgraded the Nixpkgs pin, but I did not follow my own readme, and just
ran nix flake update...
The table not found error only generates hint when the total number of
tables in a schema are less than 500.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Upgraded warp to 3.4.13 which fixed https://github.com/yesodweb/wai/issues/853
Changed interrupt handling so that instead of killing the main thread, listening sockets are closed which triggers warp graceful shutdown.
My recent commit to fix the stack cache on macOS only changed the
restore keys, but no cache would ever be saved that way. I effectively
disabled all caching...
Unfortunately, `docker manifest create` does not support image digests,
so we need to push a tag for each version of the image. Furthermore, the
docker CLI does not provide a way to delete that tag afterwards, so we
will end up with those temporary tags.
We now have two stack jobs for macOS, but they use the same cache, which
is very inefficient. Adding the architecture to the cache key will give
each of these jobs a separate cache.
Jumping past GHC 9.6, 9.8 and 9.10 immediately, because these versions
do not successfully build the static binary with TemplateHaskell and
iserv-proxy, yet.
Michal wrote the changes to hsie.
Co-authored-by: Michal Kleczek <michal@kleczek.org>
These don't really make sense, because it's non-trivial to construct a
build plan that *ensures* the dependency that renovate creates a PR for
is tested with the latest version. Cabal might just select a build plan,
that doesn't include the dependency, in which case we could just as well
have no upper bounds at all.
Since there is no good way to test these changes (I tried, a while
ago!), let's just disable these PRs. They are more annoying than
anything else.
I'm not sure whether that's Lix specific, but when I enter nix-shell, I
get quite a long TMPDIR, which results in all postgrest-with-pg-XX
scripts failing, because the unix socket path is too long. If TMPDIR was
not set, mktemp would fall back to /tmp, so we can just do that in any
case, which guarantees a short path in every case.
This adds a new log line that shows each schema cache query time individually, only on
`log-level=debug`. Like so:
```
$ PGRST_LOG_LEVEL=debug postgrest-with-pg-17 -f test/spec/fixtures/load.sql postgrest-run
....
10/Apr/2026:21:48:45 -0500: Schema cache queried in 192.2 milliseconds
10/Apr/2026:21:48:45 -0500: tables: 72.027 ms, keydeps: 20.118 ms, rels: 6.189 ms, funcs: 35.010 ms, comprels: 4.319 ms, dreps: 1.614 ms, mhandlers: 7.419 ms, tzones: 43.025 ms
```
This helps debug specific schema cache queries being slow like on
https://github.com/PostgREST/postgrest/issues/4613#issuecomment-4210191065 and
https://github.com/PostgREST/postgrest/issues/3046#issuecomment-3469059948.
It also closes https://github.com/PostgREST/postgrest/issues/3215,
which main motivation was to find out which query is slow.
Implementation details
---------------------
To time each query inside a transaction in pure SQL, we do:
```sql
-- start timer
select set_config('pgrst.tmp_x', clock_timestamp()::text, false);
-- run the query
select <query>
-- end timer
select set_config('pgrst.tmp_x', (clock_timestamp() - current_setting('pgrst.tmp_x', false)::timestamptz)::text, false);
-- .... repeated for every query
-- at the end we capture all the timings with
select extract('milliseconds' from current_setting('pgrst.tmp_x', false)::interval), extract(..;
```
Considerations
--------------
Only added this on `log-level=debug` because while the queries are fast
and the data is valuable, it triples the amount of queries we run during schema cache refresh,
which could be troublesome on slow networks. It's possible to reduce the
amount of queries by starting and stopping timers in one statement, but
this would still double the amount of queries and makes the code messy,
doesn't seem worth it.
Also it would pollute pg_stat_statements, it's only required to debug certain
extreme cases anyway.
retryingSchemaCacheLoad flushes the pool upon every retry before it starts reloading the schema. This is too early as schema reloading might take some time during which new connections might be acquired. The consequence is that:
* upon successful schema cache reload we might have some connections created with the old schema cache
* we close connections upon each retry and under load we will keep closing and re-opening connections until schema cache load succeeds
This change is to make sure we flush the pool only after successful schema cache querying but before loading (so that connections acquired during loading wait for it and do not interfere with timing the loading process).
The current version is too old and it prevented using latest features
like common stanzas and visibility fields.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
This change introduces a match_log function, that implements lenient matching of log output lines. Tests of log output are retrofited to use the function.
DISCLAIMER:
This commit was authored entirely by a human without the assistance of LLMs.
Using debouncer to trigger schema cache loading makes it difficult to understand when exactly it is triggered.
Fixed integer type mapping in OpenAPI 2.0: replaced the invalid integer format with int32/int64 and added the toSwaggerFormat function to map PostgreSQL types to valid OpenAPI 2.0 formats:
smallint -> int32
integer -> int32
bigint -> int64
To avoid repeated querying of `pg_timezone_names` every time schema
cache is reset, `Prefer: timezone` can be disabled by setting
`db-timezone-enabled = false`.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Some helpers are provided for introspecting metrics already (used in JWT cache tests). This change provides facilities to additionally validate emited Observation events.
A new Spec module is also implemented, adding basic tests of schema cache reloading - their main goal is to excercise the new infrastructure.
These files were added to cover the config dumping of
string comparison operators in `JSPath`. Renaming and
adding a comment in these to make it clear.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
- Create separate test suite for observability tests
- Create wrapper script `postgrest-test-observability`
- Add to CI and `postgrest-check`
- Move JWT cache tests under observability tests
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Also add a comment on `AuthResult` type to explain that
we can also access the claims in the db using GUCs.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Right now listening sockets initialization, management and usage is split between App, AppState and Admin modules: they are created in AppState.init and remembered in AppState but used only in App and Admin.
It has several negative consequences:
- sockets are initialized even if not needed (eg. command line invocations like dump-config or dump-schema)
- it is impossible to start listening on a socket after initial schema cache load because it requires AppState
This change decouples listen socket management from AppState. Sockets are created only when needed (ie. not in command line tools invocation) and passed to admin application and to Warp by the App module.
This commit replaces ioRef based implementation of schema cache status tracking to MVar based, so that it is possible to wait for schema cache loading.
Waiting for schema cache loading is necessary to implement zero-downtime upgrades with SO_REUSEPORT, where listening on a socket must wait for schema cache loading.
Towards #4608.
We already have access to these parameters via `ApiRequest` type,
hence no need to pass them separately.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
To preserve our scarce review capacity, we reserve the right to reject
seemingly AI generated low-value contributions.
---
Also delete the first paragraph since now it can be interpreted as
low-effort contributions.
This commit addresses the following TODO in observationMessage function:
"TODO pending refactor: The logic for printing the query cannot be done here. Join the observationMessages function into observationLogger to avoid this mempty."
Changing observationMessage to observationMessages (that returns [Text] instead of Text) made it possible to address issue documented as TODO and harmonize query logging with other observations.
There is unnecessary coupling between observation messages and emited log entries. This causes schema loading logic to emit redundant events: SchemaCacheSummaryObs and SchemaCacheLoadedObs.
Logically - we want to emit a single event containing both summary and timing information. How it is logged is a different matter and should be decoupled.
This commit
* changes observationMessage function returning Text to observationMessages returning [Text] so that it is possible to return multiple (or zero) messages to log based on an observation event
* Removes SchemaCacheSummaryObs constructor from Observation type and adds summary text to SchemaCacheLoadedObs
Observation module exports observationMessage and isDbListenerBug functions. The first one is used only in Logger module and the second one - only in Listener.
This change moves observationMessage function to Logger module and isDbListenerBug function to Listener module, making all three modules more cohesive and lessening dependencies.
It also gives the compiler more opportunities for intra-module inlining and optimization.
Add a new section "Client Error Verbosity". Move the details from
the config page to this section and link it.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Listener has its own backoff logic independent of schema cache loading. It also uses listener delay value to identify its retries and to trigger schema cache reloads upon failures.
If schema cache reloading sets listener delay it might lead to some unexpected reloading loops.
That's especially important in set ups where listener connection points to master and the pool to replica(s).
Set error verbosity using this config. The verbosity can
be set to `verbose` or `minimal` for client error responses.
This only affects client side HTTP responses, server side logs
are not affected by this config.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Towards #4088.
- Some of these instances are not used. Reduces number of lines
significantly.
- Removing this gives us more flexibility for cases like conditional
encoding based on some outside parameter, without needing to
add the conditional at type level.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The `Error.hs` module is getting bloated. This moves types
to another module for better separation of concerns.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
stateLogDebouncePoolTimeout is an MVar initialized on the first logging of PoolAcqTimeoutObs. The code in logWithDebounce has race condition that could lead to creation of multiple debouncers.
This change simplifies logic by getting rid of lazy initialization of debouncer.
PoolAcqTimeoutObs is always created with SQL.AcquisitionTimeoutUsageError so the parameter does not provide any useful information.
On the other hand, it complicates debouncing logic in Logger as logWithDebounce has to lazily create the debouncer (which is error prone and hence the logic has a race condition).
This change removes the parameter from PoolAcqTimeoutObs. It is a prerequisite to simplifying logWithDebounce.
Fixes#4646. Using the repro on #4646, this now produces the log:
```
11/Feb/2026:09:40:08 -0500: Warp server error: stack overflow
```
When:
```
$ curl localhost:3000/
curl: (52) Empty reply from server
```
- Move privileges and grants to privileges.sql
- Move schema, tables and functions to schema.sql
- Move global database settings to database.sql
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Diagnosing problems with listener channel notifications not being handled properly by PostgREST connected to read replicas is difficult. Issues might be related to lost connections and listener not being connected to the right host after failover or database server restarts.
This patch adds logging of actual host:port used by libpq connection opened by the listener. It should make it easier to find out if PostgREST is connected to the right host.
The coverage workflow mentions that we run the tests against PG 15,
however that is incorrect and misleading. We actually run it against the
latest supported PostgreSQL version according to the nix scripts.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The logic to evaluate `JSPath` belongs to `JSPath.hs` module.
Hence, moving this logic from `Auth/Jwt.hs` to here.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
* Calculation of hint message when requested relation is not present in schema cache requires creation of a FuzzySet (to use fuzzy search to find candidate tables). For schemas with many tables it is costly.
This patch introduces dbTablesFuzzyIndex in SchemaCache to memoize the FuzzySet creation.
* Additionally, because of FuzzySet large memory requirements, this patch introduces a limit of 500 relations per schema, above which FuzzySet is not created and hint calculation disabled.
Moves the `negotiateContent` function to `Plan/Negotiate.hs`
module. This also adds comments to describe the current approach
we have for negotiation.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The `Ord` instance definition for type `Routine` had a logical
error when comparing two routines. The error did not affect any
end users. However, for correctness and completeness reasons, this
commit fixes the error.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Co-authored-by: Joel Jacobson <joel@compiler.org>
Chaining the target generation command after the build is done ensures
that if a build takes too long, the dynamic target generation for vegeta
won't cause 401 Unauthorized errors due to already expired JWTs.
Also split the rsa materials writing to another python program for
easier maintenance.
The hasSingleUnnamedParam function was only checking the parameter type
but not whether the parameter actually had no name. This caused functions
with a single NAMED parameter (e.g., `foo(data json)`) to incorrectly
match the single-param fallback mode.
The result was a confusing PostgreSQL error 42883 "function does not exist"
instead of a clean PGRST202 error explaining that no matching function
was found.
Added ppName == mempty check so functions with named parameters don't
incorrectly match the single-param fallback.
Currently the `postgrest-loadtest -k jwt-*` loadtests use the OPTIONS method,
which is good for isolating the perf changes for JWT auth.
But using GET is more accurate for actual perf improvements,
since it actually hits the database.
This adds a method argument for postgrest-loadtest, so GET can be used like:
```
postgrest-loadtest -k jwt-hs --method GET
```
CI is not changed, this is mostly useful for manually checking variance.
Centralizes handling of preferences that are used to
create `Preference-Applied` header which is returned on
responses.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
We don't allow accessing schemas "pg_catalog" and "information_schema".
Instead of silently failing requests on these schemas, this now adds
logging an error on startup when these schemas are found in `db-schemas`
config.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Postgrest fails on unsupported pg versions. However before killing
the thread, it continues to print a few more log messages which
were misleading. This commit fixes this by making sure that the
no log message should be printed after the unsupported pg version
observation and kill the thread immediately.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
When running postgres from nix-shell, nix creates a directory
structure like `postgrest/postgrest-with-pg-17-XXX` in the `/tmp`
directory. This commit removes the extra `postgrest/` prefix to
shorten length of absolute path length of filenames.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The `test_io.py` module is too bloated (2100+ lines). To
logically group related tests, as a first step, this commit
separates authentication related IO tests into `test_auth.py`
module.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The `*` character is a reserved character in the postgrest
url grammar but we didn't mention it in the docs. Now added.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Fixes a typo in the documentation where it said that `()`
is a reserved character, when we actually meant to say that
`(` and `)` are reserved characters.
Related discussion in #4254.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The x86 GitHub runner will not be available anymore, soon.
We might be able to re-introduce this, once we can build a static
executable via Nix on darwin, too.
Renames these scripts to `postgrest-with-pg-xx`. The renaming
helps reduce the length of temporary filenames. This is needed
to ensure that socket file names remain under the maximum
allowed length of 107 chars.
Closes#4461.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The docker-compose example did not work with multiple containers
when PGRST_SERVER_HOST is set to `localhost`. This updates the
value to `0.0.0.0` allowing other containers to connect.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
- Adds fixtures to `test/io/fixtures.sql` to test resource
embedding related queries.
- Moves the resource embedding related tests that no longer
require big schema from `test_big_schema.py` to `test_io.py`.
Closes#4417.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
This should reduce setup time for build process.
- cache: introduced in #2928, defunct since #4084
- clock: introduced in #2928, defunct since #4084
- heredoc: introduced in #714, defunct since #4390
- iproute: introduced in #3560, defunct since #4288
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The `dumpQi` function is only used in the `Config.hs` module, so
it is moved there. This also adds a doctest for `quoteQi` and
clarifies its usage.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
When db-pre-config is accidentally set to a pg reserved word
like "true", it fails with a confusing error. The function
names should be properly quoted to avoid such errors. This commit
resolves this by quoting the pre-config function name.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The changelog description mentioned that we follow semantic
versioning but from now on we don't. Hence updated the description
to reflect new versioning policy.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
This change is an initial step to change JWT aud configuration to regular expression.
Exporting function
audMatchesCfg :: AppConfig -> Text -> Bool
from Config module allows changing the way how JWT aud is configured to be isolated and not affect code in Auth.JWT
* replaces v2-build -> v2-run (which should be v2-exec)
with one-shot v2-run
* fixes empty postgrest.prof file by sending SIGINT to the profiled
postgrest process to trigger writing it explicitly
To make schema cache loading wait tests robust it is necessary to provide three separate internal config variables:
* "internal-schema-cache-query-sleep" - introduces delay in schema queries execution
* "internal-schema-cache-load-sleep" - introduces delay between schema queries execution and processing their results
* "internal-schema-cache-relationship-load-sleep" - introduces delay in processing relationship query results
Thanks to these changes it is now possible to test various schema loading scenarios with the right granularity robustly (eg. make sure requests wait for schema loading but not for relationship loading).
We usually miss adding or changing the example config
file when we do changes to our configurations. Moving
the example config file from CLI module to Config module
would make it less easy to miss doing the associated
changes to the example config file.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
There are a few benefits for this:
- All fixtures in one module, so single source of truth.
- The fixtures are automatically imported and injected by pytest
so no explicit imports needed for these.
- Linters won't complain about redefinition of outer scope objects.
Co-authored-by: Jens Troeger <jens.troeger@light-speed.de>
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The `--ready` flag is a wrapper around the admin server
`/ready` request. This is done through using an http client
library in postgrest.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The CLI page had the config file parameter missing. This commit
adds that. Also adds the CLI usage output which shows all
available options. Closes#4300.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Moves the `set_statement_timeout` and `reset_statement_timeout`
function to postgrest.py. This cleans up test_io.py so it only
contains tests. Also makes these functions reusable in other
other tests like test_cli.py.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
This splits off the transaction concerns into a MainTx module
It's just moving some functions inside the Query module to the MainTx module.
Some types names were also changed so they don't have the "Query" prefix, to avoid confusion.
Sets hpctixfile to be unique for every postgrest process that
is run.
Previously, this was based on the test name, but issues arise
when two postgrest processes are run under the same test, which
generates two files where one gets overwritten by the other.
Consequently, coverage data used to get lost, which is now fixed
with this commit.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Closes https://github.com/PostgREST/postgrest/issues/4226.
This requires moving query generation to the top App.hs module.
At this point is also simple to log the transaction variables + the
pre-request function call but this is not done here to reduce scope.
Towards fixing #4316. The config error tests were not
correctly tested. This commit allows capturing stderr
output and then checks the error messages properly.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Replaces the "API server listening on unix socket" with simpler
"API server listening on " observation. This allows refactoring
redundant code.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The IPv6 address logged at the startup like `::1:80` was
wrong because the port isn't clearly separated. This commit
corrects it, now logging as `[::1]:80`.
This is done in accordance to RFC 3986. In short, we did this
have a clear separation between the port and host because
the components of an IPv6 are separated with the ':' character.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The prometheus metrics text format requires `Content-Type` header
for correct scraping which fails otherwise. Closes#4271.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The `fromJsonBodyF` function was very unreadable and making
changes were difficult. This commit improves the readability
for easier future changes and improvements.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Closes https://github.com/PostgREST/postgrest/issues/4245.
It adds a query on schema cache construction, but doesn't add a new
attribute to the SchemaCache type. We only need the information to be
passed to the `initialMediaHandlers` function to build the builtin media
handlers.
Changes:
1. Refactoring and some cleanup of JWT handling code:
* Instead of caching AuthResult cache decoded claims (which signature was verified). Validating claims and determining role is done after cache lookup
* Cleaned up API so that usage of it is simplified: lookupJwtCache cache key >>= parseClaims configJwtAud time
* Handling of JwtCacheState initialization and updates of configuration is encapsulated in Auth.JwtCache module
2. Generic high performance (hopefully) scalable, dynamically resizeable cache implementation based on stm, stm-hamt and sieve algorithm. It also integrates with PostgREST measurements infrastructure providing usage stats (ie. hit ratio, evictions count)
This way we benefit from NixOS' binary cache to deliver GHC for us and
don't need to cache it ourselves.
This will become relevant once we do that for more platforms.
We have fixed the static build of PostgreSQL upstream, so we don't need
our separate overlay anymore. One more step towards building a static
executable on other platforms.
Instead of rolling our own, we can use some tooling from nix / nixpkgs.
We drop the "statically linked" check, because our goal is to compile
mostly-static executables to darwin, too. However, those will never be
fully static, because they always link to the platform's libc.
The dependency on it was removed in #3608 already, but we forgot to
remove it from postgrest.cabal, which caused it to still be built.
We didn't realize because all references of it were stripped away by
dead code elimination anyway.
The OpenAPI specification was incorrectly exposing GET methods for
VOLATILE functions, even though such functions properly reject GET
requests at runtime with "405 Method Not Allowed". This created
a mismatch between the advertised API specification and the actual
runtime behavior.
VOLATILE functions should only be callable via POST since they may
have side effects, while STABLE and IMMUTABLE functions can safely
be called via GET since they don't modify database state.
Fix by checking the pdVolatility field in makeProcPathItem() and
only including GET methods in the OpenAPI PathItem for non-volatile
functions.
The runtime behavior was already correct; this fixes only the
OpenAPI documentation generation.
Completes the jwt loadtests, now we have non-cached, cached + worst case
for the cache.
- jwt-hs (edited): now has the cache disabled
- jwt-hs-cache: cache enabled for jwt-hs
- jwt-hs-cache-worst: worst case of the jwt-hs cache
- jwt-rsa (edited): now has the cache disabled
- jwt-rsa-cache: cache enabled for jwt-rsa
- jwt-rsa-cache-worst: worst case of the jwt-rsa cache
Also deletes `nix/tools/generate_targets_rsa.py` and uses a single
python script.
Should prove what's mentioned on
https://github.com/PostgREST/postgrest/pull/4084#issuecomment-2998170423
Previously, the loadtest would always compare against main and the
latest tag. This meant a loadtest running on the v13 release branch,
would compare to a *future* version of both. This is not useful, and
also not supported by `postgrest-loadtest-against`, which recently
introduced a change on `main`, that now breaks the tests on the back
branches.
With this change, the loadtest will always run against the current
target branch of a PR, so against the v13 branch for a PR against v13,
for example. Also, it will compare against the latest released tag *for
that branch*.
Thus, when running this for v12, it will test against the v12 branch and
the v12.2.12 tag.
Splitting the executable in a separate output avoids distributing all of
the libraries and documentation, which are not needed when just running
PostgREST.
Reduces closure size from 4.3G to 73.9M for the flake exported packages.
Resolves#4149
The `commitlint` script disrupts our release workflow if failed
on push event. We also don't need it on push because once pushed,
git commit history can't be amended.
Closes https://github.com/PostgREST/postgrest/issues/4107.
Adds two python scripts:
- monitor_pid.py: monitors the postgrest process each second
until it exits, then outputs a csv with the results. The nix wrappers
use the `loadtest` dir for the output.
- merge_monitor_result.py: receives a list of csvs and merges them into
a single markdown table. The nix wrappers use the `loadtest/*.csv`
files for the input.
The nix `postgrest-with-pgrst` and `postgrest-loadtest-report` commands
use these scripts to add monitoring for `postgrest-loadtest` and
`postgrest-loadtest-against`.
This is so the process monitoring results are more meaningful.
* remove the amount of total targets from the loadtest kind name
since the targets can change.
* clarifies the top comment on the generate_targets python script
* increase TOTAL_TARGETS on jwt-hs loadtest
The `jwt-aud` config was not validated when containing ':'
character according to RFC 3986. This fix validates it and
fails at startup if it is invalid.
Until now we had a load test with 50k unique JWTs signed with symmetric key.
This commit adds a new load test with 10k JWTs signed with RSA 4096.
Existing -k jwt parameter was changed to -k jwt-hs-50k.
New test is run with -k jwt-rsa-10k parameter.
Additionally a new parameter --jwtcache=off was added to turn off JWT caching in the above load tests.
This change introduces a PostgREST.Auth.Jwt module containing JWT validation functions.
The reason to extract them from Auth is to enable JwtCache module to reuse them without introducing module dependency cycle.
Moat has been bought by Oracle. The advertising subpage redirects to
oracle.com, but pointing at that would be a bit misleading.
No need to keep failing links either.
BREAKING CHANGE
The endpoint was at risk of being left unprotected when exposing it.
The accompanying `admin-server-config-enabled` config was also dropped.
This loadtests the jwt decoding logic. For this it adds an optional
`-k`(kind) parameter to `postgrest-loadtest` and
`postgrest-loadtest-against`.
Old kind (default):
```
postgrest-loadtest -k mixed
postgrest-loadtest-against -k mixed
```
New kind:
```
postgrest-loadtest -k jwt
postgrest-loadtest-against -k jwt
```
Internally it uses a dynamically generated targets file using python
which looks like:
```
GET http://postgrest/authors_only
Authorization: Bearer <jwt>
GET http://postgrest/authors_only
Authorization: Bearer <another-jwt>
...
```
Then this is used to run vegeta with the `-lazy` option.
The old error design didn't allow us to reuse error `code`,
`message` etc. With this refactor, these parts of the error body
can be reused for other potential features.
This also removes the `ErrorCode` type and replace the types
with Text codes.
Our current ErrorCode type wasn't giving us a full picture
of how many different types of errors we are handling. This
PR makes this explicit by including the error code type for
all types of errors.
This also moves the pin for nixpkgs into flake.lock instead of our
custom file. Even for the classic interface via default.nix, the pin
will be loaded from flake.lock, thus everything stays in-sync.
This just exposes the PostgREST package, not more.
The goal is to avoid duplication, so we're re-using only exports from
default.nix.
Resolves#3026
Supersedes #3105
We never need dynamic haskell libraries, because even the dynamic builds
only dynamically link non-haskell dependencies, but always link haskell
dependencies statically.
Profiled libraries are only required when running the memory test, so
explicitly enable them for the profiled package.
This also means, that we don't need to hide the memory test behind a
feature flag anymore. The reason always was assumed to be the big number
of rebuilds required for it. I assume ever since we moved off of
static-haskell-nix and back to nixpkgs-based builds, we have been
building profiled libraries for all our dependencies anway.
I removed those exceptions earlier this week, because I got fooled by my
local test output. Those only return 403 in GitHub Actions - it seems
like those websites block access from there.
Apparently Cirrus removed the 14-1 image. When I firsted looked into
this some days ago, when the job started failing, the docs were not
updated, yet - so it wasn't clear. Now the docs mention freebsd-14-2
explicitly...
This manager was recently introduced to renovate and is now creating PRs
for upper version bounds of our haskell dependencies. We still need to
figure out how to deal with those in the best way, but some basic
configuration can already be done. Here, we:
- disallow any of those updates on the backbranches.
- group GHC-provided dependencies together.
- group packages from the hasql ecosystem together.
- the fuzzyset dependency must be restricted to <0.3 - we know that
already and did that on purpose.
Apparently the STACK_ROOT has been moved to C:\sr - for unknown reasons,
at least to me.
This should enable caching again and make the stack on windows builds
much faster than recently.
When macos-14 was rolled out libpq was not installed, but by now it is
by default. Thus, we don't need to do that, it only creates a warning
annotation right now.
Since we're currently on the unstable channel and will likely stay there
for a while, let's encode this in the update script.
Once we switch back to stable, if we do, we can still adjust it again.
Currently it's redundant and not easy to read.
```
{"message":"JSON object requested, multiple (or no) rows returned",
"details":"The result contains 2 rows"}
```
Now:
```
{"message":"Cannot coerce the result to a single JSON object",
"details":"The result contains an array of 0 objects"}
``
Also correct docs which had a wrong error code.
It used to be that this was only enabled with log-level=debug.
But the default log-level is misleading, for example:
```
$ PGRST_DB_SCHEMAS="apflora" postgrest-with-postgresql-16 -f test/io/big_schema.sql postgrest-run
...
13/Nov/2024:22:08:20 -0500: Config reloaded
13/Nov/2024:22:08:20 -0500: Schema cache queried in 36.3 milliseconds
13/Nov/2024:22:08:20 -0500: Schema cache loaded 326 Relations, 305 Relationships, 7 Functions, 0 Domain Representations, 4 Media Type Handlers, 1194 Timezones
```
The "Schema cache loaded" can take a while to appear, yet the 22:08:20
time is the same. If we reveal the load time this is clarified:
```
13/Nov/2024:22:08:37 -0500: Schema cache loaded in 16770.1 milliseconds
```
This should fix CI which is failing lately like this:
https://cirrus-ci.com/task/4665005218463744
ld-elf.so.1: /lib/libc.so.7: version FBSD_1.8 required by
/usr/local/bin/stack not found
It's important for observability to have an historic trace of the pool
size. Currently we expose it on the metrics endpoint, but not all
deployments use it.
This logs the pool size after the successful connection log to make it
more visible:
<timestamp>: Connection Pool initialized with a maximum size of 4 connections
It's not immediately clear on which port the API server is listening.
Also it's not clear that the "pgrst" channel is for database
notifications.
Goes from:
<timestamp>: Admin server listening on 0.0.0.0:3001
<timestamp>: Listening on 0.0.0.0:3000
<timestamp>: Listening for notifications on the "pgrst" channel
To:
<timestamp>: Admin server listening on 0.0.0.0:3001
<timestamp>: API server listening on 0.0.0.0:3000
<timestamp>: Listening for database notifications on the "pgrst" channel
This change adds an explanation of how to handle an array of JSON objects within an RPC call. To pass multiple objects, an array of JSON objects must be the JSON value, with the key being the json or jsonb variable name of the Postgres function.
For people who want to perform multiple tasks/inserts/updates within a single API call, this is a needed explanation for that use-case.
- Fixed "column reference <col> is ambiguous" error when selecting "?select=...table(col,count())"
- Fixed "column <json_aggregate>.<alias> does not exist" error when selecting "?select=...table(aias:count())"
Fixes#3660. Load the config after getting the pg version but before loading the schema.
The regression happened on f09655b.
Also remove schema cache load wrapper and separate db queries in different functions.
Co-authored-by: Laurence Isla <lau.isla.c@gmail.com>
Fixes https://github.com/PostgREST/postgrest/issues/3523.
Now if there's a failure when obtaining the pg version OR schema cache,
we do the same retrying process. This way we don't add two retries.
Refactors and renames the "connectionWorker" to "schemaCacheLoader".
This makes more sense since what we really want is the schema cache,
the version is the pre-requisite for ensuring our
schema cache queries work.
Additionally, we no longer log ` Attempting to connect to the database...`
at startup unnecessarily. This is only logged whenever there's a retry attempt.
Previously, we'd silently report "200 OK" on the root endpoint, but
would never return any endpoints from the schema cache.
Now the schema cache query fails because of the ::regnamespace cast.
The schema cache and OpenAPI output would currently list the first found
enum with the same name instead of the correct type. One other case
where this comes up is when a regular type and an enum type have the
same name. For example in the spec fixtures, we have an enum called
"bit". Every "bit" type, no matter whether it's that enum or the
built-in bit type, will show those enum options in the OpenApi output.
Not adding a test, because OpenAPI is supposed to go away in the future
anyway.
Previously they were passed as a JSON payload. This results in a LATERAL
join for the calling expression, which prevents LIMIT from being pushed
into the inlined function call, making some requests really slow.
Resolves#2858
* fix: forbid same server-port and admin-server-port
Forbids server-port and admin-server-port from being equal altogether,
despite they might not conflict at all in case admin and app are bound
to different addresses. Implemented as per the discussion at
https://github.com/PostgREST/postgrest/issues/3508#issuecomment-2125123633
This reduces our Template Haskell dependencies.
The commit hash never made it into the nix-based static executable
anyway. Since we'd like to move to produce more executables via nix in
the future, it will be hard to maintain the commit hash.
The cabal-provided Paths_ module allows us to use the version number
from postgrest.cabal. This can be done equally well with the GHC-defined
CPP macro "VERSION_postgrest".
By making this change we avoid the inclusion of the Paths_ module, which
also stores some paths related to the cabal configuration. Those are
problematic to go into the final executable, because for nix-based
builds those are paths to the /nix/store/... - which means that our
static executable then depends on those paths.. and we can't build a
minimal docker image anymore.
To counter this, we have been using dead code elimination when building
the static executable. This has been working well, but there is a
problem on aarch64-darwin, which we will hit once can finally make our
way there: GHC on aarch64-darwin (or darwin in general?) can't do dead
code elimination - and thus it'd be impossible to create those minimal
docker images for those platforms. More information upstream in nixpkgs:
https://github.com/NixOS/nixpkgs/issues/318013
Since we changed our release workflow, we have adjusted:
- the docs to use postgrest.org/en/v12/ -style URLs, i.e. only using the
major component.
- the pre-release / devel versions to contain only two instead of four
version parts, i.e. currently 12.3.
Timing dependent tests in the IO tests don't work too well when the next
commit increases the JWT parsing performance.
The remaining IO tests are for coverage and basic breakage. Loadtests
are adapted so that performance regressions for JWT caching would be
detected that way.
The former depends on th-orphans which does not cross-compile well,
because of template haskell usage.
neat-interpolation is also much better maintained.
This also potentially helps with packaging for Debian/Ubuntu in #2273.
Contravariant.Extras uses Template Haskell, which is hard to
cross-compile. Reducing usage of Template Haskell with the ultimate goal
of solving all cross compilation challenges.
If your database schema has changed while the PostgREST server is running,
send the server a SIGUSR1 signal or restart it (http://postgrest.org/en/stable/admin.html#schema-reloading) to ensure the schema cache is not stale. This sometimes fixes apparent bugs.
-->
### Environment
* PostgreSQL version: (if using docker, specify the image)
* PostgREST version: (if using docker, specify the image)
* Operating system:
### Description of issue
Describe the behavior you expected vs the actual behavior. Include:
- A minimal SQL definition.
- How you make the request to PostgREST (curl command preferred).
description:Waits for a specific Cirrus CI run to complete, then downloads the artifact and uploads it to the current workflow. This will silently succeed if Cirrus CI did not schedule a task within 2 minutes.
inputs:
download:
description:Name of Artifact to download from Cirrus CI
required:true
task:
description:Name of Cirrus Task
required:true
token:
description:GitHub Token
required:true
upload:
description:Name of Artifact to upload on GitHub Actions
A broken link was detected in the docs. Please check the [failed run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details.
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).
All notable changes to this project will be documented in this file. From version `14.0` onwards PostgREST follows a `MAJOR.PATCH` two-part versioning. Only even-numbered MAJOR versions will be released, reserving odd-numbered MAJOR versions for development.
## Unreleased
### Fixed
- The OpenAPI output now reflects table privileges: only the granted HTTP methods are exposed (e.g. `SELECT` grants `GET`, `INSERT` grants `POST`) and column-level grants filter the columns shown on table definitions and row filters.
## [16.1] - 2026-08-10
### Fixed
- JWT validation uses wrong current time due to a bug in auto-update by @mkleczek in #5159
## [16.0] - 2026-08-07
### Changes
#### HTTP Server
- [Graceful shutdown](https://docs.postgrest.org/en/v16/references/http_server.html#graceful-shutdown) by @mkleczek, @Vlix in #4702
- [server-reuseport](https://docs.postgrest.org/en/v16/references/configuration.html#server-reuseport) allows starting multiple PostgREST instances using the same port on supported platforms by @mkleczek in #4703, #4694
#### Performance
- Optimize schema cache domain type resolution by using [pg_basetype](https://www.postgresql.org/docs/current/functions-info.html#FUNCTIONS-INFO-CATALOG) on PostgreSQL 17+ by @joelonsql in #4567
- [Prefer: count=exact](https://docs.postgrest.org/en/v16/references/api/pagination_count.html#exact-count) no longer does a double count on requests that do not use ranges or `db-max-rows` by @laurenceisla in #3957
- [Prefer: timezone](https://docs.postgrest.org/en/v16/references/api/preferences.html#prefer-timezone) no longer requires the schema cache by @steve-chavez in #5100
+ Previously this required caching [pg_timezone_names](https://www.postgresql.org/docs/current/view-pg-timezone-names.html) which was slow in some systems
#### Integrations
- PostgREST is now tested to work with [OrioleDB](https://github.com/orioledb/orioledb/) in #4845 by @wolfgangwalther
+ See [our guide for running OrioleDB on NixOS](https://docs.postgrest.org/en/v16/integrations/nixos.html)
#### JWT
- [JWT Role Extraction](https://docs.postgrest.org/en/v16/references/auth.html#jwt-role-extract) is now more flexible, supporting the standard JSON Path defined in RFC 9535 by @taimoorzaeem in #4984
#### API
- [Prefer: timezone](https://docs.postgrest.org/en/v16/references/api/preferences.html#timezone) now supports numeric offsets like `05:00` or `-4` by @steve-chavez in #5100
- Fix unexpected results when embedding and filtering the same table more than once by @laurenceisla in #4075
+ You need to set [url-use-legacy-target-names](https://docs.postgrest.org/en/v16/references/configuration.html#url-use-legacy-target-names) to `false`.
- Deprecate filters, orders and limits with the name of an embedded table when it has an alias by @steve-chavez, @laurenceisla in #4075
+ e.g. `?select=alias:table(*)&table.id=eq.1` will not be possible anymore, use `?select=alias:table(*)&alias.id=eq.1` instead.
+ You will see a warning in the logs and a `Warning` header on the client response when this happens.
+ You can disable this behavior now by setting `url-use-legacy-target-names = false`.
- Add `Vary` header to responses by @develop7 in #4609
- Fix automatic transaction retries on `40001 (serialization_failure)` errors to prevent replication lag by @laurenceisla in #3673
#### Observability
- [GHC runtime metrics](https://docs.postgrest.org/en/v16/references/observability.html#ghc-runtime-metrics) by @mkleczek in #4862
- [client-error-verbosity](https://docs.postgrest.org/en/v16/references/configuration.html#client-error-verbosity) to customize responses error verbosity by @taimoorzaeem in #4088, #3980, #3824
- [log-level](https://docs.postgrest.org/en/v16/references/configuration.html#log-level) config is now reloadable by @taimoorzaeem in #5113
- Log error when `db-schemas` config contains schema `pg_catalog` or `information_schema` by @taimoorzaeem in #4359
- Log schema cache queries timings on `log-level=debug` by @steve-chavez in #4805
#### Admin Server
- [admin-server-unix-socket](https://docs.postgrest.org/en/v16/references/configuration.html#admin-server-unix-socket)/[admin-server-unix-socket-mode](https://docs.postgrest.org/en/v16/references/configuration.html#admin-server-unix-socket-mode) to run the admin server on a unix socket by @wolfgangwalther in #5003
- Fix responding with `Something went wrong` on Admin server when under EMFILE by @mkleczek in #5077
#### Deployment
- Make executable for aarch64-linux static instead of Ubuntu-based by @wolfgangwalther in #4193
- Docker image for aarch64-linux is now built from scratch instead of being Ubuntu-based by @wolfgangwalther in #4193
- Besides Docker Hub, docker images are now published to Github Container Registry by @wolfgangwalther in #2836
#### Schema Cache
- Fix requests failing when the schema cache fails to reload, when this happens PostgREST will continue serving requests in "best effort" by @mkleczek in #4873#4869
- Fix reporting 503s errors unnecessarily while the schema cache is loading at startup by @mkleczek in #4880
- Fix schema cache dump missing RPC transaction isolation level by @taimoorzaeem in #5079
#### Listener
- Fix config `db-channel-enabled` not reloading by @taimoorzaeem in #4894
### Migration to v16
- Drop support for PostgreSQL EOL version 13 by @wolfgangwalther in #4193
+ PostgreSQL 13 end of life was on 2025 ([ref](https://www.postgresql.org/support/versioning/))
+ Upgrade your PostgreSQL version to at least 14 to use this new PostgREST version.
- Fail at startup when `db-schemas` contains schema `pg_catalog` or `information_schema` by @taimoorzaeem in #4359
+ Previously it failed at runtime with `PGRST205` on requests related to these schemas.
+ Remove `pg_catalog` and `information_schema` from `db-schemas`.
-`Prefer: timezone` no longer complies with `handling=lenient` and instead always fails by @steve-chavez in #5128
+ Supporting this required caching `pg_timezone_names`, which was expensive.
+ Ensure your requests always have a valid timezone.
-`jwt-role-claim-key` no longer uses the JSPath DSL and instead uses JSON Path by @taimoorzaeem in #4984
+ Now all config values must start with `$` character.
Example: `.roles.read` -> `$.roles.read`
+ Keys with special characters, with the exception of `_` char must be quoted.
+ Update the `jwt-role-claim-key` value accoring to the above rules. Also see the syntax reference: [RFC 9535](https://www.rfc-editor.org/rfc/rfc9535.html#name-jsonpath-syntax-and-semanti).
## [14.16] - 2026-07-27
### Fixed
- Fix admin server crashing without a way to recover by @taimoorzaeem in #5096
## [14.15] - 2026-07-13
### Fixed
- Fix admin server dying silently by @Vlix, @mkleczek, @steve-chavez in #5012
## [14.14] - 2026-06-29
### Fixed
- Fix admin server not logging cause of failure by @taimoorzaeem in #5012
## [14.13] - 2026-06-04
### Fixed
- Fix connection retrying message in `PGRST000` error by @netqo in #4980
+ Remove redundant "Retrying the connection." from message because it is logged separately
- Fix request failures when `work_mem` is set on a role by @laurenceisla in #4955
## [14.12] - 2026-05-20
### Fixed
- Fix race condition in pool_available metric causing negative values during network instability by @mkleczek in #4622
## [14.11] - 2026-05-04
### Fixed
- Fix login with uppercase and mixed case role names by @taimoorzaeem in #4678
- Restore Listener query shape so it can be found in `pg_stat_activity` by @mkleczek in #4857#4859
- The LISTEN channel now automatically recovers when it stops working due to a PostgreSQL bug @laurenceisla in #3147
- Fix misleading "Functions" name on schema cache summary in startup logs by @taimoorzaeem in #4821
## [14.10] - 2026-04-16
### Added
- Log when the pool is released during schema cache reload on `log-level=debug` by @mkleczek in #4668
### Fixed
- Fix unnecessary connection pool flushes during schema cache reloading by @mkleczek in #4645
## [14.9] - 2026-04-10
### Added
- Log host, port and pg version of listener database connection by @mkleczek in #4617#4618
### Fixed
- Remove red herring warp logs on default log-level, only emit them on `log-level=debug` by @steve-chavez in #4799
## [14.8] - 2026-04-03
### Added
- Log a `HINT` when the LISTEN channel stops working due to a PostgreSQL bug by @laurenceisla in #4581
### Fixed
- Fix invalid OpenAPI 2.0 format for integer types (`smallint`, `integer`, `bigint`) by @arturbent0 in #4641
## [14.7] - 2026-03-20
### Fixed
- Fix not logging SIGTERM and SIGINT by @steve-chavez in #4728
## [14.6] - 2026-03-06
### Fixed
- Fix leaking table and function names when calculating error hint by @taimoorzaeem in #4675
## [14.5] - 2026-02-12
### Fixed
- Don't hide async exceptions in logs by @stevechavez in #4646
## [14.4] - 2026-01-29
### Fixed
- Ensure Listener connections are released by @mkleczek in #4614
- Fix incorrectly filtering the returned representation for PATCH requests when using `or/and` filters by @laurenceisla in #3707
- Fix listener running with exception masked after first failure by @mkleczek in #4615
## [14.3] - 2026-01-03
### Fixed
- Fix performance and high memory usage of relation hint calculation by @mkleczek in #4462, #4463
## [14.2] - 2025-12-18
### Fixed
- Fix `hasSingleUnnamedParam` incorrectly matching functions with named parameters by @joelonsql in #4553
+ Functions with a single named parameter (e.g., `foo(data json)`) no longer incorrectly match the single-param fallback, returning a clean `PGRST202` error instead of a confusing PostgreSQL `42883` error.
- Fix misleading logs on unsupported PostgreSQL versions by @taimoorzaeem in #4519
- Fix regression where the `PGRST103` error response was truncated by @laurenceisla in #4455
+ Happened when an `offset` was greater than the rows requested and `Prefer: count=exact` was sent.
- Fix not returning `Content-Length` on empty HTTP `201` responses by @laurenceisla in #4518
- Fix inaccurate Server-Timing header durations by @steve-chavez in #4522
- Fix inaccurate "Schema cache queried" logs by @steve-chavez in #4522
## [14.1] - 2025-11-05
## Fixed
- Fix `db-pre-config` function failing when function names are pg reserved words by @taimoorzaeem in #4380
- Fix `server-host=!6` incorrectly binds to IPv4 address by @taimoorzaeem in #3202
## [14.0] - 2025-10-24
### Added
- Bounded JWT cache using the SIEVE algorithm by @mkleczek in #4084
+ It now uses a fixed size cache instead of arbitrary sized cache.
- Add `--ready` flag for postgrest healthcheck by @taimoorzaeem in #4239
### Fixed
- Fix not logging OpenAPI queries when `log-query=main-query` is enabled by @steve-chavez in #4226
- Fix not logging explain query when `log-query=main-query` is enabled by @steve-chavez in #4319
- Fix not logging transaction variables and db-pre-request function when `log-query=main-query` is enabled by @steve-chavez in #3934
- Fix not logging the JSON message to stderr on a `PGRST002` error by @laurenceisla in #4129
- Fix reloading the Schema Cache unnecessarily on a `PGRST002` error by @laurenceisla in #4367
- Fix schema cache loading taking a long time for large schemas by @mkleczek in #4360, #3704
### Changed
- Drop support for PostgreSQL EOL version 12 by @wolfgangwalther in #3865
- From now on PostgREST will follow a `MAJOR.PATCH` two-part versioning. Only even-numbered MAJOR versions will be released, reserving odd-numbered MAJOR versions for development.
- Replaced `jwt-cache-max-lifetime` config with `jwt-cache-max-entries` by @mkleczek in #4084
-`log-query` config now takes a boolean instead of a string value by @steve-chavez in #3934
## [13.0.8] - 2025-10-24
### Fixed
- Fix loading utf-8 config files with `ASCII` locale set by @taimoorzaeem in #4386
## [13.0.7] - 2025-09-14
### Added
- Improve the `PGRST106` error when the requested schema is invalid by @laurenceisla in #4089
+ It now shows the invalid schema in the `message` field.
+ The exposed schemas are now listed in the `hint` instead of the `message` field.
- Improve error details of `PGRST301` error by @taimoorzaeem in #4051
## [13.0.6] - 2025-08-30
### Fixed
- Fix logging the Haskell type instead of the listener error message directly by @laurenceisla in #3588
- Fix format of `IPv6` address logged at PostgREST startup by @taimoorzaeem in #4291
- Fix empty enum in `preferParams` OpenAPI parameter by @laurenceisla in #4292
## [13.0.5] - 2025-08-24
### Fixed
- Fix OpenAPI broken docs link by @taimoorzaeem in #4080
- Fix OpenAPI specification incorrectly exposing GET methods for volatile functions by @joelonsql in #4174
- Fix empty spread embeddings return unexpected SQL error by @taimoorzaeem in #3887
- Fix `/metrics` endpoint not responding with `Content-Type` header by @taimoorzaeem in #4271
## [13.0.4] - 2025-06-17
### Fixed
- Fix regression that makes full-text search not work on domain types based on `tsvector` by @laurenceisla in #4135
- Fix `jwt-aud` config not failing when set to an invalid URI by @taimoorzaeem in #4132
## [13.0.3] - 2025-06-16
- Fix `max-affected` preference not failing with RPC when `handling=strict` by @taimoorzaeem in #4100
- Fix a property definition's type in OpenAPI not showing the correct base type of a recursive domain by @laurenceisla in #4136
### Fixed
## [13.0.2] - 2025-06-02
### Fixed
- Fix regression that makes `ORDER BY` with nulls-order not work alongside limits by @laurenceisla in #4109
## [13.0.1] - 2025-06-01
### Fixed
- Fix jwt error returning HTTP status `400` for invalid role by @taimoorzaeem in #3601
- Fix `db-extra-search-path` cannot be set to nothing by @taimoorzaeem in #4074
+ It can now be disabled by setting it to empty string.
+ Schema Cache load error is now logged including `db-schemas` and `db-extra-search-path` config values.
## [13.0.0] - 2025-05-08
### Added
-#3558, Add the `admin-server-host` config to set the host for the admin server - @develop7
-#3607, Log to stderr when the JWT secret is less than 32 characters long - @laurenceisla
-#2858, Performance improvements when calling RPCs via GET using indexes in more cases - @wolfgangwalther
-#3560, Log resolved host in "Listening on ..." messages - @develop7
-#3727, Log maximum pool size - @steve-chavez
-#1536, Add string comparison feature for jwt-role-claim-key - @taimoorzaeem
-#3747, Allow `not_null` value for the `is` operator - @taimoorzaeem
-#2255, Apply `to_tsvector()` explicitly to the full-text search filtered column (excluding `tsvector` types) - @laurenceisla
-#1578, Log the main SQL query to stderr at the current `log-level` when `log-query=main-query` - @laurenceisla
-#3903, Log connection pool borrows on `log-level=debug` - @taimoorzaeem
-#3041, Allow spreading one-to-many and many-to-many embedded resources - @laurenceisla
+ The selected columns in the embedded resources are aggregated into arrays
+ Aggregates are not supported
-#2967, Add `Proxy-Status` header for better error response - @taimoorzaeem
-#3697, #3602, Handle queries on non-existing table gracefully - @taimoorzaeem
-#3600, #3926, Improve JWT errors - @taimoorzaeem
-#3013, Fix `order=` with POST, PATCH, PUT and DELETE requests - @taimoorzaeem
-#3965, Fix filter on unselected columns in a table-valued function - @taimoorzaeem
-#4052, Fix schema cache load duplicate objects with different object type but same oid - @taimoorzaeem
### Changed
-#2052, Dropped support for PostgreSQL 9.6 - @wolfgangwalther
-#2052, Dropped support for PostgreSQL 10 - @wolfgangwalther
-#2052, Dropped support for PostgreSQL 11 - @wolfgangwalther
-#3508, PostgREST now fails to start when `server-port` and `admin-server-port` config options are the same - @develop7
-#3607, PostgREST now fails to start when the JWT secret is less than 32 characters long - @laurenceisla
-#3644, Fail schema cache lookup with invalid `db-schemas` or `db-extra-search-path` config - @wolfgangwalther
- Previously, this would silently return 200 - OK on the root endpoint, but don't provide any usable endpoints.
- Note: This also applies when deleting the `public` schema - both config options default to that.
-#3757, Remove support for `Prefer: params=single-object` - @joelonsql
+ This preference was deprecated in favor of Functions with an array of JSON objects
-#3013, Drop support for Limited updates/deletes
+ The feature was complicated and largely unused.
-#3956, Drop `/config` endpoint of admin server - @steve-chavez
+ The endpoint was at risk of being left unprotected when exposing it.
+ The accompanying `admin-server-config-enabled` config was also dropped.
-#3598, PostgREST now validates the `kid` parameter of the JWT - @wolfgangwalther
+ If the JWT contains a ``kid`` parameter, then PostgREST will look for the JSON Web Key in the `jwt-secret`.
+ If the JWT doesn't contain a `kid`, the behavior should be backwards compatible. PostgREST will try each key in the `jwt-secret` one by one until it finds one that works.
- #3697, #3602, Querying non-existent table now returns `PGRST205` error instead of empty json - @taimoorzaeem
@@ -358,7 +809,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
### Added
- #1933, #2109, Add a minimal health check endpoint - @steve-chavez
+ For enabling this, the `admin-server-port` config must be set explictly
+ For enabling this, the `admin-server-port` config must be set explicitly
+ A `<host>:<admin_server_port>/live` endpoint is available for checking if postgrest is running on its port/socket. 200 OK = alive, 503 = dead.
+ A `<host>:<admin_server_port>/ready` endpoint is available for checking a correct internal state(the database connection plus the schema cache). 200 OK = ready, 503 = not ready.
- #1988, Add the current user to the request log on stdout - @DavidLindbom, @wolfgangwalther
@@ -841,7 +1292,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
- Customize content negotiation per route - @begriffs
- Allow using nulls order without explicit order direction - @steve-chavez
- Fatal error on postgres unsupported version, format supported version in error message - @steve-chavez
We adhere to [Gentoo's AI policy](https://wiki.gentoo.org/wiki/Project:Council/AI_policy):
> It is expressly forbidden to contribute [...] any content that has been created with the assistance of Natural Language Processing artificial intelligence tools. This motion can be revisited, should a case been made over such a tool that does not pose copyright, ethical and quality concerns.
You can find more about its rationale [here](https://wiki.gentoo.org/wiki/Project:Council/AI_policy#Rationale).
* Make sure you test against the latest [stable release](https://github.com/PostgREST/postgrest/releases/latest)
and also against the latest [devel release](https://github.com/PostgREST/postgrest/releases/tag/devel).
It is possible we already fixed the bug you're experiencing.
* Provide steps to reproduce the issue, including your OS version and
the specific database schema that you are using.
* Please include SQL logs for issues involving runtime problems. To obtain logs first
[enable logging all statements](http://www.microhowto.info/howto/log_all_queries_to_a_postgresql_server.html),
then [find your logs](http://blog.endpoint.com/2014/11/dear-postgresql-where-are-my-logs.html).
* If your database schema has changed while the PostgREST server is running,
[send the server a `SIGUSR1` signal](http://postgrest.org/en/latest/admin.html#schema-reloading) or restart it to ensure the schema cache
is not stale. This sometimes fixes apparent bugs.
## Code
We have a fully nix-based development environment with many tools for a smooth development workflow available.
Check the [development docs](https://github.com/PostgREST/postgrest/blob/main/nix/README.md) on how to set it up and use it.
* All contributions must pass the tests before being merged. When
you create a pull request your code will automatically be tested.
* All fixes or features must have a test proving the improvement.
* All features must document the new behavior. Critical fixes that introduce new behavior must be documented too.
* All code must also pass a [linter](http://community.haskell.org/~ndm/hlint/) and [styler](https://github.com/jaspervdj/stylish-haskell)
with no warnings. This helps enforce a uniform style for all committers. Continuous integration will check this as well on every
pull request. There are useful tools in the nix-shell that help with checking this locally. You can run `postgrest-check` to do this manually but
we recommend adding it to `.git/hooks/pre-commit` as `nix-shell --run postgrest-check` to automatically check this before doing a commit.
### Running Tests
For instructions on running tests, see the [development docs](https://github.com/PostgREST/postgrest/blob/main/nix/README.md#testing).
### Structuring commits in pull requests
To simplify reviews, make it easy to split pull requests if deemed necessary, and to maintain clean and meaningful history of changes, you will be asked to update your PR if it does not follow the below rules:
* It must be possible to merge the PR branch into target using `git merge --ff-only`, ie. the source branch must be rebased on top of target.
* No merge commits in the source branch.
* All commits in the source branch must be self contained, meaning: it should be possible to treat each commit as a separate PR.
* Commits in the source branch must contain only related changes (related means the changes target a single problem/goal). For example, any refactorings should be isolated from the actual change implementation into separate commits.
* Tests, documentation, and changelog updates should be contained in the same commits as the actual code changes they relate to. An exception to this rule is when test or documentation changes are made in separate PR.
* Commit messages must be prefixed with one of the prefixes defined in [the list used by commit verification scripts](https://github.com/PostgREST/postgrest/blob/main/nix/tools/gitTools.nix#L11).
* Commit messages should contain a longer description of the purpose of the changes contained in the commit and, for non-trivial changes, a description of the changes themselves.
[](https://gitter.im/begriffs/postgrest)
@@ -59,8 +52,8 @@ Big thanks to our sponsors! You can join them by supporting PostgREST on [Patreo
## Usage
1.Download the binary ([latest release](https://github.com/PostgREST/postgrest/releases/latest))
for your platform.
1.See the docs for [how to install PostgREST on your platform](https://docs.postgrest.org/en/stable/explanations/install.html). You can also [use Docker](https://docs.postgrest.org/en/stable/explanations/install.html#docker).
2. Invoke for help:
```bash
@@ -132,7 +125,7 @@ and limited with - range headers. More about
## Data Integrity
Rather than relying on an Object Relational Mapper and custom
imperative coding, this system requires you put declarative constraints
imperative coding, this system requires you to put declarative constraints
directly into your database. Hence no application can corrupt your
data (including your API server).
@@ -145,14 +138,14 @@ and the [API guide](http://postgrest.org/en/stable/api.html).
## Supporting development
You can help PostgREST ongoing maintenance and development by:
- Making a regular donation through Patreon https://www.patreon.com/postgrest
- Alternatively, you can make a one-time donation via Paypal https://www.paypal.me/postgrest
You can help PostgREST ongoing maintenance and development by making a regular donation through Patreon https://www.patreon.com/postgrest
Every donation will be spent on making PostgREST better for the whole community.
## Contributing
Contributions are always welcome and appreciated. Please see the [Contributing guidelines](https://github.com/PostgREST/postgrest/blob/main/CONTRIBUTING.md).
[](https://gitter.im/begriffs/postgrest)
*`Building a Contacts List with PostgREST and Vue.js <https://www.youtube.com/watch?v=iHtsALtD5-U>`_ -
In this video series, DigitalOcean shows how to build and deploy an Nginx + PostgREST(using a managed PostgreSQL database) + Vue.js webapp in an Ubuntu server droplet.
*`PostgREST + Auth0: Create REST API in mintutes, and add social login using Auth0 <https://samkhawase.com/blog/postgrest/>`_ - A step-by-step tutorial to show how to dockerize and integrate Auth0 to PostgREST service.
*`PostgREST + PostGIS API tutorial in 5 minutes <https://gis-ops.com/postgrest-postgis-api-tutorial-geospatial-api-in-5-minutes/>`_ -
In this tutorial, GIS • OPS shows how to perform PostGIS calculations through PostgREST :ref:`functions` interface.
*`PostgREST + Auth0: Create REST API in minutes, and add social login using Auth0 <https://samkhawase.com/blog/postgrest-1-introduction/>`_ - A step-by-step tutorial to show how to dockerize and integrate Auth0 to PostgREST service.
*`"CodeLess" backend using postgres, postgrest and oauth2 authentication with keycloak <https://www.mathieupassenaud.fr/codeless_backend/>`_ -
A step-by-step tutorial for using PostgREST with KeyCloak(hosted on a managed service).
*`delibrium-postgrest <https://gitlab.com/delibrium/delibrium-postgrest/>`_ - example school API and front-end in Vue.js
*`ETH-transactions-storage <https://github.com/Adamant-im/ETH-transactions-storage>`_ - indexer for Ethereum to get transaction list by ETH address
*`fullstack template <https://github.com/jenstroeger/fullstack-webapp-template>`_ - a complete fullstack webapp template using PG as db and message queue, Python and Dramatiq to implement async jobs, db migrations, test runners, and more.
*`general <https://github.com/PierreRochard/general>`_ - example auth back-end
*`guild-operators <https://github.com/cardano-community/koios-artifacts/tree/main/files/grest>`_ - example queries and functions that the Cardano Community uses for their Guild Operators' Repository
*`PostGUI <https://github.com/priyank-purohit/PostGUI>`_ - React Material UI admin panel
@@ -31,65 +31,60 @@ This section talks briefly about various important modules.
Main
----
The starting point of the program is `Main.hs <https://github.com/PostgREST/postgrest/blob/main/main/Main.hs>`_.
The starting point of the program is `Main.hs <https://github.com/PostgREST/postgrest/blob/v16/src/executable/Main.hs>`_.
CLI
---
Main then calls `CLI.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/CLI.hs>`_, which is in charge of :ref:`cli`.
Main then calls `CLI.hs <https://github.com/PostgREST/postgrest/blob/v16/src/library/PostgREST/CLI.hs>`_, which is in charge of :ref:`cli`.
App
---
`App.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/App.hs>`_ is then in charge of composing the different modules.
`App.hs <https://github.com/PostgREST/postgrest/blob/v16/src/library/PostgREST/App.hs>`_ is then in charge of composing the different modules.
Auth
----
`Auth.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Auth.hs>`_ is in charge of :ref:`authn`.
`Auth.hs <https://github.com/PostgREST/postgrest/blob/v16/src/library/PostgREST/Auth.hs>`_ is in charge of :ref:`authn`.
Api Request
-----------
`ApiRequest.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/ApiRequest.hs>`_ is in charge of parsing the URL query string (following PostgREST syntax), the request headers, and the request body.
`ApiRequest.hs <https://github.com/PostgREST/postgrest/blob/v16/src/library/PostgREST/ApiRequest.hs>`_ is in charge of parsing the URL query string (following PostgREST syntax), the request headers, and the request body.
A request might be rejected at this level if it's invalid. For example when providing an unknown media type to PostgREST or using an unknown HTTP method.
Plan
----
Using the Schema Cache, `Plan.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Plan.hs>`_ fills in out-of-band SQL details (like an ``ON CONFLICT (pk)`` clause) required to complete the user request.
Using the Schema Cache, `Plan.hs <https://github.com/PostgREST/postgrest/blob/v16/src/library/PostgREST/Plan.hs>`_ generates an internal AST, filling out-of-band SQL details (like an ``ON CONFLICT (pk)`` clause) required to complete the user request.
A request might be rejected at this level if it's invalid. For example when doing resource embedding on a nonexistent resource.
Query
-----
`Query.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Query.hs>`_ generates the SQL queries (parametrized and prepared) required to satisfy the user request.
`Query.hs <https://github.com/PostgREST/postgrest/blob/v16/src/library/PostgREST/Query.hs>`_ generates the SQL queries (parametrized and prepared) required to satisfy the user request.
Only at this stage a connection from the pool might be used.
Schema Cache
------------
`SchemaCache.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/SchemaCache.hs>`_ is in charge of :ref:`schema_cache`.
`SchemaCache.hs <https://github.com/PostgREST/postgrest/blob/v16/src/library/PostgREST/SchemaCache.hs>`_ is in charge of :ref:`schema_cache`.
Config
------
`Config.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Config.hs>`_ is in charge of :ref:`configuration`.
`Config.hs <https://github.com/PostgREST/postgrest/blob/v16/src/library/PostgREST/Config.hs>`_ is in charge of :ref:`configuration`.
Admin
-----
`Admin.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Admin.hs>`_ is in charge of the :ref:`admin_server`.
HTTP
----
The HTTP server is provided by `Warp <https://aosabook.org/en/posa/warp.html>`_.
`Admin.hs <https://github.com/PostgREST/postgrest/blob/v16/src/library/PostgREST/Admin.hs>`_ is in charge of the :ref:`admin_server`.
Listener
--------
`Listener.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Listener.hs>`_ is in charge of the :ref:`listener`.
`Reload.hs <https://github.com/PostgREST/postgrest/blob/v16/src/library/PostgREST/AppState/Reload.hs>`_ is in charge of the :ref:`listener`.
@@ -13,9 +13,9 @@ A role can be thought of as either a database user, or a group of database users
Roles for Each Web User
-----------------------
PostgREST can accommodate either viewpoint. If you treat a role as a single user then the :ref:`jwt_impersonation` does most of what you need. When an authenticated user makes a request PostgREST will switch into the database role for that user, which in addition to restricting queries, is available to SQL through the :code:`current_user` variable.
PostgREST can accommodate either viewpoint. If you treat a role as a single user then :ref:`user_impersonation` does most of what you need. When an authenticated user makes a request PostgREST will switch into the database role for that user, which in addition to restricting queries, is available to SQL through the :code:`current_user` variable.
You can use row-level security to flexibly restrict visibility and access for the current user. Here is an `example <https://www.2ndquadrant.com/en/blog/application-users-vs-row-level-security/>`_ from Tomas Vondra, a chat table storing messages sent between users. Users can insert rows into it to send messages to other users, and query it to see messages sent to them by other users.
You can use row-level security to flexibly restrict visibility and access for the current user. Here is an `example <https://www.enterprisedb.com:443/blog/application-users-vs-row-level-security>`_ from Tomas Vondra, a chat table storing messages sent between users. Users can insert rows into it to send messages to other users, and query it to see messages sent to them by other users.
..code-block::postgres
@@ -163,7 +163,7 @@ Another option is to define the function with the :code:`SECURITY DEFINER` optio
..code-block::postgres
-- login as a user wich has privileges on the private schemas
-- login as a user which has privileges on the private schemas
PostgREST works with all PostgreSQL versions starting from 9.6.
PostgREST works with all PostgreSQL versions still `officially supported <https://www.postgresql.org/support/versioning/>`_.
..note::
Support for PostgreSQL versions 9.6, 10 and 11 is deprecated. From this on version onwards, PostgREST will only support non-end-of-life PostgreSQL versions.
See https://www.postgresql.org/support/versioning/.
Running PostgREST
=================
@@ -80,8 +75,12 @@ You can get the `official PostgREST Docker image <https://hub.docker.com/r/postg
..code-block::bash
# pull the latest version
docker pull postgrest/postgrest
# to pull a particular version, use one of the versions on https://hub.docker.com/r/postgrest/postgrest/tags
docker pull postgrest/postgrest:<version>
To configure the container image, use :ref:`env_variables_config`.
There are two ways to run the PostgREST container: with an existing external database, or through docker-compose.
@@ -147,6 +146,7 @@ To avoid having to install the database at all, you can run both it and the serv
ports:
- "3000:3000"
environment:
PGRST_SERVER_HOST:0.0.0.0# necessary for `postgrest --ready` flag to work
@@ -181,6 +181,23 @@ If you want to have a visual overview of your API in your browser you can add sw
With this you can see the swagger-ui in your browser on port 8080.
.._docker_cpu_contraint:
Docker Resource Constraints
---------------------------
PostgREST does not support ``--cpus```constraint option <https://docs.docker.com/engine/containers/resource_constraints/#configure-the-default-cfs-scheduler>`_.
As a workaround, you may use the `GHC RTS <https://ghc.gitlab.haskell.org/ghc/doc/users_guide/runtime_control.html#runtime-system-rts-options>`_``-N`` option. For instance, to limit it to 2 CPU cores, do:
@@ -43,7 +43,7 @@ As in :ref:`sql_user_management`, we create the :code:`pgcrypto` and :code:`pgjw
CREATEEXTENSIONpgcryptoWITHSCHEMAext_pgcrypto;
Concerning the `pgjwt extension <https://github.com/michelp/pgjwt>`_, please cf. to :ref:`client_auth`.
Concerning the `pgjwt extension <https://github.com/michelp/pgjwt>`_, please cf. to :ref:`jwt-from-sql`.
..code-block::postgres
@@ -157,7 +157,7 @@ Here we use the username instead of the email address to identify a user.
Logins
~~~~~~
As described in :ref:`client_auth`, we'll create a JWT token inside our login function. Note that you'll need to adjust the secret key which is hard-coded in this example to a secure (at least thirty-two character) secret of your choosing.
As described in :ref:`jwt-from-sql`, we'll create a JWT token inside our login function. Note that you'll need to adjust the secret key which is hard-coded in this example to a secure (at least thirty-two character) secret of your choosing.
As mentioned on :ref:`jwt_generation`, an external service can provide user management and coordinate with the PostgREST server using JWT. It’s also possible to support logins entirely through SQL. It’s a fair bit of work, so get ready.
As mentioned on :ref:`jwt_generation`, an external service can provide user management and coordinate with the PostgREST server using JWT. It's also possible to support logins entirely through SQL. It's a fair bit of work, so get ready.
Storing Users and Passwords
---------------------------
@@ -110,6 +110,8 @@ Then, add ``db-anon-role`` to the configuration file to allow anonymous requests
@@ -318,144 +318,6 @@ You can insert a new product using a JSON object for the ``extra_info`` column:
To query and filter the data see :ref:`json_columns` for a complete reference.
.._ww_postgis:
PostGIS
-------
You can use the string representation for `PostGIS <https://postgis.net/>`_ data types such as ``geometry`` or ``geography`` (you need to `install PostGIS <https://postgis.net/documentation/getting_started/>`_ first).
..code-block::postgres
-- Activate the postgis module in the current database
createextensionifnotexistspostgis;
createtablecoverage(
idintprimarykey,
nametextunique,
areageometry
);
To add areas in polygon format, you can use string representation:
Now, when you request the information, PostgREST will automatically cast the ``area`` column into a ``Polygon`` geometry type. Although this is useful, you may need the whole output to be in `GeoJSON <https://geojson.org/>`_ format out of the box, which can be done by including the ``Accept: application/geo+json`` in the request. This will work for PostGIS versions 3.0.0 and up and will return the output as a `FeatureCollection Object <https://www.rfc-editor.org/rfc/rfc7946#section-3.3>`_:
..code-block::bash
curl "http://localhost:3000/coverage"\
-H "Accept: application/geo+json"
..code-block::json
{
"type":"FeatureCollection",
"features":[
{
"type":"Feature",
"geometry":{
"type":"Polygon",
"coordinates":[
[[0,0],[1,0],[1,1],[0,1],[0,0]]
]
},
"properties":{
"id":1,
"name":"small"
}
},
{
"type":"Feature",
"geometry":{
"type":"Polygon",
"coordinates":[
[[0,0],[10,0],[10,10],[0,10],[0,0]]
]
},
"properties":{
"id":2,
"name":"big"
}
}
]
}
If you need to add an extra property, like the area in square units by using ``st_area(area)``, you could add a generated column to the table and it will appear in the ``properties`` key of each ``Feature``.
PostgREST is a standalone web server that turns your PostgreSQL database directly into a RESTful API. The structural constraints and permissions in the database determine the API endpoints and operations.
.. The static/empty.png(created with `convert -size 320x95 xc:#fcfcfc empty.png`) is an ugly workaround
to create space and center the logos. It's not easy to layout with restructuredText.
.. .. image:: _static/empty.png
:target: #sponsors
.. image:: _static/empty.png
:target: #sponsors
|
@@ -113,12 +111,22 @@ PostgREST has a focused scope. It works well with other tools like Nginx. This f
Getting Support
----------------
The project has a friendly and growing community. For discussions, use the Github `discussions page <https://github.com/PostgREST/postgrest/discussions>`_ or join our `chat room <https://gitter.im/begriffs/postgrest>`_. You can also report or search for bugs/features on the Github `issues <https://github.com/PostgREST/postgrest/issues>`_ page.
The project has a friendly and growing community. For discussions, use the Github `discussions page <https://github.com/PostgREST/postgrest/discussions>`_. You can also report or search for bugs/features on the Github `issues <https://github.com/PostgREST/postgrest/issues>`_ page.
Release Notes
-------------
Releases
--------
The release notes are published on `PostgREST's GitHub release page <https://github.com/PostgREST/postgrest/releases>`_.
-``MAJOR``: feature release, may deprecate or remove things.
-``PATCH``: fix/security release only, no features and no behavior changes.
MAJOR releases are published twice a year, with their scope and target dates tracked through `GitHub milestones <https://github.com/PostgREST/postgrest/milestones>`_.
PATCH releases are published on an as-needed basis.
Starting from ``v14.0``, only even-numbered MAJOR versions are released, reserving odd-numbered MAJOR versions for development.
All releases are published on `PostgREST's GitHub release page <https://github.com/PostgREST/postgrest/releases>`_, along with the corresponding upgrade guides.
Tutorials
---------
@@ -215,22 +223,14 @@ In Production
Here are some companies that use PostgREST in production.
Nixpkgs contains a `NixOS module to run PostgREST <https://search.nixos.org/options?channel=unstable&query=services.postgrest&type=options>`_, which can be enabled with ``services.postgrest.enable = true``.
A PostgreSQL server can be enabled on the same machine with ``services.postgresql.enable = true``. Connections will use the name of the system user as user and database names by default, in this case ``postgrest``.
A minimal example could look like this:
..code-block::nix
{
pkgs,
...
}:
{
services.postgresql={
enable=true;
initialScript=pkgs.writeText"init.sql"''
CREATEROLEpostgrestLOGINNOINHERIT;
CREATEROLEanonROLEpostgrest;
'';
};
services.postgrest={
enable=true;
settings.db-anon-role="anon";
settings.db-uri.dbname="postgres";
};
}
This will expose the PostgREST server on localhost on the NixOS machine and allow anonymous access.
..tip::
NixOS also allows to quickly spin up different PostgreSQL versions or even forks this way. For example, to test the current beta version of `OrioleDB <https://www.orioledb.com>`_, use ``services.postgresql.package = pkgs.orioledb``.
To work with `PostGIS <https://postgis.net/>`_ data types such as ``geometry`` or ``geography``, you'll need to `install PostGIS <https://postgis.net/documentation/getting_started/>`_ first.
..code-block::postgres
-- Activate the postgis module in the current database
PostgREST supports the `standard <https://www.iana.org/assignments/media-types/application/geo+json>`_``application/geo+json`` media type which can be used to get the output in `GeoJSON <https://geojson.org/>`_ format. This will work for PostGIS versions 3.0.0 and up and will return the output as a `FeatureCollection Object <https://www.rfc-editor.org/rfc/rfc7946#section-3.3>`_:
..code-block::bash
curl "http://localhost:3000/coverage"\
-H "Accept: application/geo+json"
..code-block::json
{
"type":"FeatureCollection",
"features":[
{
"type":"Feature",
"geometry":{
"type":"Polygon",
"coordinates":[
[[0,0],[1,0],[1,1],[0,1],[0,0]]
]
},
"properties":{
"id":1,
"name":"small"
}
},
{
"type":"Feature",
"geometry":{
"type":"Polygon",
"coordinates":[
[[0,0],[10,0],[10,10],[0,10],[0,0]]
]
},
"properties":{
"id":2,
"name":"big"
}
}
]
}
Using generated columns
-----------------------
If you need to add an extra property, like the area in square units by using ``st_area(area)``, you could add a generated column to the table and it will appear in the ``properties`` key of each ``Feature``.
PostgREST provides an admin server that can be enabled by setting :ref:`admin-server-port`.
PostgREST provides an admin server that can be enabled by setting :ref:`admin-server-port` or :ref:`admin-server-unix-socket`.
Multiple PostgREST instances can share the same public API host and port when :ref:`server-reuseport` is enabled. Admin ports are not shared: give each instance a different :ref:`admin-server-port`, otherwise the new instance will fail to start.
.._health_check:
@@ -12,7 +14,7 @@ Health Check
You can enable a health check to verify if PostgREST is available for client requests. Also to check the status of its internal state.
Two endpoints ``live`` and ``ready`` will then be available.
Two endpoints ``live`` and ``ready`` will then be available. Both these endpoints reply with a status code and empty response body.
..important::
@@ -55,23 +57,6 @@ Metrics
Provides :ref:`metrics`.
Runtime Configuration
=====================
Provides a ``config`` endpoint that returns the runtime :ref:`configuration`.
..code-block::bash
curl "http://localhost:3001/config"
..code-block::
db-aggregates-enabled = false
db-anon-role = "web_anon"
db-channel = "pgrst"
db-channel-enabled = false
...
Runtime Schema Cache
====================
@@ -89,5 +74,4 @@ Provides the ``schema_cache`` endpoint that prints the runtime :ref:`schema_cach
Aggregate functions allow you to summarize data by performing calculations across groups of rows. For instance, if you have an ``orders`` table that has an ``amount`` column, you could use an aggregate function to get the sum of the ``amount`` column, either for all rows, or for each group of rows that share specific values, for instance all rows that share the same ``order_date``.
PostgREST supports the following aggregate functions: ``avg()``, ``count()``, ``max()``, ``min()``, and ``sum()``.
Please refer to the `section on aggregate functions in the PostgreSQL documentation <https://www.postgresql.org/docs/current/functions-aggregate.html>`_ for a detailed explanation of these functions.
..note::
Aggregate functions are *disabled* by default in PostgREST, as without appropriate safeguards, aggregate functions can create performance problems. See :ref:`db-aggregates-enabled` for further details.
Aggregate functions are *disabled* by default in PostgREST, because they can create performance problems without appropriate safeguards.
See :ref:`db-aggregates-enabled` for further details.
PostgREST supports the following aggregate functions: ``avg()``, ``count()``, ``max()``, ``min()``, and ``sum()``. Please refer to the `section on aggregate functions in the PostgreSQL documentation <https://www.postgresql.org/docs/current/functions-aggregate.html>`_for a detailed explanation of these functions.
To use an aggregate function, you append the function to a value in the ``select`` parameter, like so:
To use an aggregate function, append it to a column in the ``select``parameter, like so:
With the above query, PostgREST will return a single row with a single column named ``sum`` that contains the sum of all the values in the ``amount`` column:
This will return a ``sum`` of all the values of the ``amount`` column in a single row:
..code-block::json
@@ -26,15 +26,29 @@ With the above query, PostgREST will return a single row with a single column na
}
]
You can use multiple aggregate functions by just adding more columns with aggregate functions to the ``select`` parameter.
You can ``select`` multiple aggregate functions at the same time (you may need to :ref:`rename them <renaming_columns>` to disambiguate).
To group by other columns, you simply add those columns to the ``select`` parameter. For instance:
This will return a row for each unique value in the ``order_date`` column, with the sum and average of the ``amount`` column for all rows that share the same ``order_date``:
This will get the sum and average of the amounts grouped by each unique value in the ``order_date`` column:
..code-block::json
@@ -51,67 +65,55 @@ This will return a row for each unique value in the ``order_date`` column, with
}
]
..note::
Aggregate functions work alongside other PostgREST features, like :ref:`h_filter`, :ref:`json_columns`, and :ref:`ordering`. Please note at this time aggregate functions are not compatible with :ref:`domain_reps`. Additionally, PostgreSQL's ``HAVING`` clause and ordering by aggregated columns are not yet supported.
The Case of ``count()``
===========================
The ``count()`` Aggregate
=========================
..note::
Before the addition of aggregate functions, it was possible to count by adding ``count`` (without parentheses) to the ``select`` parameter. While this is still supported, it may be deprecated in the future, and thus use of this legacy feature is **not recommended.** Please use ``count()`` (with parentheses) instead.
Before the addition of aggregate functions, it was possible to count by adding ``count`` (without parentheses) to the ``select`` parameter.
While this is still supported, it may be deprecated in the future, and thus use of this legacy feature is **not recommended**.
Please use ``count()`` (with parentheses) instead.
``count()`` is treated specially, as it can be used without an associated column. Take for example the following query:
``count()`` is a special case because it can be used with or without an aggregated column. For example:
When ``count()`` is used with an associated column, its behavior is slightly different: It will return the count of all values that are not ``NULL``. This is due to how PostgreSQL itself implements the ``count()`` function.
Renaming and Casting
====================
Renaming Aggregates
-------------------
Just like with other columns, you can rename aggregated columns too. See :ref:`renaming_columns` for details.
Renaming columns is especially helpful in the context of aggregate functions, as by default a column with an aggregate function applied will take on the name of the applied aggregate function. You may want to provide a more semantically meaningful name or prevent collisions when using multiple aggregate functions of the same type.
Note that there is a difference between the result of ``count()`` and ``observation.count()``.
The former counts the whole row, while the latter counts the non ``NULL`` values of the ``observation`` column (both grouped by ``order_date``).
This is due to how PostgreSQL itself implements the ``count()`` function.
Casting Aggregates
------------------
==================
When applying an aggregate function to a column, you are able to cast both the value of the input to the aggregate function *and* the value of the output from the aggregate function. In both cases, the syntax works as described in :ref:`casting_columns`, with the only difference being the placement of the cast.
It is :ref:`possible to cast <casting_columns>` the aggregated column or the aggregate itself, or both at the same time.
Casting the Value of the Input
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Casting the Aggregated Column
-----------------------------
For instance, imagine that the ``orders`` table has a JSON column, ``order_details``, and this column contains a JSON object that has a key,``tax_amount``. Let's say you want to get the sum of the tax amount for every order. You can use the ``->`` or ``->>`` operators to extract the value with this key (see :ref:`json_columns`), but these operators will return values of the types JSON and``text``respectively, and neither of these types can be used with ``sum()``.
Therefore, you will need to first cast the input value to a type that is compatible with ``sum()`` (e.g. ``numeric``). Casting the input value is done in exactly the same way as casting any other value:
For example, let's say that``orders`` has an ``order_details``:ref:`JSON column <json_columns>` with a``tax_amount``key.
We cannot sum ``tax_amount`` directly because using ``->`` or ``->>`` will return the data in ``json`` or ``text`` format.
So we need to cast it to a compatible type (e.g. ``numeric``) right before the aggregate function:
With this, you will receive the sum of the casted ``tax_amount`` value:
..code-block::json
[
@@ -120,17 +122,15 @@ With this, you will receive the sum of the casted ``tax_amount`` value:
}
]
Casting the Value of the Output
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Casting the Aggregate
---------------------
Now let's return to an example involving the ``amount`` column of the ``orders`` table. Imagine that we want to get the rounded average of the ``amount`` column. One way to do this is to use the ``avg()`` aggregate function and then to cast the output value of the function to ``int``. To cast the value of the output of the function, we simply place the cast *after* the aggregate function:
For instance, if we wanted to round the average of the ``amount`` column, we could do so by casting ``avg()`` to an ``int``:
You will then receive the rounded average as the result:
..code-block::json
[
@@ -139,27 +139,22 @@ You will then receive the rounded average as the result:
}
]
Of course, you can use both input and output casts at the same time, if you so desire.
Aggregates and Resource Embedding
=================================
You can group an aggregate function by an :ref:`embedded resource <resource_embedding>` and also use the aggregates inside them.
Using Aggregate Functions with Resource Embedding
=================================================
Grouping by an Embedded Resource
--------------------------------
Aggregate functions can be used in conjunction with :ref:`resource_embedding`. You can use embedded resources as grouping columns, use aggregate functions within the context of an embedded resource, or use columns from a spreaded resource as grouping columns or as inputs to aggregate functions.
Using Embedded Resources as Grouping Columns
--------------------------------------------
Using an embedded resource as a grouping column allows you to use data from an association to group the results of an aggregation.
For example, imagine that the ``orders`` table from the examples above is related to a ``customers`` table. If you want to get the sum of the ``amount`` column grouped by the ``name`` column from the ``customers`` table, you can include the customer name, using the standard :ref:`resource_embedding` syntax, and perform a sum on the ``amount`` column.
Similar to grouping by columns, aggregate functions can also be grouped by embedded resources.
For example, let's say that the ``orders`` table is related to a ``customers`` table.
To get the sum of the ``amount`` column grouped by the ``name`` column from the ``customers`` table, we would do the following:
You will then get the summed amount, along with the embedded customer resource:
..code-block::json
[
@@ -177,15 +172,16 @@ You will then get the summed amount, along with the embedded customer resource:
}
]
..note::
The previous example uses a has-one association to demonstrate this functionality, but you may also use has-many associations as grouping columns, although there are few obvious use cases for this.
The previous example uses a "to-one" relationship, but this can be done on "to-many" relationships as well (although there are few obvious use cases).
Using Aggregate Functions Within the Context of an Embedded Resource
This also works in a similar way for :ref:`spread embedded resources <spread_embed>`.
For example, ``select=amount.sum(),...customers(name)`` would sum the ``amount`` grouped by the ``name`` column.
When embedding a resource, you can apply aggregate functions to columns from the associated resource to perform aggregations within the context of an embedded resource.
Using Aggregates Inside Embedded Resources
------------------------------------------
Continuing with the example relationship between ``orders`` and ``customers`` from the previous section, imagine that you want to fetch the ``name``, ``city``, and ``state`` for each customer, along with the sum of amount of the customer's orders, grouped by the order date. This can be done in the following way:
Using the relationship from the previous example, let's take all the ``customers`` and embed their ``orders``.
If we also want to get the total ``amount`` grouped by the ``order_date`` of the ``orders``, we would do the following:
..code-block::bash
@@ -226,51 +222,20 @@ Continuing with the example relationship between ``orders`` and ``customers`` fr
}
]
In this example, the ``amount`` column is summed and grouped by the ``order_date``*within* the context of the embedded resource. That is, the ``name``, ``city``, and ``state`` from the ``customers`` table have no bearing on the aggregation performed in the context of the ``orders`` association; instead, each aggregation can be seen as being performed independently on just the orders belonging to a particular customer, using only the data from the embedded resource for both grouping and aggregation.
Note that the aggregate is done within the embedded resource ``orders``.
It is not affected by any of the columns from the top-level relationship ``customers``.
Using Columns from a Spreaded Resource
--------------------------------------
Aggregates in To-One Spreads
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When you :ref:`spread an embedded resource <spread_embed>`, the columns from the spreaded resource are treated as if they were columns of the top-level resource, both when using them as grouping columns and when applying aggregate functions to them.
Grouping with Columns from a Spreaded Resource
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For instance, assume you want to sum the ``amount`` column from the ``orders`` table, using the ``city`` and ``state`` columns from the ``customers`` table as grouping columns. To achieve this, you may select these two columns from the ``customers`` table and spread them; they will then be used as grouping columns:
All the aggregates inside a :ref:`one-to-one or many-to-one spread embedded resource <spread_to_one_embed>` will be hoisted to the top-level relationship.
In other words, it will behave as if the aggregate was done in the top-level relationship itself. For example:
Now imagine that the ``customers`` table has a ``joined_date`` column that represents the date that the customer joined. You want to get both the most recent and the oldest ``joined_date`` for customers that placed an order on every distinct order date. This can be expressed as follows:
As columns from a spreaded resource are treated as if they were columns from the top-level resource, the ``max()`` and ``min()`` are applied *within* the context of the top-level, rather than within the context of the embedded resource, as in the previous section.
The result will be the same as if the aggregations were applied to columns from the top-level:
This will take the ``max`` and ``min``subscription date of every customer and group it by the ``order_date``column:
..code-block::json
@@ -286,3 +251,7 @@ The result will be the same as if the aggregations were applied to columns from
"min":"2016-02-11"
}
]
..note::
Aggregates inside to-many spreads are not supported
@@ -173,4 +173,4 @@ Domain Representations avoid all the above drawbacks. Their only drawback is tha
Why not create a `base type <https://www.postgresql.org/docs/current/sql-createtype.html#id-1.9.3.94.5.8>`_ instead? ``CREATE TYPE app_uuid (INTERNALLENGTH = 22, INPUT = app_uuid_parser, OUTPUT = app_uuid_formatter)``.
Creating base types need superuser, which is restricted on cloud hosted databases. Additionally this way lets “how the data is presented” dictate “how the data is stored” which would be backwards.
Creating base types need superuser, which is restricted on cloud hosted databases. Additionally this way lets "how the data is presented" dictate "how the data is stored" which would be backwards.
If you want to pass multiple JSON objects to a Postgres function (an array of objects), you can create a function with a parameter of type ``json`` or ``jsonb``.
Within the curl request, this JSON must be embedded in an object where they key matches the same name as the function's ``json`` or ``jsonb`` parameter.
This will allow you to loop over the array of JSON objects within the Postgres function.
This practice may allow you to reduce the number of ``curl`` requests required to accomplish a task.
For instance, assume we have created this function in the database.
@@ -97,10 +151,6 @@ For this the ``Content-Type: application/json`` header must be included in the r
If an overloaded function has a single ``json`` or ``jsonb`` unnamed parameter, PostgREST will call this function as a fallback provided that no other overloaded function is found with the parameters sent in the POST request.
..warning::
Sending the JSON request body as a single argument is also possible with :ref:`Prefer: params=single-object <prefer_params>` but this method is **deprecated**.
.._function_single_unnamed:
Functions with a single unnamed parameter
@@ -264,6 +314,23 @@ Let's get its :ref:`explain_plan` when calling it with filters applied:
Notice there's no "Function Scan" node in the plan, which tells us it has been inlined.
Horizontal Filtering
~~~~~~~~~~~~~~~~~~~~
Table-valued functions support horizontal filtering on selected and unselected columns.
For example, the following RPC with filter on unselected column returns:
@@ -15,7 +15,7 @@ Using these domains, :ref:`functions <functions>` can become handlers and `user-
..important::
- PostgREST vendor media types (``application/vnd.pgrst.plan``, ``application/vnd.pgrst.object`` and ``application/vnd.pgrst.array``) cannot be overriden.
- PostgREST vendor media types (``application/vnd.pgrst.plan``, ``application/vnd.pgrst.object`` and ``application/vnd.pgrst.array``) cannot be overridden.
- Long media types like ``application/vnd.openxmlformats-officedocument.wordprocessingml.document`` cannot be expressed as domains since they surpass `PostgreSQL identifier length <https://www.postgresql.org/docs/current/limits.html#LIMITS-TABLE>`_.
@@ -9,6 +9,8 @@ PostgREST automatically serves a full `OpenAPI <https://www.openapis.org/>`_ des
By default, this output depends on the permissions of the role that is contained in the JWT role claim (or the :ref:`db-anon-role` if no JWT is sent). If you need to show all the endpoints disregarding the role's permissions, set the :ref:`openapi-mode` config to :code:`ignore-privileges`.
When following privileges, the output reflects both the granted HTTP methods and columns: a relation with only ``SELECT`` will only expose ``GET``, a relation with only ``INSERT`` will only expose ``POST``, and column-level grants limit the columns shown on the table definitions and row filters.
For extra customization, the OpenAPI output contains a "description" field for every `SQL comment <https://www.postgresql.org/docs/current/sql-comment.html>`_ on any database object. For instance,
..code-block::postgres
@@ -19,6 +21,9 @@ For extra customization, the OpenAPI output contains a "description" field for e
COMMENTONTABLEmonotremesIS
'Freakish mammals lay the best eggs for breakfast';
@@ -12,10 +12,9 @@ The following preferences are supported.
-``Prefer: return``. See :ref:`prefer_return`.
-``Prefer: count``. See :ref:`prefer_count`.
-``Prefer: resolution``. See :ref:`prefer_resolution`.
-``Prefer: missing``. See :ref:`bulk_insert_default`.
-``Prefer: missing``. See :ref:`prefer_missing`.
-``Prefer: max-affected``, See :ref:`prefer_max_affected`.
-``Prefer: tx``. See :ref:`prefer_tx`.
-``Prefer: params``. See :ref:`prefer_params`.
.._prefer_handling:
@@ -63,8 +62,12 @@ The server ignores unrecognized or unfulfillable preferences by default. You can
Timezone
========
The ``timezone`` preference allows you to change the `PostgreSQL timezone <https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-TIMEZONE>`_. It accepts all time zones in `pg_timezone_names <https://www.postgresql.org/docs/current/view-pg-timezone-names.html>`_.
..important::
``handling=lenient`` is ignored for ``timezone``. Invalid time zones always return an error.
The ``timezone`` preference allows you to change the `PostgreSQL timezone <https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-TIMEZONE>`_.
It accepts all time zones in `pg_timezone_names <https://www.postgresql.org/docs/current/view-pg-timezone-names.html>`_ and numeric offsets.
..code-block::bash
@@ -85,34 +88,35 @@ The ``timezone`` preference allows you to change the `PostgreSQL timezone <https
{"t":"2023-10-18T09:37:59.611-07:00"}
]
For an invalid time zone, PostgREST returns values with the default time zone (configured on ``postgresql.conf`` or as a setting on the :ref:`authenticator <roles>`).
Offsets are also accepted:
..code-block::bash
curl -i "http://localhost:3000/timestamps"\
-H "Prefer: timezone=Jupiter/Red_Spot"
-H "Prefer: timezone=05:30"
..code-block::http
HTTP/1.1200OK
Content-Type: application/json; charset=utf-8
Preference-Applied: timezone=05:30
..code-block::json
[
{"t":"2023-10-18T12:37:59.611+00:00"},
{"t":"2023-10-18T14:37:59.611+00:00"},
{"t":"2023-10-18T16:37:59.611+00:00"}
{"t":"2023-10-18T17:07:59.611+05:30"},
{"t":"2023-10-18T19:07:59.611+05:30"},
{"t":"2023-10-18T21:07:59.611+05:30"}
]
Note that there's no ``Preference-Applied`` in the response.
You can also use negative offsets like ``-03:00``.
However, with ``handling=strict``, an invalid time zone preference will throw an :ref:`error <pgrst122>`.
For an invalid time zone, PostgREST returns a database error.
@@ -197,12 +201,53 @@ The ``tx`` preference can be set to specify if the :ref:`transaction <transactio
{"id": 35, "name": "Project X"}
.._prefer_missing:
Missing
=======
When doing ``POST`` and ``PATCH`` requests, any missing columns in the payload will be inserted as ``null`` value by default. To use the ``DEFAULT`` column value instead, use the ``Prefer: missing=default`` header.
You can set a limit to the amount of resources affected in a request by sending ``max-affected`` preference. This feature works in combination with ``handling=strict`` preference. ``max-affected`` would be ignored with lenient handling. The "affected resources" are the number of rows returned by ``DELETE`` and ``PATCH`` requests. This is also supported through ``RPC`` calls.
You can set a limit to the amount of resources affected in a request by sending ``max-affected`` preference. This feature works in combination with ``handling=strict`` preference. ``max-affected`` would be ignored with lenient handling. The "affected resources" are the number of rows returned by ``DELETE`` and ``PATCH`` requests.
To illustrate the use of this preference, consider the following scenario where the ``items`` table contains 14 rows.
@@ -225,30 +270,34 @@ To illustrate the use of this preference, consider the following scenario where
"hint":null
}
.._prefer_params:
Single JSON object as Function Parameter
----------------------------------------
..warning::
Using this preference is **deprecated** in favor of :ref:`function_single_json`.
:code:`Prefer:params=single-object` allows sending the JSON request body as the single argument of a :ref:`function <functions>`.
With :ref:`RPC <functions>`, the preference is honored completely on the basis of the number of rows returned in the result set of the function. This can be useful for complex mutation queries using `data-modifying statements <https://www.postgresql.org/docs/current/queries-with.html#QUERIES-WITH-MODIFYING>`_. A simple example:
..code-block::postgres
CREATEFUNCTIONmult_them(paramjson)RETURNSintAS$$
SELECT(param->>'x')::int*(param->>'y')::int
CREATEFUNCTIONtest.delete_items()
RETURNSSETOFitemsAS$$
DELETEFROMitemsWHEREid<15RETURNING*;
$$LANGUAGESQL;
..code-block::bash
curl "http://localhost:3000/rpc/mult_them"\
-X POST -H "Content-Type: application/json"\
-H "Prefer: params=single-object"\
-d '{ "x": 4, "y": 2 }'
curl -i "http://localhost:3000/rpc/delete_items"\
-H "Content-Type: application/json"\
-H "Prefer: handling=strict, max-affected=10"
..code-block::http
HTTP/1.1400Bad Request
..code-block::json
8
{
"code":"PGRST124",
"message":"Query result exceeds max-affected preference constraint",
"details":"The query affects 14 rows",
"hint":null
}
..note::
It is important for functions to return ``SETOF`` or ``TABLE`` when called with ``max-affected`` preference. A violation of this would cause a :ref:`PGRST128 <pgrst128>` error.
@@ -143,7 +143,7 @@ Since the table name is plural, we can be more accurate by making it singular wi
One-to-many relationships
-------------------------
The **foreign key reference** establishes the inverse one-to-many relationship. In this case, ``films`` returns as a JSON array because of the “to-many” end.
The **foreign key reference** establishes the inverse one-to-many relationship. In this case, ``films`` returns as a JSON array because of the "to-many" end.
..code-block::bash
@@ -209,18 +209,25 @@ The join table is also detected if the composite key has additional columns.
One-to-one relationships
------------------------
One-to-one relationships are detected in two ways.
One-to-one relationships are detected in two ways. (We'll use the ``films`` and ``technical_specs`` tables from the :ref:`sample film database <erd_film>` as an example).
- When the foreign key is a primary key as specified in the :ref:`sample film database <erd_film>`.
- When the foreign key has a unique constraint.
- When the foreign key is also a primary key.
.. code-block:: postgres
create table technical_specs(
film_idintreferencesfilms(id)unique,
runtimetime,
cameratext,
soundtext
film_id int references films(id) primary key
-- ...
);
- Or when the foreign key has a unique constraint.
.. code-block:: postgres
create table technical_specs(
id int primary key
, film_id int references films(id) unique
-- ...
);
..code-block::bash
@@ -244,6 +251,12 @@ Computed Relationships
You can manually define relationships by using functions. This is useful for database objects that can't define foreign keys, like `Foreign Data Wrappers <https://wiki.postgresql.org/wiki/Foreign_data_wrappers>`_.
Computed relationships have good performance as their intended design enable `function inlining <https://wiki.postgresql.org/wiki/Inlining_of_SQL_functions#Inlining_conditions_for_table_functions>`_.
..important::
- Always use ``SETOF`` when creating computed relationships. Functions can return a table without using ``SETOF``, but bear in mind that PostgreSQL will not inline them. e.g. ``RETURNS <table_name>`` is not inlinable.
Assuming there's a foreign table ``premieres`` that we want to relate to ``films``.
..code-block::postgres
@@ -276,6 +289,10 @@ The name of the function ``film`` is arbitrary and can be used to do the embeddi
".."
]
..warning::
- Make sure to correctly label the ``to-one`` part of the relationship. When using the ``ROWS 1`` estimation, PostgREST will expect a single row to be returned. If that is not the case, it will unnest the embedding and return repeated values for the top level resource.
Now let's define the opposite one-to-many relationship.
..code-block::postgres
@@ -324,12 +341,6 @@ Thanks to overloaded functions, you can use the same function name for different
Computed relationships have good performance as their intended design enable `function inlining <https://wiki.postgresql.org/wiki/Inlining_of_SQL_functions#Inlining_conditions_for_table_functions>`_.
..warning::
- Always use ``SETOF`` when creating computed relationships. Functions can return a table without using ``SETOF``, but bear in mind that PostgreSQL will not inline them.
- Make sure to correctly label the ``to-one`` part of the relationship. When using the ``ROWS 1`` estimation, PostgREST will expect a single row to be returned. If that is not the case, it will unnest the embedding and return repeated values for the top level resource.
.._embed_disamb:
.._target_disamb:
.._hint_disamb:
@@ -927,7 +938,12 @@ Filters can also be applied on nested embedded resources:
The result will show the nested actors named Tom and order them by last name. Aliases can also be used instead of the resource names to filter the nested tables.
@@ -1138,14 +1154,19 @@ For example, to arrange the films in descending order using the director's last
Spread embedded resource
========================
On many-to-one and one-to-one relationships, you can "spread" the embedded resource. That is, remove the surrounding JSON object for the embedded resource columns.
You can modify the shape of the embedded resources by using the spread syntax (``...``).
.._spread_to_one_embed:
Spread To-One relationships
---------------------------
Spread on resources forming :ref:`one-to-one <one-to-one>` and :ref:`many-to-one <many-to-one>` relationships, will lift the embedded columns to the top object.
@@ -1153,17 +1174,148 @@ On many-to-one and one-to-one relationships, you can "spread" the embedded resou
[
{
"title":"Workers Leaving The Lumière Factory In Lyon",
"director_first_name":"Louis",
"director_last_name":"Lumière"
}
]
Note that there is no ``"directors"`` object. Also the embed columns can be aliased normally.
Note that there is no wrapping ``"directors"`` object, unlike regularly embedding :ref:`many-to-one <many-to-one>` relationships. Also note that embedded columns can be aliased normally.
You can use this to get the columns of a join table in a many-to-many relationship. For instance, to get films and its actors, but including the ``character`` column from the roles table:
.._spread_to_many_embed:
Spread To-Many relationships
----------------------------
Spread on resources forming :ref:`one-to-many <one-to-many>` and :ref:`many-to-many <many-to-many>` relationships, will convert the embedded columns into correlated arrays.
Note that ``films`` is no longer an array of objects, unlike regularly embedding :ref:`one-to-many`. The embedded columns become arrays and they're correlated-in the above result, we can say that "Pulp Fiction" premiered in 1994 and "Reservoir Dogs" in 1992.
Order in spread to-many
~~~~~~~~~~~~~~~~~~~~~~~
In the above example, the order of the values inside the correlated arrays is unspecified, but all the values are guaranteed to be in the same unspecified order.
You can order the correlated arrays explicitly. For example, to order by the film year:
You can use multiple spreads at any level. For example, let's spread ``technical_specs`` and ``roles`` into ``films`` and then spread ``films`` into ``directors``:
- All the ``film_*`` arrays are correlated-"Reservoir Dogs" premiered in 1992, its runtime is 1:39:00 and it has the following characters: ``[ "Mr. Pink", "Mr. White" ]``.
- The ``film_*`` arrays are ordered by ``year`` (due to ``films.order=year``).
- The bottom level array ``film_characters`` is ordered (due to ``films.roles.order=character``).
Spread a join table
-------------------
Spread can be used to move the columns of a join table in a :ref:`many-to-many <many-to-many>` to the top object. For instance, to get the ``character`` column of the ``roles`` join table into ``actors``:
@@ -1184,6 +1336,4 @@ You can use this to get the columns of a join table in a many-to-many relationsh
}
]
..note::
The spread operator ``...`` is borrowed from the Javascript `spread syntax <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax>`_.
@@ -16,6 +16,32 @@ Use the Accept request header to specify the acceptable format (or formats) for
curl "http://localhost:3000/people"\
-H "Accept: application/json"
..note::
The ordering of columns in the response isn't guaranteed to align with the order specified in the ``select`` clause. For example, with resource embedding:
This is in line with the `JSON schema spec <https://json-schema.org/draft/2020-12/json-schema-core#name-instance-data-model>`_:
*"object: An unordered set of properties mapping a string to an instance"*
.._builtin_media:
Builtin Media Type Handlers
@@ -25,7 +51,7 @@ Builtin handlers are offered for common standard media types.
*``text/csv`` and ``application/json``, for all API endpoints. See :ref:`tables_views` and :ref:`functions`.
*``application/openapi+json``, for the root endpoint. See :ref:`open-api`.
*``application/geo+json``, see :ref:`ww_postgis`.
*``application/geo+json``, see :ref:`application/geo+json`.
*``*/*``, resolves to ``application/json`` for API endpoints and to ``application/openapi+json`` for the root endpoint.
The following vendor media types handlers are also supported.
@@ -74,17 +100,15 @@ This returns
{"id":1}
with a :code:`Content-Type: application/vnd.pgrst.object+json`.
When a singular response is requested but no entries are found, the server responds with an error message and 406 Not Acceptable status code rather than the usual empty array and 200 status:
PostgREST can expose a single or multiple schema's tables, views and functions. The :ref:`active database role <roles>` must have the usage privilege on the schemas to access them.
..important::
``pg_catalog`` and ``information_schema`` are not allowed in :ref:`db-schemas`. This is done to prevent leaking sensitive information and hence they cannot be accessed directly. If you wish to expose objects of these schemas, expose another schema that contains wrapper views or functions over ``pg_catalog`` or ``information_schema`` objects.
@@ -72,7 +72,7 @@ imatch :code:`~*` ~* operator, see :ref:`pattern_matching`
in :code:`IN` one of a list of values, e.g. :code:`?a=in.(1,2,3)`
– also supports commas in quoted strings like
:code:`?a=in.("hi,there","yes,you")`
is :code:`IS` checking for exact equality (null,true,false,unknown)
is :code:`IS` checking for exact equality (null,not_null,true,false,unknown)
isdistinct :code:`IS DISTINCT FROM` not equal, treating :code:`NULL` as a comparable value
fts :code:`@@`:ref:`fts` using to_tsquery
plfts :code:`@@`:ref:`fts` using plainto_tsquery
@@ -82,7 +82,7 @@ cs :code:`@>` contains e.g. :code:`?tags=cs.{example,
cd :code:`<@` contained in e.g. :code:`?values=cd.{1,2,3}`
ov :code:`&&` overlap (have points in common), e.g. :code:`?period=ov.[2017-01-01,2017-06-30]`–
also supports array types, use curly braces instead of square brackets e.g.
:code:`?arr=ov.{1,3}`
:code:`?arr=ov.{1,3}`
sl :code:`<<` strictly left of, e.g. :code:`?range=sl.(1,10)`
sr :code:`>>` strictly right of
nxr :code:`&<` does not extend to the right of, e.g. :code:`?range=nxr.(1,10)`
@@ -175,25 +175,56 @@ To ensure best performance on larger data sets, an `appropriate index <https://w
Full-Text Search
~~~~~~~~~~~~~~~~
The :code:`fts`filter mentioned above has a number of options to support flexible textual queries, namely the choice of plain vs phrase search and the language used for stemming. Suppose that :code:`tsearch` is a table with column :code:`my_tsv`, of type `tsvector <https://www.postgresql.org/docs/current/datatype-textsearch.html>`_. The following examples illustrate the possibilities.
The :code:`fts`operator has a number of options to support flexible textual queries, namely the choice of plain vs phrase search and the language used for stemming.
The following examples illustrate the possibilities, assuming column :code:`my_tsv` is of type `tsvector <https://www.postgresql.org/docs/current/datatype-textsearch.html>`_.
Using `websearch_to_tsquery` requires PostgreSQL of version at least 11.0 and will raise an error in earlier versions of the database.
.._fts_to_tsvector:
Automatic ``tsvector`` conversion
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If the filtered column is not of type ``tsvector``, then it will be automatically converted using `to_tsvector() <https://www.postgresql.org/docs/current/functions-textsearch.html#TEXTSEARCH-FUNCTIONS-TABLE>`_.
This allows using the ``fts`` operator on ``text`` and ``json`` types out of the box.
To ensure this operation is fast, you need to create an index on the expression:
..code-block::postgres
CREATEINDEXidx_people_colONpeople
USINGGIN(to_tsvector('french',my_text_column));
.._v_filter:
@@ -525,45 +556,6 @@ To bulk insert JSON post an array of objects having all-matching keys
]
EOF
.._bulk_insert_default:
Bulk Insert with Default Values
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Any missing columns in the payload will be inserted as ``null`` values. To use the ``DEFAULT`` column value instead, use the ``Prefer: missing=default`` header.
{ "sku": "AC1022", "name": "New Cap", "price": 30 }
]
EOF
By default, upsert operates based on the primary key columns, you must specify all of them. You can also choose to ignore the duplicates with :code:`Prefer: resolution=ignore-duplicates`. This works best when the primary key is natural, but it's also possible to use it if the primary key is surrogate (example: "id serial primary key"). For more details read `this issue <https://github.com/PostgREST/postgrest/issues/1118>`_.
By default, upsert operates based on the primary key columns, so you must specify all of them.
You can also choose to ignore the duplicates with :code:`Prefer: resolution=ignore-duplicates`.
Upsert works best when the primary key is natural (e.g. ``sku``).
However, it can work with surrogate primary keys (e.g. ``id serial primary key``), if you also do a :ref:`bulk_insert` with :ref:`prefer_missing`:
After creating a table or changing its primary key, you must refresh PostgREST schema cache for upsert to work properly. To learn how to refresh the cache see :ref:`schema_reloading`.
@@ -700,31 +714,6 @@ Deletions also support :ref:`prefer_return`, :ref:`resource_embedding` and :ref:
Beware of accidentally deleting all rows in a table. To learn to prevent that see :ref:`block_fulltable`.
.._limited_update_delete:
Limited Update/Delete
=====================
You can limit the amount of affected rows by :ref:`update` or :ref:`delete` with the ``limit`` query parameter. For this, you must add an explicit ``order`` on a unique column(s).
If your table has no unique columns, you can use the `ctid <https://www.postgresql.org/docs/current/ddl-system-columns.html>`_ system column.
Using ``offset`` to target a different subset of rows is also possible.
..note::
There is no native ``UPDATE...LIMIT`` or ``DELETE...LIMIT`` support in PostgreSQL; the generated query simulates that behavior and is based on `this Crunchy Data blog post <https://www.crunchydata.com/blog/simulating-update-or-delete-with-limit-in-postgres-ctes-to-the-rescue>`_.
..raw:: html
<script type="text/javascript">
@@ -739,6 +728,7 @@ Using ``offset`` to target a different subset of rows is also possible.
The PostgREST URL grammar limits the kinds of queries clients can perform. It prevents arbitrary, potentially poorly constructed and slow client queries. It's good for quality of service, but means database administrators must create custom views and functions to provide richer endpoints. The most common causes for custom endpoints are
*Table unions
*SET operators like `UNION, INTERSECT and EXCEPT <https://www.postgresql.org/docs/current/queries-union.html>`_.
* More complicated joins than those provided by :ref:`resource_embedding`.
* Geo-spatial queries that require an argument, like "points near (lat,lon)"
@@ -51,7 +51,7 @@ You can request table/columns with spaces in them by percent encoding the spaces
Reserved characters
~~~~~~~~~~~~~~~~~~~
If filters include PostgREST reserved characters(``,``, ``.``, ``:``, ``()``) you'll have to surround them in percent encoded double quotes ``%22`` for correct processing.
If filters include PostgREST reserved characters(``,``, ``.``, ``:``, ``*``, ``(``, ``)``) you'll have to surround them in percent encoded double quotes ``%22`` for correct processing.
Here ``Hebdon,John`` and ``Williams,Mary`` are values.
@@ -31,7 +31,7 @@ The authenticator role is used for connecting to the database and should be conf
.._user_impersonation:
User Impersonation
------------------
~~~~~~~~~~~~~~~~~~
The picture below shows how the server handles authentication. If auth succeeds, it switches into the user role specified by the request, otherwise it switches into the anonymous role (if it's set in :ref:`db-anon-role`).
@@ -43,12 +43,13 @@ This role switching mechanism is called **user impersonation**. In PostgreSQL it
The impersonated roles will have their settings applied. See :ref:`impersonated_settings`.
.._jwt_impersonation:
.._jwt_auth:
JWT-Based User Impersonation
----------------------------
JWT Authentication
------------------
We use `JSON Web Tokens <https://jwt.io/>`_ to authenticate API requests, this allows us to be stateless and not require database lookups for verification. As you'll recall a JWT contains a list of cryptographically signed claims. All claims are allowed but PostgREST cares specifically about a claim called role.
We use `JSON Web Tokens <https://datatracker.ietf.org/doc/html/rfc7519/>`_ to authenticate API requests, this allows us to be stateless and not require database lookups for verification.
As you'll recall a JWT contains a list of cryptographically signed claims. All claims are allowed but PostgREST cares specifically about a claim called role (configurable with :ref:`jwt_role_extract`).
..code::json
@@ -72,17 +73,10 @@ Note that the database administrator must allow the authenticator role to switch
If the client included no JWT (or one without a role claim) then PostgREST switches into the anonymous role. The database administrator must set the anonymous role permissions correctly to prevent anonymous users from seeing or changing things they shouldn't.
.._jwt_generation:
.._bearer_auth:
JWT Generation
~~~~~~~~~~~~~~
You can create a valid JWT either from inside your database (see :ref:`sql_user_management`) or via an external service (see :ref:`external_jwt`).
.._client_auth:
Client Auth
~~~~~~~~~~~
Bearer Authentication
~~~~~~~~~~~~~~~~~~~~~
To make an authenticated request the client must include an :code:`Authorization` HTTP header with the value :code:`Bearer <jwt>`. For instance:
@@ -93,24 +87,29 @@ To make an authenticated request the client must include an :code:`Authorization
The ``Bearer`` header value can be used with or without capitalization(``bearer``).
.._jwt_caching:
.._jwt_generation:
JWT Caching
-----------
JWT Generation
~~~~~~~~~~~~~~
PostgREST validates ``JWTs`` on every request. We can cache ``JWTs`` to avoid this performance overhead.
You can create a valid JWT either from inside your database (see :ref:`sql_user_management`) or via an external service (see :ref:`external_auth`).
To enable JWT caching, the config :code:`jwt-cache-max-lifetime` is to be set. It is the maximum number of seconds for which the cache stores the JWT validation results. The cache uses the :code:`exp` claim to set the cache entry lifetime. If the JWT does not have an :code:`exp` claim, it uses the config value. See :ref:`jwt-cache-max-lifetime` for more details.
.._jwt_signature:
..note::
JWT Signature Verification
--------------------------
You can use the :ref:`server-timing_header` to see the effect of JWT caching.
PostgREST supports both symmetric and asymmetric keys for verifying the signature of the token.
Symmetric Keys
~~~~~~~~~~~~~~
Each token is cryptographically signed with a secret key. In the case of symmetric cryptography the signer and verifier share the same secret passphrase, which can be configured with :ref:`jwt-secret`.
If it is set to a simple string value like “reallyreallyreallyreallyverysafe” then PostgREST interprets it as an HMAC-SHA256 passphrase.
In the case of symmetric cryptography the signer and verifier share the same secret passphrase, which can be configured with :ref:`jwt-secret`.
If it is set to a simple string then PostgREST interprets it as an HMAC-SHA256 passphrase.
..code-block::ini
jwt-secret="reallyreallyreallyreallyverysafe"
.._asym_keys:
@@ -156,13 +155,103 @@ You can specify the literal value as we saw earlier, or reference a filename to
jwt-secret="@rsa.jwk.pub"
JWT Claims Validation
~~~~~~~~~~~~~~~~~~~~~
``kid`` verification
^^^^^^^^^^^^^^^^^^^^
PostgREST honors the :code:`exp` claim for token expiration, rejecting expired tokens.
PostgREST has built-in verification of the `key ID parameter <https://www.rfc-editor.org/rfc/rfc7517#section-4.5>`_, useful when working with a JSON Web Key Set.
It goes as follows:
- If the JWT contains a ``kid`` parameter, then PostgREST will look for the JSON Web Key in the :ref:`jwt-secret`.
+ If no key has a matching ``kid`` (or if they don't have one defined), the token will be rejected with a :ref:`401 Unauthorized <pgrst301>` error.
+ If a key matches the ``kid`` value then it will validate the token against that key accordingly.
- If the JWT doesn't have a ``kid``, PostgREST will try each key in the :ref:`jwt-secret` one by one until it finds one that works.
.._jwt_claims_validation:
JWT Claims Validation
---------------------
Time-Based claims validation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The time-based JWT claims specified in `RFC 7519 <https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.4>`_ are validated:
-``exp`` Expiration Time
-``iat`` Issued At
-``nbf`` Not Before
We allow a 30-second clock skew when validating the above claims. In other words, we give an extra 30 seconds before the JWT is rejected if there is a slight discrepancy in the timestamps.
.._jwt_aud:
``aud`` validation
~~~~~~~~~~~~~~~~~~
PostgREST has built-in validation of the `JWT audience claim <https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.3>`_.
It works this way:
- If :ref:`jwt-aud` is not set (the default), PostgREST identifies with all audiences and allows the JWT for any ``aud`` claim.
- If :ref:`jwt-aud` is set to a specific audience, PostgREST will check if this audience is present in the ``aud`` claim:
+ If the ``aud`` value is a JSON string, it will match it to the :ref:`jwt-aud`.
+ If the ``aud`` value is a JSON array of strings, it will search every element for a match.
+ If the match fails or if the ``aud`` value is not a string or array of strings, then the token will be rejected with a :ref:`401 Unauthorized <pgrst303>` error.
+ If the ``aud`` key **is not present** or if its value is ``null`` or ``[]``, PostgREST will interpret this token as allowed for all audiences and will complete the request.
.._jwt_caching:
JWT Cache
---------
JWT signature validation (specially :ref:`asym_keys` such as RSA) is slow, we can cache ``JWT`` validation results to avoid this performance overhead.
The JWT cache is bounded and uses the `SIEVE algorithm <https://cachemon.github.io/SIEVE-website>`_ for efficient eviction. The cache is enabled by default and can be configured with :ref:`jwt-cache-max-entries`.
It's recommended to leave the JWT cache enabled as our load tests indicate ~20% more throughput for simple GET requests when using it. This while reducing CPU utilization in exchange for a bit more memory.
:ref:`jwt_cache_metrics` are available.
..note::
- If the ``jwt-secret`` is changed and the config is reloaded, the JWT cache will reset.
- JWTs that pass :ref:`jwt_signature` are cached, regardless if they pass :ref:`jwt_claims_validation`. We do this to ensure responses stays fast under common failure cases (such as expired JWTs).
- You can use the :ref:`server-timing_header` to see the performance benefit of JWT caching.
.._jwt_role_extract:
JWT Role Extraction
-------------------
A JSON Path (`RFC 9535 <https://www.rfc-editor.org/rfc/rfc9535.html>`_) can be specified for the location of the :code:`role` key in the JWT claims. It's configured by :ref:`jwt-role-claim-key`. This can be used to consume a JWT provided by a third party service like Auth0, Okta, Microsoft Entra or Keycloak.
You can quickly try out JSON Path by visiting https://serdejsonpath.live.
- If JSON Path query returns multiple values, the first one gets selected.
- Only when using the :ref:`file_config`, all ``$`` characters in the value must be escaped with an additional ``$`` char. For :ref:`env_variables_config` and :ref:`in_db_config`, only use a single ``$`` char.
- In our implementation, only the `search()` function from `JSON Path Functions <https://www.rfc-editor.org/rfc/rfc9535.html#name-function-extensions>`_ is available for filtering.
JWT Security
~~~~~~~~~~~~
------------
There are at least three types of common critiques against using JWT: 1) against the standard itself, 2) against using libraries with known security vulnerabilities, and 3) against using JWT for web sessions. We'll briefly explain each critique, how PostgREST deals with it, and give recommendations for appropriate user action.
@@ -198,3 +287,23 @@ doing custom logic based on the web user info.
`Unix file mode <https://en.wikipedia.org/wiki/File_system_permissions>`_ to be set for the socket specified in :ref:`admin-server-unix-socket`
Needs to be a valid octal between 600 and 777.
..code::bash
admin-server-unix-socket-mode ="660"
.._app.settings.*:
@@ -176,6 +231,37 @@ app.settings.*
Arbitrary settings that can be used to pass in secret keys directly as strings, or via OS environment variables. For instance: :code:`app.settings.jwt_secret = "$(MYAPP_JWT_SECRET)"` will take :code:`MYAPP_JWT_SECRET` from the environment and make it available to PostgreSQL functions as :code:`current_setting('app.settings.jwt_secret')`.
When using the environment variable `PGRST_APP_SETTINGS_*` form, the remainder of the variable is used as the new name. Case is not important : :code:`PGRST_APP_SETTINGS_MY_ENV_VARIABLE=some_value` can be accessed in postgres as :code:`current_setting('app.settings.my_env_variable')`.
The :code:`current_setting` function has `an optional boolean second <https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-SET>`_ argument to avoid it from raising an error if the value was not defined. Default values to :code:`app.settings` can then be given by combining this argument with :code:`coalesce` and :code:`nullif` : :code:`coalesce(nullif(current_setting('app.settings.my_custom_variable', true), ''), 'default value')`. The use of :code:`nullif` is necessary because if set in a transaction, the setting is sometimes not "rolled back" to :code:`null`. See also :ref:`this section <guc_req_headers_cookies_claims>` for more information on this behaviour.
.._client-error-verbosity:
client-error-verbosity
----------------------
=============== =======================
**Type** String
**Default** verbose
**Reloadable** Y
**Environment** PGRST_CLIENT_ERROR_VERBOSITY
**In-Database** pgrst.client_error_verbosity
=============== =======================
Specifies the verbosity of PostgREST errors. See :ref:`client_error_verbosity`.
..code::bash
# Return error "code", "message", "details" and "hint"
client-error-verbosity ="verbose"
# Return only "code" and "message"
client-error-verbosity ="minimal"
..note::
This setting only affects client side error messages. Server side logs are not affected by this setting.
.._db-aggregates-enabled:
db-aggregates-enabled
@@ -245,7 +331,7 @@ db-channel-enabled
When this is set to :code:`true`, the notification channel specified in :ref:`db-channel` is enabled.
You should set this to ``false`` when using PostgresSQL behind an external connection pooler such as PgBouncer working in transaction pooling mode. See :ref:`this section <external_connection_poolers>` for more information.
You should set this to ``false`` when using PostgreSQL behind an external connection pooler such as PgBouncer working in transaction pooling mode. See :ref:`this section <external_connection_poolers>` for more information.
.._db-config:
@@ -296,6 +382,10 @@ db-extra-search-path
Multiple schemas can be added in a comma-separated string, e.g. ``public, extensions``.
..important::
We default this config to ``public`` because it is the most common schema used to install PostgreSQL extensions such as :ref:`PostGIS <ww_postgis>`. You can disable this by setting this config to ``""``.
.._db-hoisted-tx-settings:
db-hoisted-tx-settings
@@ -386,7 +476,7 @@ db-pool-max-idletime
**In-Database**`n/a`
=============== =================================
*For backwards compatibility, this config parameter is also available as “db-pool-timeout”.*
*For backwards compatibility, this config parameter is also available as "db-pool-timeout".*
Time in seconds to close idle pool connections.
@@ -456,7 +546,7 @@ db-prepared-statements
When disabled, the generated queries will be parameterized (invulnerable to SQL injection) but they will not be prepared (cached in the database session). Not using prepared statements will noticeably decrease performance, so it's recommended to always have this setting enabled.
You should only set this to ``false`` when using PostgresSQL behind an external connection pooler such as PgBouncer working in transaction pooling mode. See :ref:`this section <external_connection_poolers>` for more information.
You should only set this to ``false`` when using PostgreSQL behind an external connection pooler such as PgBouncer working in transaction pooling mode. See :ref:`this section <external_connection_poolers>` for more information.
.._db-root-spec:
@@ -503,7 +593,7 @@ db-tx-end
**In-Database** pgrst.db_tx_end
=============== =================================
Specifies how to terminate the database transactions.
Specifies how to terminate the database transactions. See :ref:`prefer_tx`.
..code::bash
@@ -580,11 +670,7 @@ jwt-aud
**In-Database** pgrst.jwt_aud
=============== =================================
Specifies the `JWT audience claim <https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.3>`_. If this claim is present in the client provided JWT then you must set this to the same value as in the JWT, otherwise verifying the JWT will fail.
..warning::
Using this setting will only reject tokens with a different audience claim. Tokens **without** audience claim will still be accepted.
Specifies an audience for the JWT ``aud`` claim. See :ref:`jwt_aud`.
.._jwt-role-claim-key:
@@ -593,7 +679,7 @@ jwt-role-claim-key
=============== =================================
**Type** String
**Default** .role
**Default**$.role
**Reloadable** Y
**Environment** PGRST_JWT_ROLE_CLAIM_KEY
**In-Database** pgrst.jwt_role_claim_key
@@ -601,17 +687,11 @@ jwt-role-claim-key
*For backwards compatibility, this config parameter is also available without prefix as "role-claim-key".*
A JSPath DSL that specifies the location of the :code:`role` key in the JWT claims. This can be used to consume a JWT provided by a third party service like Auth0, Okta or Keycloak. Usage examples:
See :ref:`jwt_role_extract` on how to specify key paths and usage examples.
..code::bash
..warning::
# {"postgrest":{"roles": ["other", "author"]}}
# the DSL accepts characters that are alphanumerical or one of "_$@" as keys
Warning: 299 PostgRESTv16 "Embedded resource was referenced by relation name even though it has an alias. This is deprecated and will stop working in a future release. Update `target` to `alias` in query string filters, orders or limits."
[...]
Note that the response includes a deprecation message in the ``Warning`` header.
This will also show in the PostgREST logs:
..code::
28/May/2026:20:33:22 -0500: WARNING: Embedded resource was referenced by relation name even though it has an alias. This is deprecated and will stop working in a future release.
28/May/2026:20:33:22 -0500: Update filters, orders or limits that use `target` to `alias` in `GET /table?select=alias:target(*)&target.order=id`
This feature will be removed in a future release, so you should start using the ``alias`` in these cases.
To conserve system resources, PostgREST uses a dynamic connection pool. This enables the number of connections in the pool to increase and decrease depending on request traffic.
- If all the connections are being used, a new connection is added. The pool can grow until it reaches the :ref:`db-pool` size. Note that it’s pointless to set this higher than the ``max_connections`` setting in your database.
- If all the connections are being used, a new connection is added. The pool can grow until it reaches the :ref:`db-pool` size. Note that it's pointless to set this higher than the ``max_connections`` setting in your database.
- If a connection is unused for a period of time (:ref:`db-pool-max-idletime`), it will be released.
- For connecting to the database, the :ref:`authenticator <roles>` role is used. You can configure this using :ref:`db-uri`.
@@ -47,6 +47,8 @@ Under a busy system, the :ref:`db-pool-max-idletime` won't be reached and the co
To avoid this problem and save resources, a connection max lifetime (:ref:`db-pool-max-lifetime`) is enforced.
After the max lifetime is reached, connections from the pool will be released and new ones will be created. This doesn't affect running requests, only unused connections will be released.
.._pool_timeout:
Acquisition Timeout
-------------------
@@ -106,4 +108,4 @@ Also set :ref:`db-channel-enabled` to ``false`` since ``LISTEN`` is not compatib
..note::
It’s not recommended to use an external connection pooler. `Our benchmarks <https://github.com/PostgREST/postgrest/issues/2294#issuecomment-1139148672>`_ indicate it provides much lower performance than PostgREST built-in pool.
It's not recommended to use an external connection pooler. `Our benchmarks <https://github.com/PostgREST/postgrest/issues/2294#issuecomment-1139148672>`_ indicate it provides much lower performance than PostgREST built-in pool.
PostgREST error messages follow the PostgreSQL error structure. It includes ``MESSAGE``, ``DETAIL``, ``HINT``, ``ERRCODE`` and will add an HTTP status code to the response.
.._postgresql_errors:
Errors from PostgreSQL
======================
@@ -197,7 +199,7 @@ Related to the HTTP request elements.
| | | :ref:`switching schemas <multiple-schemas>` is not present |
| PGRST106 | | in the :ref:`db-schemas` configuration variable. |
If PostgREST can't parse the JSON objects ``message`` and ``detail``, it will throw a ``PGRST121`` error. See :ref:`Errors from PostgREST<pgrst1**>`.
.._proxy-status_header:
Proxy-Status Header
===================
For error cases, the standard `Proxy-Status <https://www.rfc-editor.org/rfc/rfc9209.html#name-the-proxy-status-http-field>`_ header is returned with the error code. The error code comes from either :ref:`PostgREST <pgrst_errors>`, :ref:`PostgreSQL <postgresql_errors>` or :ref:`Custom <custom_errors>` errors. This is useful when doing ``HEAD`` requests where the HTTP status is not descriptive enough.
For example, doing a request on a table with high count (say 30_000_000), we get:
..code-block::http
HEAD/tableHTTP/1.1
Prefer: count=exact
..code-block::http
HTTP/1.1500Internal Server Error
Proxy-Status: PostgREST; error=57014
The PostgreSQL error code ``57014`` (`ref <https://www.postgresql.org/docs/current/errcodes-appendix.html>`_) reveals that the error is due to a short ``statement_timeout`` value.
.._client_error_verbosity:
Client Error Verbosity
======================
For HTTP clients, the error verbosity can be set via :ref:`client-error-verbosity` config.
With ``verbose``, it returns ``code``, ``message``, ``details`` and ``hint``.
..code::bash
curl "localhost:3000/itemsxx"
..code-block::json
{
"code":"PGRST205",
"message":"Could not find the table 'public.itemsxx' in the schema cache",
"details":"Perhaps you meant the table 'public.items'",
"hint":null
}
With ``minimal``, just ``code`` and ``message`` is returned.
..code::bash
curl "localhost:3000/itemsxx"
..code-block::json
{
"code":"PGRST205",
"message":"Could not find the table 'public.itemsxx' in the schema cache"
The HTTP server is provided by `Warp <https://aosabook.org/en/posa/warp.html>`_.
Graceful shutdown
-----------------
PostgREST uses Warp's graceful shutdown, when a ``SIGTERM`` is received:
- It stops accepting new requests.
- Allows requests that are already in progress to finish.
- Closes idle ``Keep-Alive`` connections instead of waiting for them to expire.
- Responses sent during shutdown indicate that the connection should not be reused (e.g. for HTTP/1.x, it sends ``Connection: close``).
This allows PostgREST to shut down promptly without interrupting in-flight requests. Useful for zero-downtime upgrades and autoscaling/load-balancing under cloud environments (AWS ECS, Kubernetes).
PostgREST uses `LISTEN <https://www.postgresql.org/docs/current/sql-listen.html>`_ to reload its :ref:`Schema Cache <schema_reloading_notify>` and :ref:`Configuration <config_reloading_notify>` via `NOTIFY <https://www.postgresql.org/docs/current/sql-notify.html>`_.
This is useful in environments where you can’t send SIGUSR1 or SIGUSR2 Unix Signals.
This is useful in environments where you can't send SIGUSR1 or SIGUSR2 Unix Signals.
Like on cloud managed containers or on Windows systems.
..code::postgresql
@@ -46,7 +46,9 @@ This will cause the :ref:`connection_pool` to connect to the read replica host a
..note::
Under the hood, PostgREST forces `target_session_attrs=read-write <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS>`_ for the ``LISTEN`` session.
- Under the hood, PostgREST forces `target_session_attrs=read-write <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS>`_ for the ``LISTEN`` session.
So if you specify ``target_session_attrs=read-only`` as mentioned above, PostgREST will override it for the ``LISTEN``.
-``read-only`` is only available on libpq >= 14, if you use a lower version you will get an error like ``invalid target_session_attrs value: \"read-only\"``.
@@ -15,12 +15,14 @@ Observability allows measuring a system's current state based on the data it gen
Logs
====
PostgREST logs basic request information to ``stdout``, including the authenticated user if available, the requesting IP address and user agent, the URL requested, and HTTP response status.
PostgREST logs basic request information to ``stdout``, including the authenticated user if available, the requesting IP address and user agent, the URL requested, the HTTP response status and the response body size in bytes if available.
06/May/2024:08:16:11 -0500: Attempting to connect to the database...
06/May/2024:08:16:11 -0500: Successfully connected to PostgreSQL 14.10 (Ubuntu 14.10-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
06/May/2024:08:16:11 -0500: Listening on port 3000
06/May/2024:08:16:11 -0500: Listening for notifications on the "pgrst" channel
06/May/2024:08:16:11 -0500: Connection Pool initialized with a maximum size of 10 connections
06/May/2024:08:16:11 -0500: API server listening on port 3000
06/May/2024:08:16:11 -0500: Listening for database notifications on the "pgrst" channel
06/May/2024:08:16:11 -0500: Config reloaded
06/May/2024:08:16:11 -0500: Schema cache queried in 3.8 milliseconds
06/May/2024:08:16:11 -0500: Schema cache loaded 15 Relations, 8 Relationships, 8 Functions, 0 Domain Representations, 4 Media Type Handlers
06/May/2024:08:16:11 -0500: Schema cache loaded 15 Relations, 8 Relationships, 8 RPCs, 0 Domain Representations, 4 Media Type Handlers
06/May/2024:14:11:27 -0500: Received a config reload message on the "pgrst" channel
06/May/2024:14:11:27 -0500: Config reloaded
..note::
Logs are based on the ``log-level`` setting. See :ref:`log-level`.
.._sql_query_logs:
SQL Query Logs
--------------
To log the SQL queries executed for a request, set the :ref:`log-query` to ``true``. It will be logged based on the current :ref:`log-level` setting.
..code-block::bash
log-level ="warn"
log-query ="true"
The SQL queries will only be logged on ``400`` HTTP errors and up.
So, if the user requests a resource without sufficient privileges:
..code-block::bash
curl "localhost:3000/protected_table"
This will be logged by PostgREST:
..code::
17/Feb/2025:17:28:15 -0500: WITH pgrst_source AS ( SELECT "public"."protected_table".* FROM "public"."protected_table" ) SELECT null::bigint AS total_result_set, pg_catalog.count(_postgrest_t) AS page_total, coalesce(json_agg(_postgrest_t), '[]') AS body, nullif(current_setting('response.headers', true), '') AS response_headers, nullif(current_setting('response.status', true), '') AS response_status, '' AS response_inserted FROM ( SELECT * FROM pgrst_source ) _postgrest_t
Currently PostgREST doesn't log the SQL commands executed against the underlying database.
To find the SQL operations, you can watch the database logs. By default PostgreSQL does not keep these logs, so you'll need to make the configuration changes below.
Additionally, to find all the SQL operations, you can watch the database logs. By default PostgreSQL does not keep these logs, so you'll need to make the configuration changes below.
Find :code:`postgresql.conf` inside your PostgreSQL data directory (to find that, issue the command :code:`show data_directory;`). Either find the settings scattered throughout the file and change them to the following values, or append this block of code to the end of the configuration file.
@@ -90,12 +120,17 @@ Restart the database and watch the log file in real-time to understand how HTTP
Metrics
=======
The ``metrics`` endpoint on the :ref:`admin_server` endpoint provides metrics in `Prometheus text format <https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format>`_.
The ``metrics`` endpoint on the :ref:`admin_server` endpoint provides metrics in `Prometheus text format <https://prometheus.io/docs/instrumenting/exposition_formats/#prometheus-text-format>`_.
..code-block::bash
curl "http://localhost:3001/metrics"
..code-block::http
HTTP/1.1200OK
Content-Type: text/plain; charset=utf-8
# HELP pgrst_schema_cache_query_time_seconds The query time in seconds of the last schema cache load
# TYPE pgrst_schema_cache_query_time_seconds gauge
for runtime allocation, garbage collection, memory, and CPU/elapsed time.
These are useful for monitoring PostgREST process health and diagnosing memory
pressure or GC behavior.
To expose these metrics, enable GHC RTS statistics when starting PostgREST:
..code-block::bash
postgrest +RTS -T -RTS
When enabled, the admin ``/metrics`` endpoint includes samples such as:
..code-block::text
# HELP ghc_gcs_total Total number of GCs
# TYPE ghc_gcs_total counter
ghc_gcs_total 1
# HELP ghc_allocated_bytes_total Total bytes allocated
# TYPE ghc_allocated_bytes_total counter
ghc_allocated_bytes_total 12345678
Other available GHC runtime metrics include:
-``ghc_gcs_total``
-``ghc_major_gcs_total``
-``ghc_allocated_bytes_total``
-``ghc_max_live_bytes``
-``ghc_max_mem_in_use_bytes``
-``ghc_mutator_cpu_seconds_total``
-``ghc_gc_cpu_seconds_total``
-``ghc_elapsed_seconds_total``
Traces
======
@@ -204,12 +312,17 @@ You can enable tracing HTTP requests by setting :ref:`server-trace-header`. Spec
HTTP/1.1 200 OK
X-Request-Id: 123
Proxy-Status Header
-------------------
See :ref:`proxy-status_header`.
.._server-timing_header:
Server-Timing Header
--------------------
You can enable the `Server-Timing <https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing>`_ header by setting :ref:`server-timing-enabled` on.
You can enable the `Server-Timing <https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Server-Timing>`_ header by setting :ref:`server-timing-enabled` on.
This header communicates metrics of the different phases in the request-response cycle.
..code-block::bash
@@ -223,7 +336,7 @@ This header communicates metrics of the different phases in the request-response
- All the durations (``dur``) are in milliseconds.
- The ``jwt`` stage is when :ref:`jwt_impersonation` is done. This duration can be lowered with :ref:`jwt_caching`.
- The ``jwt`` stage is when :ref:`jwt_auth` is done. This duration can be lowered with :ref:`jwt_caching`.
- On the ``parse`` stage, the :ref:`url_grammar` is parsed.
- On the ``plan`` stage, the :ref:`schema_cache` is used to generate the :ref:`main_query` of the transaction.
- The ``transaction`` stage corresponds to the database transaction. See :ref:`transactions`.
@@ -233,6 +346,27 @@ This header communicates metrics of the different phases in the request-response
We're working on lowering the duration of the ``parse`` and ``plan`` stages on https://github.com/PostgREST/postgrest/issues/2816.
.._content-length_header:
Content-Length Header
---------------------
You can verify the response body size in bytes in the `Content-Length header <https://httpwg.org/specs/rfc9110.html#field.content-length>`_.
..code-block::bash
curl -i 'localhost:3000/users'
..code-block::http
HTTP/1.1200OK
Content-Length: 104
Note that this header won't be returned on ``HEAD`` requests for optimization purposes (see :ref:`head_req`).
This is in line with `RFC 9110 <https://httpwg.org/specs/rfc9110.html#field.content-length>`_.
The body size is also present in the :ref:`PostgREST logs <pgrst_logging>`.
.._explain_plan:
Execution plan
@@ -296,6 +430,8 @@ By default the plan is assumed to generate the JSON representation of a resource
The other available parameters are ``analyze``, ``verbose``, ``settings``, ``buffers`` and ``wal``, which correspond to the `EXPLAIN command options <https://www.postgresql.org/docs/current/sql-explain.html>`_. To use the ``analyze`` and ``wal`` parameters for example, you would add them like ``Accept: application/vnd.pgrst.plan; options=analyze|wal``.
For a workflow that takes the ``Query Identifier`` from a verbose PostgREST plan and uses it to inspect the same query in ``pg_stat_statements``, see :ref:`debugging_performance_pg_stat_statements`.
Note that akin to the EXPLAIN command, the changes will be committed when using the ``analyze`` option. To avoid this, you can use the :ref:`db-tx-end` and the ``Prefer: tx=rollback`` header.
PostgREST requires metadata from the database schema to provide a REST API that abstracts SQL details. One example of this is the interface for :ref:`resource_embedding`.
PostgREST requires metadata from the database to provide a REST API that abstracts SQL details. One example of this is the interface for :ref:`resource_embedding`.
Getting this metadata requires expensive queries. To avoid repeating this work, PostgREST uses a schema cache.
..note::
- Schema cache queries have been optimized over time to stay fast, even on complex databases. You can see a summary of their execution time in :ref:`pgrst_logging` and :ref:`metrics`.
- If the schema cache queries are slow, the most likely cause is *system catalog bloat*, see `issue#3212 <https://github.com/PostgREST/postgrest/issues/3212>`_ for more details.
- You can turn the :ref:`log-level` to ``debug`` to see the time of each schema cache query.
.._schema_reloading:
Schema Cache Reloading
@@ -18,7 +24,7 @@ You can do this with UNIX signals or with PostgreSQL notifications. It's also po
..note::
-Requests will wait until the schema cache reload is done. This to prevent client errors due to an stale schema cache.
-If the schema cache fails to reload (e.g. due to a ``statement_timeout`` or :ref:`pool timeout <pool_timeout>`), PostgREST will continue serving requests in a "best effort" basis.
- If you are using the :ref:`in_db_config`, a schema cache reload will :ref:`reload the configuration<config_reloading>` as well.
.._schema_reloading_signals:
@@ -53,6 +59,19 @@ To reload the schema cache from within the database, you can use the ``NOTIFY``
NOTIFYpgrst,'reload schema'
Debouncing
~~~~~~~~~~
PostgREST does not reload the schema cache for each notification when several ``NOTIFY pgrst`` events are generated quickly after one another.
There are two cases to consider: when notifications are sent within a single transaction and when they are sent across multiple transactions.
In the first case, PostgreSQL deduplicates identical ``NOTIFY`` events within the same transaction. This means that even if multiple ``NOTIFY pgrst`` statements are executed before a ``COMMIT``, only a single notification is delivered to PostgREST.
In the second case, when notifications are sent from separate transactions in a short time span, PostgREST applies a debouncing mechanism to avoid excessive schema cache reloads.
Instead of reloading the schema cache for each notification, events are grouped within a small time window of 100 milliseconds. The reload function is executed once immediately when the first notification is received and once more after the burst of events settles, resulting in at most two executions within that time window.
@@ -221,7 +221,7 @@ Notice that the ``response.headers`` should be set to an *array* of single-key o
..note::
PostgREST provided headers such as ``Content-Type``, ``Location``, etc. can be overriden this way. Note that irrespective of overridden ``Content-Type`` response header, the content will still be converted to JSON, unless you use :ref:`custom_media`.
PostgREST provided headers such as ``Content-Type``, ``Location``, etc. can be overridden this way. Note that irrespective of overridden ``Content-Type`` response header, the content will still be converted to JSON, unless you use :ref:`custom_media`.
.._guc_resp_status:
@@ -284,11 +284,10 @@ However, starting from PostgreSQL 15, you can grant privileges for these setting
GRANTSETONPARAMETER<setting>TO<authenticator>;
Function Settings
~~~~~~~~~~~~~~~~~
Hoisted Function Settings
~~~~~~~~~~~~~~~~~~~~~~~~~
In addition to :ref:`impersonated_settings`, PostgREST will also apply function settings as transaction-scoped settings. This allows functions settings to override
the impersonated and connection role settings.
PostgREST can "hoist" function settings to transaction-scoped settings. This allows functions settings to override the impersonated and connection role settings.
..code-block::postgres
@@ -303,7 +302,7 @@ When calling the above function (see :ref:`functions`), the statement timeout wi
..note::
Only the transactions that are hoisted by config:ref:`db-hoisted-tx-settings` will be applied.
Only the settings in:ref:`db-hoisted-tx-settings` will be hoisted.
@@ -17,17 +17,12 @@ To make an API we'll simply be building a database. All the endpoints and permis
By the end of this tutorial you'll have a working database, PostgREST server, and a simple single-user todo list API.
Step 1. Relax, we'll help
-------------------------
As you begin the tutorial, pop open the project `chat room <https://gitter.im/begriffs/postgrest>`_ in another tab. There are a nice group of people active in the project and we'll help you out if you get stuck.
Step 2. Install PostgreSQL
Step 1. Install PostgreSQL
--------------------------
If you're already familiar with using PostgreSQL and have it installed on your system you can use the existing installation (see :ref:`pg-dependency` for minimum requirements). For this tutorial we'll describe how to use the database in Docker because database configuration is otherwise too complicated for a simple tutorial.
If Docker is not installed, you can get it `here <https://www.docker.com/get-started>`_. Next, let's pull and start the database image:
If Docker is not installed, you can get it `here <https://www.docker.com/get-started>`_. Make sure that Docker service is `started <https://docs.docker.com/engine/daemon/start/#start-the-daemon-using-operating-system-utilities>`_. Next, let's pull and start the database image:
..code-block::bash
@@ -48,7 +43,7 @@ This will run the Docker instance as a daemon and expose port 5432 to the host s
In this case, you will need to change the **first** of the two 5432 to something else, for example to :code:`5433:5432`. Remember to also adjust the port in your config file in Step 5!
Step 3. Install PostgREST
Step 2. Install PostgREST
-------------------------
Using a Package Manager
@@ -122,7 +117,7 @@ If everything is working correctly it will print out its version and the availab
</details>
</p>
Step 4. Create Database for API
Step 3. Create Database for API
-------------------------------
Connect to the SQL console (psql) inside the container. To do so, run this from your command line:
@@ -185,7 +180,7 @@ Now quit out of psql; it's time to start the API!
\q
Step 5. Run PostgREST
Step 4. Run PostgREST
---------------------
PostgREST can use a configuration file to tell it how to connect to the database. Create a file :code:`tutorial.conf` with this inside:
@@ -218,12 +213,8 @@ You should see something similar to:
..code-block::text
Starting PostgREST 12.0.2...
Attempting to connect to the database...
Connection successful
Listening on port 3000
Config reloaded
Listening for notifications on the pgrst channel
Schema cache loaded
Successfully connected to PostgreSQL 14.10 (Ubuntu 14.10-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
API server listening on port 3000
It's now ready to serve web requests. There are many nice graphical API exploration tools you can use, but for this tutorial we'll use :code:`curl` because it's likely to be installed on your system already. Open a new terminal (leaving the one open that PostgREST is running inside). Try doing an HTTP request for the todos.
@@ -52,17 +52,31 @@ Check that the :code:`tutorial.conf` (created in the previous tutorial) has the
If the PostgREST server is still running from the previous tutorial, restart it to load the updated configuration file.
.._tut1_step3:
Step 3. Sign a Token
--------------------
Ordinarily your own code in the database or in another server will create and sign authentication tokens, but for this tutorial we will make one "by hand." Go to `jwt.io <https://jwt.io/#debugger-io>`_ and fill in the fields like this:
Ordinarily your own code in the database or in another server will create and sign authentication tokens, but for this tutorial we will make one "by hand" using ``bash`` and ``openssl``.
..figure::../_static/tuts/tut1-jwt-io.png
:alt:jwt.io interface
..code::bash
How to create a token at https://jwt.io
#!/bin/bash
set -e
**Remember to fill in the secret you generated rather than the word "secret".** After you have filled in the secret and payload, the encoded data on the left will update. Copy the encoded token.
**Remember to fill in the secret you generated rather than keeping the "test_secret_that_is_at_least_32_characters_long".** After you have filled in the secret and payload, the encoded data on the left will update. Copy the encoded token.
..note::
@@ -145,14 +159,22 @@ To observe expiration in action, we'll add an :code:`exp` claim of five minutes
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
