Compare commits

..
486 Commits
Author SHA1 Message Date
Taimoor Zaeem 02d83d1c01 test(io): move remaining tests in test_io.py to their modules
We had just 3 tests remaining in test_io.py. This commit moves them to
their modules. So we have:

* test_graceful_shutdown.py

* test_zero_downtime.py

* test_pg_internal.py

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-08-13 03:42:29 +05:00
Taimoor Zaeem e89e0bc255 test(io): move config related behavior tests to test_config.py
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-08-13 03:42:29 +05:00
Taimoor Zaeem 9e20e5df90 test(io): move logs and observations tests to test_log.py
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-08-13 03:42:29 +05:00
Taimoor Zaeem 2122dcef97 test(io): move connection related tests to test_connection.py
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-08-13 03:42:29 +05:00
Taimoor Zaeem d84d00be8e test(io): move reloading related tests to test_reloading.py
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-08-13 03:42:29 +05:00
Taimoor ZaeemandSteve Chavez 3a034fcc2a chore: add changelog entry for v16.1
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-08-12 10:14:34 -05:00
Michał KłeczekandSteve Chavez a2a592a617 fix: JWT validation uses wrong current time due to a bug in auto-update
Upgrade auto-update to 0.2.7 which contains a fix to a bug causing some threads not seeing updates to the cached values.
2026-08-10 17:20:56 -05:00
Taimoor ZaeemandWolfgang Walther 89fe63fce0 nix(release): automate updating source file links in the docs
Closes #5133.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-08-10 19:14:04 +00:00
steve-chavez f21949a517 chore(changelog): missed entry for ghcr 2026-08-07 19:15:50 -05:00
steve-chavez 6ead34e70b chore(changelog): add integrations section
Also clarify some parts.
2026-08-07 16:17:56 -05:00
steve-chavezandWolfgang Walther e3206262d2 chore(changelog): make it in postgres style
This is to make it easier for users to understand the changes,
much like PostgreSQL does for its changelog: https://www.postgresql.org/docs/19/release-19.html#RELEASE-19-SERVER
(see how they classify "Optimizer", "General Performance", etc).

We classify features, fixes and deprecation in subsections.
Breaking changes are put inside a "Migration to v16" guide.
2026-08-07 19:59:15 +00:00
Taimoor Zaeem a3f36d6184 test: fix failing test of legacy target names warning
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-08-07 22:47:09 +05:00
renovate[bot]andWolfgang Walther 6cc1698c05 chore(deps): update vmactions/freebsd-vm action to v1.5.3 2026-08-07 07:43:06 +00:00
Wolfgang Walther 77d6055683 chore: bump version to 17 2026-08-07 09:33:46 +02:00
Wolfgang Walther 05d1e0f83d chore: bump version to 16.0 2026-08-07 09:33:45 +02:00
Wolfgang Walther 6fdc230637 nix: remove outdated code / comment in release script 2026-08-07 09:33:29 +02:00
Wolfgang Walther a841dc9c39 chore: ignore stackoverflow in linkcheck
Seems like stackoverflow started blocking GHA runners as well.
2026-08-07 09:14:33 +02:00
steve-chavez 18371ec372 chore(changelog): clarified JSON Path feature 2026-08-06 18:36:16 -05:00
steve-chavez 6951e1301d docs: keep working when scache reload fails
Docs for 54430c3a2c
2026-08-06 17:54:23 -05:00
steve-chavez 75891975f5 docs: fix wrong ref in admin 2026-08-06 17:24:56 -05:00
steve-chavez 142b54158e docs: graceful shutdown docs
Considered the explanations and use cases on:

- https://github.com/yesodweb/wai/issues/853
- https://github.com/PostgREST/postgrest/pull/4580
2026-08-06 17:24:53 -05:00
steve-chavez 876dfad9d9 chore(changelog): move fixed entry to added
Graceful shutdown is really a feature
2026-08-06 17:19:45 -05:00
steve-chavez 2ad6d239a0 docs: add new http server page 2026-08-06 17:19:45 -05:00
Taimoor Zaeem 426e15bbb4 chore(cabal): update haskell language edition to GHC2021
This includes a good set of default language extensions that are
often used by us. It frees us of explicitly importing common extensions.

Ref: https://ghc.gitlab.haskell.org/ghc/doc/users_guide/exts/control.html#extension-GHC2021

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-08-05 12:38:29 +05:00
steve-chavez 7e8212870d docs: release cycle 2026-08-04 19:49:48 -05:00
renovate[bot]andWolfgang Walther 6a9c99d54e chore(deps): update haskell-actions/setup action to v2.12.0 2026-08-04 21:05:25 +00:00
steve-chavez e44178445d docs: timezone offsets 2026-08-04 13:41:40 -05:00
steve-chavez 59a644bb99 test: prefer timezone offsets 2026-08-04 13:41:40 -05:00
steve-chavez 932c4f6328 perf: no caching for prefer timezone
`Prefer: timezone` no longer requires the schema cache.
Previously this required caching `pg_timezone_names` which was slow in some systems.

Closes https://github.com/PostgREST/postgrest/issues/5100 and
https://github.com/PostgREST/postgrest/issues/4751.
2026-08-04 13:41:40 -05:00
steve-chavez a41396c425 Revert "add: config db-timezone-enabled for optional querying of timezones"
This reverts commit d6c338d588.
2026-08-04 13:41:40 -05:00
steve-chavez ca4a6d9e99 Revert "add: configs as json GUC for db-root-spec"
This reverts commit f34ca15e84.
2026-08-04 00:50:18 -05:00
steve-chavez f34ca15e84 add: configs as json GUC for db-root-spec
These are needed for a complete OpenAPI spec
2026-08-03 23:41:03 -05:00
Wolfgang Walther f84c44dafa nix: add postgrest-with-oriole-18 2026-08-02 14:51:50 +00:00
Wolfgang Walther 549ab36c30 docs: add NixOS integration section
Just a very basic example of how to use the NixOS module that Nixpkgs
provides.
2026-08-02 14:51:50 +00:00
Wolfgang Walther 42754cb6c9 chore(deps): update nixpkgs pin
Nixpkgs has removed support for x86_64-darwin, so removing that from
CI's flake check.

A x86_64-darwin binary is still built via Stack.
2026-08-02 14:51:50 +00:00
Wolfgang Walther 57e8d20811 nix: show stderr output for postgrest-style
Hiding this will also hide errors when they happen...

In my case, my nix-shell's TMPDIR was too long, so black was failing
silently because it couldn't create a unix socket for its forkserver.
2026-08-02 14:51:50 +00:00
Taimoor ZaeemandWolfgang Walther 8ede8fb0dc test(io): move settings related tests to test_settings.py
Towards #4946.

Moves app and pg settings related tests from test_io.py to
test_settings.py.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-08-02 11:53:32 +00:00
Joel JacobsonandWolfgang Walther 131df9b578 perf: use pg_basetype on PG 17+ for schema cache 2026-08-02 11:47:26 +00:00
Taimoor Zaeem 6a7ad33524 fix: db-channel-enabled not reloadable on config reload
Fixes #4894.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-31 21:39:27 +05:00
Taimoor Zaeem b174295bf2 refactor(chore): rename conf to oldConf in readInDbConfig function
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-31 21:39:27 +05:00
renovate[bot]andWolfgang Walther 6304265cd9 chore(deps): update docker/login-action action to v4.6.0 2026-07-30 20:41:39 +00:00
Taimoor Zaeem cb9b00d374 docs: update link of listener in architecture page
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-30 11:23:56 +05:00
Taimoor Zaeem 59081195f0 chore: add changelog entry for v14.16
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-28 23:31:31 +05:00
Taimoor Zaeem 022f0faa38 add: make config log-level reloadable
Closes #5113.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-28 11:33:33 +05:00
Taimoor Zaeem f6d34fd4fc refactor: move reloading related logic to Reload.hs
This commit moves, schema cache reload, config reload and listener
reload logic to `Reload.hs` module.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-25 14:39:48 +05:00
Taimoor Zaeem e32e2b07b5 refactor: move connection pool functions to Pool.hs
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-25 14:39:48 +05:00
Taimoor Zaeem a108d66968 refactor: move AppState type and update functions to AppState/Types.hs
`AppState.hs` is imported by many modules. Some of those modules contain
important functions that need to be imported back to `AppState.hs`, causing
circular dependency problem.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-25 14:39:48 +05:00
renovate[bot]andWolfgang Walther e820cc7370 chore(deps): update docker/login-action action to v4.5.1 2026-07-24 19:26:16 +00:00
renovate[bot]andWolfgang Walther 74c8a3580d chore(deps): update vmactions/freebsd-vm action to v1.5.2 2026-07-24 18:51:13 +00:00
steve-chavez 03c67f3410 test: move psql_as_superuser as util and reuse it 2026-07-24 13:44:58 -05:00
renovate[bot]andWolfgang Walther 1d195bf865 chore(deps): update docker/login-action action to v4.5.0 2026-07-23 20:11:53 +00:00
renovate[bot]andWolfgang Walther 473734f717 chore(deps): update actions/checkout action to v7.0.1 2026-07-23 20:11:28 +00:00
Taimoor Zaeem 178b1d3e38 fix: kill postgrest main thread on admin server crash
Fixes #5096.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-21 20:53:40 +05:00
Taimoor Zaeem 080abcb695 chore: increase columns length in .stylish-haskell.yaml
This gives the import statements more space, essentially, it collectively
removes 90+ unneeded lines in the imports statements.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-19 20:56:12 +05:00
Taimoor ZaeemandWolfgang Walther 8faf9c0519 test: resolve pytest warning when running io tests
When running `postgrest-test-io`, pytest raises a `InsecureKeyLengthWarning`
for a test. To clear that warning, this commit increases the key length to
more than 32 characters.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-17 17:42:51 +00:00
Taimoor Zaeem f81dd29cd1 chore: build haskell source modules in parallel
`postgrest-build` now builds twice as fast as compared to previous
build time. This can be tested by running `rm -rf dist-newstyle/` and
then run `postgrest-build`.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-16 13:34:59 +05:00
Michał KłeczekandSteve Chavez c297d051dc add: use SO_REUSEPORT on platform supporting it 2026-07-15 11:51:21 -05:00
Wolfgang Walther de19b04fe4 amend(aa7d442d): fix docs link to new Main.hs location 2026-07-15 08:55:19 +02:00
Taimoor Zaeem 51e9fd6efd fix: schema cache dump missing RPC transaction isolation level
Fixes #5079.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-14 23:13:21 +05:00
Taimoor Zaeem 63a2267f53 chore: add changelog entry for v14.15
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-14 21:04:15 +05:00
Wolfgang Walther aa7d442d32 chore: move executable code to src/
src/ now contains all source code - in subdirectories, according to the
.cabal component they belong to. This will allow us to put vendored
libraries in the same place - and later split our own code into multiple
components/libraries as well.
2026-07-14 06:58:15 +00:00
Wolfgang Walther 44e15e4e9b chore: run doctests in parallel
Runs the doctests much faster, which is potentially useful in
combination with postgrest-watch for local development.

This implies that doctests run on compiled code, not in a GHCi session,
which has some implications:
- Only exported functions can be tested.
- Imports need to be made explicit in doctests themselves.

On the flipside, this would allow us to potentially include doctest
results in code coverage, I believe.

This change is a requirement to vendor hasql, which otherwise breaks the
existing doctests: hasql contains a .hsc file, which *needs* to be
compiled - not interpreted - to make the tests work.
2026-07-14 06:55:32 +00:00
Wolfgang Walther aa6c19ebda test: remove useless doctest in Network.hs
This essentially just tests upstream's show instance.
2026-07-14 06:55:32 +00:00
Taimoor ZaeemandWolfgang Walther 8776ece7d5 test(io): move admin server tests to test_admin.py
Towards #4946.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-11 19:20:59 +00:00
steve-chavez 205135853d amend: update cabal index so cabal build succeeds 2026-07-11 13:24:09 -05:00
steve-chavez a71ab36096 amend: use warp 3.4.14 for 3522917
Use a release instead of a git commit
2026-07-11 13:12:03 -05:00
steve-chavezandLaurence Isla 490d1dc5d3 add: config to emit warning for legacy target names
Adds the `url_use_legacy_target_names` config.

Enabled (default):
* It allows using the resource name in filters,
  orders or limits when it has an alias, e.g.
  `table?select=alias:target(*)&target.id=eq.1`
* Logs a WARNING with a hint to use the alias
* Returns a Warning header in the response

Disabled:
* It returns an error, only the alias is allowed
* No warnings returned

This feature is deprecated
2026-07-11 02:15:07 +00:00
steve-chavez 2fa8de4e52 chore: add changelog entry for 5077 2026-07-10 19:54:11 -05:00
steve-chavez 3522917021 fix: admin server dies silently by upgrading warp
Uses warp main branch https://github.com/yesodweb/wai/commit/ad041216b643f69a2a9c87cbf4c2988aa4633dd5
since there's no release yet.
2026-07-10 17:56:15 -05:00
Michał KłeczekandSteve Chavez 4fc47754c2 refactor: do not open a TCP connection in liveness check
This change introduces a new way to perform liveness check - instead of trying to open a TCP connection to main server, we check if main socket is listening and if main server accept loop thread is alive.

Opening a TCP connection in liveness check was problematic because:
* it used available file descriptors which might have been a problem under load
* made liveness check unreliable when multiple PostgREST instances are available on the same port (eg. using SO_REUSEPORT)
2026-07-10 12:45:59 -05:00
Taimoor Zaeem 0bda2bcdff refactor: remove redundant destroyPool function
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-08 21:09:20 +05:00
Taimoor Zaeem c03ac09dbb test: move prepared statements config related tests to spec tests
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-07 23:32:40 +05:00
Taimoor Zaeem 44edf962d8 test: move db-pre-config with pg reserved word func to spec tests
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-07 23:32:40 +05:00
Taimoor Zaeem 6670b1a4b6 test(io): add test to db-pre-config with non-existent function
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-07 23:32:40 +05:00
Taimoor Zaeem c73c676849 test: move error verbosity test from io tests to spec tests
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-07 23:32:40 +05:00
steve-chavez 6191243c64 ci: add git to freebsd
Otherwise when using a git commit on stack the build fails because it
lacks the git dependency.
2026-07-06 21:50:26 -05:00
Michał KłeczekandSteve Chavez c5c9dc33c9 refactor: make sure proper sockets cleanup is performed in App.run
Right now code in App.run does not properly use bracket/finally to close sockets and clean-up mainSocketRef. This is not a big problem at the moment because the application is going to exit enyway but introducting proper resource handling will make future refactorings safer.
2026-07-06 10:59:05 -05:00
Michał KłeczekandSteve Chavez 56df1dc532 refactor: encapsulate application termination
This change makes AppState and Listener modules independent from the way how application is terminated. It removes stateMainThreadId from AppState and introduces stateKillApp. It also removes exported  function AppState.getMainThreadId replacing it with more general killApp.
2026-07-06 10:29:17 -05:00
Michał KłeczekandSteve Chavez 06bda07db1 refactor: encapsulate how main application liveness check is done
This change makes Admin module independent from how liveness check is performed. It moves reachMainApp function to App module and makes Admin.runAdmin, instead of action to get the main socket, take an IO action checking liveness as a parameter.

Thanks to this, any change in how liveness check is performed and, more importantly, what data it uses, does not require changing multiple modules (Admin and App).
2026-07-05 15:22:37 -05:00
Wolfgang Walther 0bd2821937 chore(deps): update hackage freeze, stackage snapshot and nixpkgs pin
This brings in PostgreSQL 19 beta 1 to start testing against.
2026-07-05 12:50:14 +00:00
renovate[bot]andWolfgang Walther 936d9df13d chore(deps): update docker/login-action action to v4.4.0 2026-07-05 12:49:52 +00:00
Taimoor Zaeem 75b315d077 nix: provide default file name when generating module graph
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-07-05 00:44:05 +05:00
renovate[bot]andWolfgang Walther 76c1ec5b06 chore(deps): update docker/setup-buildx-action action to v4.2.0 2026-07-02 20:08:24 +00:00
renovate[bot]andWolfgang Walther a836fb5e37 chore(deps): update docker/login-action action to v4.3.0 2026-07-02 08:56:54 +00:00
ncikandWolfgang Walther 69d21a82c9 docs: fix example jwt expiration epoch
Issue: the example epoch date is wrapped in parentheses. This causes the server to return PGRST303: "The JWT 'exp' claim must be a number"

Fix: remove parentheses
2026-07-01 19:22:59 +00:00
steve-chavez 412c6a1d26 chore: mentions docs on CONTRIBUTING 2026-06-30 10:06:45 -05:00
renovate[bot]andWolfgang Walther 5a987c06fe chore(deps): update korthout/backport-action action to v4.6 2026-06-30 06:24:53 +00:00
Taimoor ZaeemandSteve Chavez af17a75ae1 chore: add changelog entry for v14.14
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-29 16:42:14 -05:00
renovate[bot]andWolfgang Walther 9f3dc46502 chore(deps): update vmactions/freebsd-vm action to v1.5.0 2026-06-29 17:46:01 +00:00
Taimoor ZaeemandSteve Chavez b967a5033a fix(admin): log admin server exceptions and close admin socket
Admin server was crashing silently and requests hang indefinitely.
With this fix, we are now logging the admin server exceptions and also
close the socket afterwards so admin request don't hang.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-29 12:43:54 -05:00
Taimoor Zaeem 18646a6c03 chore: enable GHC NumericUnderscores language extension
The `_` character between numeric literals improve their readablity.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-28 01:01:12 +05:00
Wolfgang Walther 84e2a0c829 nix(loadtest): generate targets at nix build time
Instead of generating them at loadtest runtime, we generate these
dynamic targets once via Nix. Because we're using libfaketime, we're not
bound to doing it at the same time as running the tests.

This allows us to run each loadtest right after each other, with minimal
other stuff going on inbetween. It's also a tiny bit more reproducible,
because the loadtests against both branches run with exactly the same
set of targets, where they did not before.
2026-06-27 18:01:50 +00:00
Wolfgang Walther f1649bf6ee nix(loadtest): run all target branches almost at the same time
Instead of building, running, building, running, ... we now build all
executables once ahead of time and then run all loadtests right after
each other. This can sometimes reduce noise when load on the GHA runner
varies over time.

Since this requires us to move building into the loadtest-against
script, it also allows to go back to have the regular postgrest-loadtest
command default to building with cabal for faster local iteration.
2026-06-27 18:01:50 +00:00
Wolfgang Walther bc5d7a0e92 nix: remove postgrest-git-hooks
I'm not sure whether anyone is actually using them, but they never
worked really well, I believe. We should add something based on an
existing pre-commit hook manager, not that brittle custom thing we have.

Allows me to remove postgrest-with-git in the next commit instead of
duplicating it.
2026-06-27 18:01:50 +00:00
Wolfgang Walther d5875c1554 nix(loadtest): refactor loadtest-against
No need to duplicate this code, all of it is right there.
2026-06-27 18:01:50 +00:00
Wolfgang Walther 576805b7b0 nix: remove hint about installing tools locally from README
This is a heavily discouraged way of using these tools - they will
become outdated almost immediately and would need to be reinstalled on
every change to main.
2026-06-27 18:01:30 +00:00
Wolfgang Walther 2b463ab1c6 nix: remove docker tools note from README
This is outdated, we don't have that docker argument anymore - docker
tools are exposed by default, when on Linux.
2026-06-27 18:01:30 +00:00
Wolfgang Walther 6f899ee493 nix: reduce list of commands in README
This list is chronically outdated and hard to maintain with the two
column layout. Simplify and reduce, not important to keep up-to-date to
bring the point across.
2026-06-27 18:01:30 +00:00
Wolfgang Walther 556a5dad47 chore: remove eol-whitespace in postgrest.cabal 2026-06-27 19:20:24 +02:00
renovate[bot]andWolfgang Walther cbaf5ac8fc chore(deps): update vmactions/freebsd-vm action to v1.4.9 2026-06-27 09:41:20 +00:00
renovate[bot]andWolfgang Walther 76261d5bfc chore(deps): update actions/cache action to v6.1.0 2026-06-26 19:48:22 +00:00
Wolfgang Walther dcc99040d6 nix(loadtest): run more requests in jwt-cache test
We don't need to restrict us to run every request once. Instead we can
run as many as possible within 60s by removing the `-lazy` flag.

This also allows us to generate fewer targets, because vegeta will do
the repetition for us.
2026-06-26 19:33:51 +00:00
Wolfgang Walther 07a9752866 nix(loadtest): remove jwt-cache-worst
This test is not really functional (anymore?). It depends on some tokens
expiring - but they don't, really. This is because:
1) Before merging the RSA and HS variants earlier in this patch series,
they ran roughly 15s and 34s respectively in CI.
2) Since PostgREST takes a clock skew of up to 30 seconds into account,
there would be *at most* 4s of expiring tokens in a test run - and
*only* in the HS case, not for RSA.
2026-06-26 19:33:51 +00:00
Wolfgang Walther 7cafa67f0a nix(loadtest): use faketime in jwt-cache test
While we didn't have much problem in this test, it's surely not a
problem either to make the test more reproducible with a fixed base
timestamp. This gives us a tiny bit less of a diff on the next change.
2026-06-26 19:33:51 +00:00
Wolfgang Walther 3909e85f67 nix(loadtest): remove jwt test
This test is not very interesting, really: It's hard to imagine to be
able to produce a regression that would slow down the no-cache case, but
would *not* show when a cache is used.
2026-06-26 19:33:51 +00:00
Wolfgang Walther e83a25a698 nix(loadtest): remove genKeyMaterials
Merge the same code into genTargets, which removes the need to save the
private key to disk and read it again.
2026-06-26 19:33:51 +00:00
Wolfgang Walther 32706284f0 nix(loadtest): refactor key selection
Just a bit simpler.
2026-06-26 19:33:51 +00:00
Wolfgang Walther a079d23de1 nix(loadtest): remove useless try/except blocks
Rethrowing the same error, but without stacktrace is not really useful,
but only makes the code (and debugging!) harder than it should be.
2026-06-26 19:33:51 +00:00
Wolfgang Walther d470213857 nix(loadtest): remove pyjwt dependency
Let's use the same dependency when creating key material and when
creating tokens, no need to carry multiple different dependencies and
converting keys between them.
2026-06-26 19:33:51 +00:00
Wolfgang Walther 1446f11222 nix(loadtest): refactor genTargets argument
Instead of passing a path to the output target file, just pass the
directory in which to create the file. This allows the script to create
multiple files later.
2026-06-26 19:33:51 +00:00
Wolfgang Walther ef72787907 nix(loadtest): remove left-over run_command helper
When we changed the test structure to use libfaketime, we moved away
from generating targets as part of the chained command running
with-pg/with-pgrst. This function was left-over from that, the
genTargets script does not need to relay its remaining arguments
arguments anymore.
2026-06-26 19:33:51 +00:00
Wolfgang Walther 66d40c0159 nix(loadtest): merge jwt-rsa-* and jwt-hs-* tests
Instead of creating separate test suites for the key type, the PostgREST
instance now accepts both keys via a JWKSet and the targets are
generated 50/50 for both.

The results are still reported seperately by using a different URL,
which shows up as separate rows in the results.
2026-06-26 19:33:51 +00:00
Wolfgang Walther 2d340e8a8c nix(loadtest): refactor generate_jwt -> generate_target
This makes the next change easier to do, but should be easier to review
step-by-step.
2026-06-26 19:33:51 +00:00
Wolfgang Walther 94be1b7250 nix(loadtest): remove --rsa option from genTargets
The public key is not needed for target generation, only the private
key, so it's enough to pass that.
2026-06-26 19:33:51 +00:00
Wolfgang Walther b32c477b1d nix(loadtest): remove --method argument for JWT loadtests
This is not used by CI and I doubt it's actually used by anyone locally.
2026-06-26 19:33:51 +00:00
Wolfgang Walther e5eb98c8a0 nix(loadtest): fix postgrest-loadtest-report locally
The script currently depends on results from "head" and "main" to be
present to calculate the change ratio. Fallback nicely in case these are
not available, which allows using the same to report URL-by-URL stats
locally.
2026-06-26 19:33:51 +00:00
Taimoor ZaeemandWolfgang Walther bb63c3fade change: use RFC 9535 syntax for jwt-role-claim-key config
BREAKING CHANGE

Breaks the string comparison operators implemented in #3813. Those can
be replaced with regex searches using JSON Path `search()` function.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-26 18:32:05 +00:00
Wolfgang Walther a0bb87d693 nix(loadtest): fix cpu/mem monitoring for tests using libfaketime
Instead of starting a separate process, which would make the pid we
store the one from `faketime` instead of `postgrest`, we can also just
start libfaketime directly via LD_PRELOAD.
2026-06-26 18:30:25 +00:00
Wolfgang Walther ba9336f402 nix: change reference from master to main in README 2026-06-26 18:16:36 +02:00
Wolfgang Walther 5feb55a95b nix: remove PGRST_DELAY AND PGDELAY from README
These had been removed a while ago.
2026-06-26 18:16:25 +02:00
Taimoor Zaeem 67956b3cb1 refactor: log server listening observations after listening socket is ready
Correct behavior is to log this after the listening socket is ready.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-26 16:01:55 +05:00
renovate[bot]andWolfgang Walther a6840f9bf9 chore(deps): update actions/cache action to v6 2026-06-25 20:10:47 +00:00
Taimoor ZaeemandSteve Chavez 081c4c6d70 test(spec): sort imports in SpecHelper.hs
The imports were scattered all around. This commit
sorts and group them.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-25 11:21:28 -05:00
Taimoor ZaeemandSteve Chavez b5f10be167 test: move jwt cache tests from io tests to spec tests
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-25 11:21:28 -05:00
Taimoor ZaeemandSteve Chavez 0bc9fe813f test: move jwt iat claim test from io tests to spec tests
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-25 11:21:28 -05:00
Taimoor ZaeemandSteve Chavez feb4853273 test(io): add test for admin server listening observation
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-25 10:39:16 -05:00
Taimoor Zaeem 9e0f9bc4ee refactor: correct admin server port setting
For admin server, port server-setting was set to the same port as
app server port. This doesn't have any effect because it is overridden
by the already initialized admin server socket, but it's better to
explicitly correct it to avoid anything unexpected.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-23 23:05:02 +05:00
Taimoor Zaeem f5fd2e71b6 test: move jwt error tests from io tests to spec tests
Towards #4946.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-22 23:52:45 +05:00
Taimoor Zaeem ee30bd03ac test(io): remove needless server-timing config coverage tests
These 2 tests don't test any specific behavior - they were added
only to please code coverage.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-22 23:52:45 +05:00
Taimoor Zaeem 1a35af5024 test(spec): rename misleading function name in ErrorSpec.hs
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-22 23:52:45 +05:00
steve-chavez ded9be6774 nix(loadtest): add comment on loadtest-against 2026-06-22 12:55:56 -05:00
Wolfgang Walther ffe4cc6f42 nix(loadtest): allow running multiple postgrest-loadtest in same working directory
There is no need to store the out-links to the various build artifacts
in the current working directory while running postgrest-with-pg, which
is used by postgrest-loadtest. Instead reference the nix store paths
directly.

Resolves #5028
2026-06-21 12:56:42 +02:00
renovate[bot]andWolfgang Walther 79e9e77e46 chore(deps): update vmactions/freebsd-vm action to v1.4.8 2026-06-20 21:57:24 +00:00
renovate[bot]andWolfgang Walther 87b480bc0a chore(deps): update actions/checkout action to v7 2026-06-19 19:09:44 +00:00
renovate[bot]andWolfgang Walther 336d32d1ce chore(deps): update nixbuild/nix-quick-install-action action to v35 2026-06-19 19:09:02 +00:00
renovate[bot]andWolfgang Walther 9aef6f30a8 chore(deps): update vmactions/freebsd-vm action to v1.4.7 2026-06-19 19:08:17 +00:00
Taimoor ZaeemandSteve Chavez 242e5ba87e chore(cabal): explain performance related build flags
These flags were added in 9f97147cc3. Adding
a comment to clarify the use.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-17 14:15:45 -05:00
Michał KłeczekandTaimoor Zaeem 9f97147cc3 perf: add ghc 9.12 options -fexpose-overloaded-unfoldings -fspecialise-aggressively
Before 9.12.1 it was necessary to mark functions as INLINABLE or INLINE to make GHC consider cross-module specialization of polymorphic functions.
9.12 added a new -fexpose-overloaded-unfoldings flag that exposes optimized polymorphic functions in interface files. -fspecialise-aggressively then makes GHC apply aggresive specialization.
2026-06-17 13:13:48 +05:00
Taimoor ZaeemandSteve Chavez 9048332b0e test(io): clarify test names
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-16 13:13:51 -05:00
Taimoor ZaeemandSteve Chavez b6c12abf0e test: move http header related tests from io tests to spec tests
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-16 13:13:51 -05:00
Michał KłeczekandSteve Chavez 8fa26ee865 fix: Start listening after schema cache load
This change ensures PostgREST starts listening on a server socket only after it loaded the schema cache and is ready to handle requests. It is no longer going to return 503 errors during startup until the schema cache is loaded.
2026-06-16 12:30:51 -05:00
Michał KłeczekandWolfgang Walther fae6253932 refactor: make nextListenerDelay local to Listener module
Right now stateNextListenerDelay is only used in Listener module.
This change is a small refactoring moving nextListenerDelay to Listener to increase cohesion and decrease coupling.
2026-06-14 18:39:59 +00:00
Wolfgang Walther 2976eb047b add: support running admin server on unix socket
This is useful when multiple instances run on the same machine, for
example behind a proxy. Unix sockets for web and admin servers can then
be put in the same folder for each instance.

Can be helpful when writing tests as well.
2026-06-14 17:18:37 +00:00
Wolfgang Walther f24bc7092f test(io): fix UnboundLocalError
The `if response` check a few rows down would not actually work without
initializing the variable first. It'd throw:

```
E       UnboundLocalError: cannot access local variable 'response' where
it is not associated with a value
```
2026-06-14 17:18:37 +00:00
Wolfgang Walther 6d87fc5879 test(io): move some tests to unix sockets
No need to run these with TCP and freeport, I believe.
2026-06-14 17:18:37 +00:00
Michał KłeczekandSteve Chavez 54430c3a2c fix: Do not clear the schema cache during retries
retryingSchemaCacheLoad should not clear existing schema cache upon failure - there is no reason to do that. If there is a communication issue with the database server or db is down, clients are going to get 502 anyway. If it was a glitch when loading the schema cache - the clients are going to use old (stale) schema cache for some time until next retry re-loads it successfully.
2026-06-11 13:14:37 -05:00
Taimoor Zaeem e9a016670f Revert "add: string slicing operator for jwt-role-claim-key"
This reverts commit fe0386e9c4.

As discussed in https://github.com/PostgREST/postgrest/pull/4984#issuecomment-4652725178.
2026-06-11 23:07:05 +05:00
steve-chavez 1ebf480258 amend: required membership for postgrest_test_anonymous
Previous test correction required GRANTing membership for
postgrest_test_anonymous.
2026-06-11 12:00:02 -05:00
steve-chavez e7c8d70333 test: correct test_admin_ready_includes_schema_cache_state
There was an oversight on d556cea, `timeout_authenticator` doesn't have
any privileges on the tables so it shouldn't be making requests to the
API server or it will end up with 401 responses.
2026-06-11 11:51:14 -05:00
Wolfgang Walther 9097752888 ci(release): lowercase ghcr repo
ghcr does not support uppercase letters, which was not a problem when
testing in my fork. But the PostgREST org's name obviously has uppercase
letters in it, so we need to lowercase these for the ghcr namespace.
2026-06-10 14:18:12 +02:00
Wolfgang Walther 62503466a0 Revert "ci(ci): avoid running build jobs twice on pushes to main"
This reverts commit 840d3f99ed.

This was a nice idea in theory, but in practice this means that the
build jobs do not run at all on the default branch anymore. This means
the caches they push to won't be used by any other jobs, neither on the
tag pipelines, nor on PRs. This in turn makes all of these jobs really
slow.

Not helpful!
2026-06-10 14:04:59 +02:00
Wolfgang Walther 1560078ce2 ci: push container images to ghcr.io
Resolves #2836
2026-06-10 13:17:37 +02:00
Taimoor ZaeemandWolfgang Walther 8bde0ad474 test: move cors related tests from io tests to spec tests
Towards #4946.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-10 10:27:49 +00:00
Taimoor ZaeemandWolfgang Walther 21433d10c2 refactor(config): use correct data type for server-cors-allowed-origins
Replaces the `Maybe [Text]` with `[Text]`. The `Maybe` is unnecessary
because we handle `Just []` and `Nothing` the same way.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-09 12:35:47 +00:00
Wolfgang Walther 044d623424 ci(test): fail loadtest correctly
Previously this would only fail if *each* row had `:x:` in it, which was
never the case, because the header never has it. It is supposed to fail
when *at least one* row has an .
2026-06-09 12:35:25 +00:00
renovate[bot]andWolfgang Walther 0b5655b9d3 chore(deps): update codecov/codecov-action action to v7 2026-06-07 11:01:07 +00:00
steve-chavez 4df0654998 docs: add how-to for Query Identifier 2026-06-06 06:47:33 -05:00
steve-chavez ad790baefb test: explain verbose reveals query identifier
This also requires enabling shared_preload_libraries=pg_stat_statements.
Really we only need compute_query_id=on, but pg_stat_statements is
useful for other tests and it already enables compute_query_id (since
it's `auto` by default).
2026-06-06 06:47:33 -05:00
Wolfgang Walther cb6fc55a2d chore(deps): update nixpkgs, stackage and hackage pins 2026-06-05 20:29:48 +00:00
Wolfgang Walther b114bb9395 nix: replace silver-searcher with fd
The former has been removed from Nixpkgs and won't be available on the
next bump.
2026-06-05 20:29:48 +00:00
Taimoor ZaeemandSteve Chavez da8738b88e chore: add changelog entry for v14.13
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-05 13:30:59 -05:00
renovate[bot]andWolfgang Walther 3ca876da62 chore(deps): update actions/checkout action to v6.0.3 2026-06-04 09:09:49 +00:00
Laurence IslaandSteve Chavez 416a15480f fix: request failures when work_mem is set on a role 2026-06-03 13:45:34 -05:00
Wolfgang Walther a149405d11 ci: remove deprecation warning
app-id was replaced with client-id
2026-06-03 10:09:38 +02:00
netandWolfgang Walther 3d596e2dbc fix: don't mention retrying in the database connection error message
The PGRST000 database connection error message was "Database connection
error. Retrying the connection.", but reconnection attempts are already
logged separately by the reconnection observation, and on fatal errors
(e.g. authentication failure) PostgREST does not retry at all. Drop the
"Retrying the connection." part, leaving "Database connection error.".
2026-06-03 08:04:40 +00:00
Pedro Gil MoraandWolfgang Walther fae58174fa docs: Add efcore-postgrest library to client-side libraries 2026-06-02 08:19:45 +00:00
Wolfgang Walther 840d3f99ed ci(ci): avoid running build jobs twice on pushes to main
It's enough to run the build jobs in the release pipeline, no need to
run them on the push event as well.
2026-06-02 10:08:16 +02:00
Wolfgang Walther 871aa058f8 ci(release): only update docker description from devel release
This condition was accidentally removed when merging 4 release jobs into
2.
2026-06-02 10:07:37 +02:00
Wolfgang Walther 8552c66c1e ci: Use FreeBSD VM instead of cirrus 2026-06-02 07:36:59 +00:00
netandWolfgang Walther 097a1de67d chore: sort config dump alphabetically
The config dump in Config.hs listed db-config, db-pre-config and the
db-pool-* settings out of alphabetical order, while the rest of the dump
was sorted. Now the whole dump is consistently alphabetical and update
the expected IO test configs to match.
2026-06-02 07:32:11 +00:00
Wolfgang Walther ed4b68c364 test(spec): remove obsolete extra search path from postgis tests
This was brought over in the last commit, because the postgis was
re-using the same config as another test. Now it has a separate config,
so we can reduce it a bit.
2026-06-02 06:49:15 +00:00
Wolfgang Walther 268ab00ed9 test(spec): inline config into test suite
Previously, information about each test-suite was repeated in 3 separate
places:
- as a label and as implicit knowledge in the test-suite itself,
- as a comment in Main.hs, and
- as a configuration in SpecHelper.hs.

With this change, there will be a single source of truth in the test
suite itself. This will allow a single test-suite to easily test
multiple different configurations.
2026-06-02 06:49:15 +00:00
Wolfgang Walther 7803960cd1 test(spec): fix test labels 2026-06-02 06:49:15 +00:00
Wolfgang Walther 0e46474cb2 test(spec): fix typo in comment 2026-06-02 06:49:15 +00:00
Wolfgang Walther 7c806a2bf9 test(spec): move two tests with basic config into list
Everything that uses withApp is in that list.
2026-06-02 06:49:15 +00:00
Wolfgang Walther c2d32be3c5 test(spec): remove obsolete ANALYZE calls
These are now part of the with- helpers.
2026-06-02 06:49:15 +00:00
Michał KłeczekandTaimoor Zaeem 13c0e7061e refactor: simplify control flow in App.postgrest
Currently, authentication and response execution each unwrap ExceptT with separate runExceptT calls, which split the main request flow across nested pattern matching and Either handling. Control flow is complex and difficult to understand.

The goal of this change is to make request execution as sequential
monadic code with clear error handling.

To implement that, request handling is now run in ExceptT over WriterT (Last ByteString) IO monad stack. Auth role is written after authentication succeeds and further returned along the response. Thanks to it response observation generation is centralized at the end of request handling.

It was necessary to abstract monad stack in getAuthResult, lookupJwtCache, postgrestResponse, and withTiming to enable introduction of WriterT.
2026-06-02 11:28:19 +05:00
Wolfgang Walther 1d6e0bd35f nix(loadtest): report percentage change
This reports the percentage change between the current head branch and
the main branch, which is exactly the number we'll want to make our
decisions on "success or fail" on.

CI failures will initially be reported for regressions of 5% or more on
an individual number.
2026-06-01 20:49:35 +00:00
Wolfgang Walther ff92846e64 nix(loadtest): report multiple percentiles at once
We can look at the data over time and see which percentile works best
for our purpose.
2026-06-01 20:49:35 +00:00
Wolfgang Walther 640039a180 nix(loadtest): move errors into mixed
Because we separate loadtest results per URL now, we can move the error
tests into the regular mixed bag of loadtests - we will be able to tell
from the misspelled URLs when we hit a regression in that area.

We should be able to do similar things for JWT tests, but we'll need
more infrastructure here.
2026-06-01 20:49:35 +00:00
Wolfgang Walther 59dffc258a nix(loadtest): report latency instead of rate
We previously used "rate", i.e. number of requests per second, as the
primary metric to judge loadtest results. However, this has always been
varying from run to run quite a bit, especially in CI where other jobs
possibly run on the same VM host.

The run-to-run variance has massively increased after splitting the
results up per request. Example run in CI with rate on the PR
introducing this change (on which we would expect no change at all):

| rate [1/s]                         |   main |   head |    Δ |
|:-----------------------------------|-------:|-------:|-----:|
| /                                  |    9.4 |    9.5 |   1% |
| /actors                            |  870.4 | 1023.0 |  18% |
| /actors?actor=eq.1                 |  188.5 |  198.6 |   5% |
| /actors?actor=eq.1&columns=name    |  197.3 |  167.1 | -15% |
| /actors?select=*,roles(*,films(*)) |  153.9 |  144.9 |  -6% |
| /films?columns=id,title            |  157.9 |  182.6 |  16% |
| /films?columns=id,title,year,...   |   87.0 |   87.1 |   0% |
| /roles                             |  204.5 |  267.3 |  31% |
| /rpc/call_me                       |  231.3 |  208.8 | -10% |
| /rpc/call_me?name=John             |  212.2 |  201.7 |  -5% |

From the data we can easily tell that the very reason that rate as a
paramter has only worked, so far, because the data was *heavily*
dominated by the requests on the root endpoint for OpenAPI. The longer
duration makes the request much less vulnerable for concurrent activity.
For all other requests its essentially not possible to judge the effect
of a PR this way.

One way to counter this would be to massively increase the time the
loadtest runs. More samples will result in a smoother average. However,
that's not practical for usability of CI. In the original PR #1812 I
already evaluated using the *minimum latency* as the most reliable
criteriumi, but this has never really caught on. The theory behind this
is: The variation in timings between requests is happening because of
concurrent activity, priority chosen by the scheduler, availability of
resources and such - all factors *outside* our control, and *irrelevant*
to the Haskell code we're writing.

Using the minimum latency is an estimation of how fast the code can run
*in the best case*. This might not be a number relevant for production,
but it's much more directly related to the code we write.

Here's to show how variation becomes *much* smaller with minimum latency
as the parameter:

| min latency [μs]                   |     main |   head |    Δ |
|:-----------------------------------|---------:|-------:|-----:|
| /                                  |   1275.3 | 1263.6 |  -1% |
| /actors                            |     10.0 |    9.9 |  -1% |
| /actors?actor=eq.1                 |     50.7 |   48.3 |  -5% |
| /actors?actor=eq.1&columns=name    |     54.1 |   54.0 |   0% |
| /actors?select=*,roles(*,films(*)) |     63.2 |   61.9 |  -2% |
| /films?columns=id,title            |     51.1 |   50.7 |  -1% |
| /films?columns=id,title,year,...   |    121.9 |  121.8 |   0% |
| /roles                             |     42.9 |   42.6 |  -1% |
| /rpc/call_me                       |     45.6 |   45.4 |   0% |
| /rpc/call_me?name=John             |     44.4 |   44.2 |   0% |

Since we're separating results per request now, we can only sensibly
focus on *one* parameter - otherwise this would get really clunky
UI-wise. Especially for automated CI failures, minimum latency is the
logical choice.

This commit starts using minimum latency, i.e. P0, but any percentile
should be an improvement over the status quo. A later commit will change
to a different P-value.
2026-06-01 20:49:35 +00:00
Wolfgang Walther 8a9f98371e nix(loadtest): group results by status, method and URL
Different requests hit different code paths and perform very
differently. By looking at each request type separately, we should be
able to get a much better idea of what kind of change in performance
we're looking at and where the root cause might be.

It will hopefully also allow us to migrate some of the other test-cases
into the main loadtest.
2026-06-01 20:49:35 +00:00
Wolfgang Walther 5cdf2e0569 nix(loadtest): remove noise from report
Ultimately, we only look at the `rate` column, so we can just as well
remove all other columns.

This makes the next step, when we split results by request type, much
less noisy.
2026-06-01 20:49:35 +00:00
Taimoor ZaeemandSteve Chavez 669fc9ede8 docs: restructure PostGIS page
Restructure PostGIS page with order:

- `Accept: application/geo+json` request example
- Generated column example
- String representation example

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-06-01 12:59:15 -05:00
Michał KłeczekandSteve Chavez 0df56f9ea8 refactor: assemble main transaction session in MainTx
This change makes the API surface between MainTx and App smaller.

Currently, App reconstructs a database transaction by unpacking the isolation
level, transaction mode, DbHandler, and transaction runner returned by MainTx.
That exposes MainTx internals at the call site even though MainTx already owns
query setup, execution, decoding, and rollback behavior.

The goal is to keep transaction assembly in MainTx while App remains responsible
for pool execution, database error mapping, and response orchestration. DbTx now
carries the assembled SQL session, and App passes that session directly to the
connection pool.
2026-05-31 15:52:56 -05:00
Michał KłeczekandSteve Chavez 56ad5a9324 refactor: get rid of dbQueryTimings from SchemaCache
Schema cache query timings are only needed immediately after a schema-cache reload to emit SchemaCacheQueriedObs. Storing them inside SchemaCache makes the cache carry transient observability data that is not part of the cached schema state and is never used by request handling.

This change makes querySchemaCache to return query timings in a tuple in parallel to SchemaCache and removes dbQueryTimings field.
2026-05-31 14:20:35 -05:00
Taimoor ZaeemandSteve Chavez 9162cea7ba test(io): remove unused config variables from config test
Config variables are tested already via reading the config files in
the `configs/` directory.

If more are to be tested, it should be done via adding a file in
`configs/` and compare it with its associated file in `configs/expected/`.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-30 09:51:26 -05:00
Taimoor ZaeemandSteve Chavez e1d547623e nix(docs): print message to console on script runs
It's not immediately visible that these scripts are run when
doing `postgrest-docs-check`.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-27 15:23:59 -05:00
Michał KłeczekandSteve Chavez 998d232ac0 docs: PGRST107 error should map to 406 not to 415 2026-05-27 14:37:23 -05:00
Michał KłeczekandWolfgang Walther 1a6ba2072c refactor: Simplify App.initServerSocket
This change gets rid of unnecessary explicit bindRandomPortTCP in initServerSocket. Returned port value was ignored in removed code anyway as assigned port retrieval from an open socket is handled elsewhere.
2026-05-24 23:56:05 +02:00
renovate[bot]andWolfgang Walther ae00c04faf chore(deps): update docker/login-action action to v4.2.0 2026-05-22 20:54:17 +00:00
renovate[bot]andWolfgang Walther e0e14a1fe4 chore(deps): update docker/setup-buildx-action action to v4.1.0 2026-05-22 20:52:03 +00:00
Taimoor ZaeemandSteve Chavez 901624a891 chore: add changelog entry for v14.12
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-21 10:26:17 -05:00
Michał KłeczekandSteve Chavez 913fe001e5 refactor(remove): schema cache load delay config
Remove internal schema cache load and relationship load sleep settings plus
the delay wrappers they enabled. Drop IO tests that depended on the removed
settings.
2026-05-19 10:59:27 -05:00
Michał KłeczekandSteve Chavez 86d6ed10bb test: cover stale schema cache database error
Add an IO test that drops a table while schema cache reload is delayed. It verifies the stale cache path returns PostgreSQL 42P01 and the refreshed cache returns PGRST205.
2026-05-19 09:11:04 -05:00
renovate[bot]andWolfgang Walther 55f9c45899 chore(deps): update codecov/codecov-action action to v6.0.1 2026-05-19 07:04:42 +00:00
Michał KłeczekandTaimoor Zaeem 07938aaab2 amend: Move #4622 to proper Changelog section 2026-05-19 10:55:53 +05:00
Wolfgang Walther 61da5dd88b nix(loadtest): prevent timing errors for worst-case JWT test
Instead of taking wild guesses at the runtime of the target generation
itself, we're just making sure to reset the system time to a fixed value
when we ultimately start PostgREST. This allows us to create the right
JWT expiry values ahead of time.
2026-05-18 16:23:37 +00:00
Wolfgang Walther d9f73ac71b nix(loadtest): remove duplicate line 2026-05-18 16:23:37 +00:00
Taimoor ZaeemandWolfgang Walther e6d8d95c0a chore: remove unplanned TODO in Logger.hs
We decided against doing this in #4913, therefore removing it. With that,
also mentioning that control flow never reaches there to avoid confusion.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-18 16:23:08 +00:00
Wolfgang Walther 9d67febb93 ci(release): merge 4 jobs into 2
No need to do this in as many jobs. Splitting all the github stuff from
docker stuff into two jobs is enough. This still allows to conditionally
enable docker jobs in contributors repos, depending on whether the
relevant docker credentials are provided - but avoids using too many
concurrent runners.
2026-05-18 16:22:49 +00:00
steve-chavez 6220ab3f31 test: adjust test_positive_pool_metric
Adjusted so it waits for liveness instead of requiring sleeping
2026-05-18 10:46:02 -05:00
steve-chavez 1eba9855c1 test: change wait_for_readiness to an enum
This so we can wait for both readiness and liveness
2026-05-18 10:45:36 -05:00
Michał KłeczekandSteve Chavez a297391071 fix: implement connection tracking in metrics
Right now metrics observation handler does not track database connections but updates a single Gauge based on HasqlPoolObs events. This is problematic because Hasql pool reports various connection events in multiple phases. The connection state machine is not simple and to precisely report the number of connections in various states, it is necessary to track their lifecycles.

This change adds a ConnTrack data structure and logic to track database connections lifecycles. At the moment it supports "connected" and "inUse" connection counts precisely. The "pgrst_db_pool_available" metric is implemented on top of ConnTrack instead of a simple Gauge.
2026-05-18 10:09:00 -05:00
Wolfgang Walther 4f9bc89ab2 nix(loadtest): remove outdated PGRST_DB_URI default
This was required for v9 and earlier, but these don't build with the
current nix invocation anymore anyway. Even loadtesting against v10 does
not work, because `--version` is used in one of the wait scripts and
this was only added in v11.2.

So no need to pretend we'd support comparing against older versions.
2026-05-18 15:18:05 +02:00
Taimoor ZaeemandWolfgang Walther 084a8eca55 test(io): move utility functions to util.py
The functions `drain_stdout` and `match_log` should be in `util.py`
so they can be reused in other modules.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-18 12:54:08 +00:00
Wolfgang Walther ed344c795a ci(release): run docker job on regular VM runner
Tried to run it on a -slim runner in fb24b6b3, but the result is that no
docker socket is available. This jobs needs one of the big runners.
2026-05-18 14:46:21 +02:00
Wolfgang Walther c335da9334 ci: avoid running pg18 tests twice
We already test PG18, the default, with the coverage job, so why run it
twice?
2026-05-18 14:06:16 +02:00
Wolfgang Walther fb24b6b37f ci: use -slim runner for small jobs
No need to spin up full VM runners for small automation tasks, when we
can use single-CPU runners in containers instead.

https://docs.github.com/en/actions/reference/runners/github-hosted-runners#single-cpu-runners

(some of this will potentially not work, because dependencies in the
slim image might not be available - however, it makes no sense to create
this as a PR, because all jobs touched here run on branches only. Thus
pushing directly to main)
2026-05-18 14:05:41 +02:00
steve-chavez 8868d13214 test: adjust sleep in test_positive_pool_metric
Otherwise it's not enough time for the pool metric to
reach a negative value.
2026-05-17 16:39:11 -05:00
Wolfgang Walther c7260bddab test: consistently analyze tables exactly once
Disabling the autovacuum daemon should also help reproducibility in
theory, although I don't know of any cases where we hit a problem with
that.

VACUUM changes the order of rows that PostgreSQL returns for some table
without explicit ordering, thus doing the latter to make it consistently
reproducible.

After ANALYZE estimates are 100% exact for the moment, so some requests
which returned 206 Partial Response now return 200 instead. The fact
that PostgREST returns 206 on an unfiltered endpoint can probably be
considered a bug.
2026-05-17 20:03:15 +00:00
Wolfgang Walther abe3e511bf nix(postgrest-with-pg-...): use postgresql.conf instead of CLI arguments
This is easier to reason about and easier to extend. Also removes a tiny
bit of repetition.
2026-05-17 20:03:15 +00:00
steve-chavez 70327cf869 test: negative pgrst_db_pool_available in metrics
Proves the failure on https://github.com/PostgREST/postgrest/issues/4622.

This doesn't require additional test infra, only nginx. Taking advantage
of the `stream {}` context which is also compatible with unix socket
besides TCP.
2026-05-17 12:58:58 -05:00
Wolfgang Walther a9713208e4 ci: ignore loadtest failures for jwt-rsa-cache-worst 2026-05-15 23:25:38 +02:00
Wolfgang Walther 2783ec7b51 ci: run other loadtests to completion when one of them fails
The most likely offender to fail early is jwt-rsa-cache-worst - which
then cancels *all* loadtests at once. This is unnecessary churn for CI.
Instead we can re-run only that one test later.
2026-05-15 23:21:54 +02:00
Taimoor Zaeem ab43ee1fbb docs: add redirect of the moved postgis section
In 055921ea, we missed redirecting the old link to the new one.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-15 23:01:36 +05:00
renovate[bot]andWolfgang Walther 2ab2ea2c6c chore(deps): update dependency macos to v26 2026-05-14 19:45:32 +00:00
Aiden ParkandWolfgang Walther 703a5535cf chore: remove duplicated words in Listener and Plan comments
Signed-off-by: Aiden Park <275402320+vip892766gma@users.noreply.github.com>
2026-05-14 14:23:09 +00:00
renovate[bot]andWolfgang Walther 187ea6d546 chore(deps): update actions/create-github-app-token action to v3.2.0 2026-05-13 21:01:47 +00:00
Taimoor Zaeem 055921eaf5 docs: move PostGIS under Integrations
For better visibility.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-13 12:41:44 +05:00
Taimoor ZaeemandWolfgang Walther f90d7d7296 ci: catch linkcheck failure
The scheduled linkcheck job fails without notification to maintainers.
To catch the failure, a github workflow step has been added which
comments on issue #4106 so that the maintainers get notified.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-12 17:27:45 +00:00
renovate[bot]andWolfgang Walther 27aea13798 chore(deps): update cachix/cachix-action digest to 5f2d7c5 2026-05-12 13:10:57 +00:00
renovate[bot]andWolfgang Walther d75beed5dd chore(deps): update korthout/backport-action digest to 6606540 2026-05-12 13:03:16 +00:00
Wolfgang Walther 243bbb1b61 test: make some tests more stable
These fail with orioledb, because it randomly returns a different order.
Of course, that's allowed - ordering is not guaranteed unless specified.
We should write our tests with more explicit ordering in general, even
if multiple major versions show the same behavior.
2026-05-08 18:03:54 +02:00
Taimoor ZaeemandSteve Chavez 98f8e52b46 refactor: remove auth and logging middleware
This commit removes auth middleware for it hides
side effects and obscures logic. The auth operations
are now done in its own stage in the request-response
cycle.

It also removes the logging middleware because now
we instead use observation module to log the response.
2026-05-07 11:47:17 -05:00
Wolfgang Walther 0bba1d265a ci: test NixOS' VM test for PostgREST
This adds a CI job to run our latest postgrest version against the NixOS
VM test currently available in Nixpkgs. Now, this will not always be in
sync, so has the potential to be failing. However, since the Nixpkgs VM
test is really simple, this should only happen when we introduce a
breaking change on a very fundamental level. The failing test will then
be resolved once the new version is available in Nixpkgs and we have
updated our lock file.

This is essentially just a sanity check to make sure we're not breaking
something fundamentally - and if we do, it's a head up for me to adjust
the Nixpkgs tests accordingly. Those might otherwise break unnoticed
since Nixpkgs does not have a good notification system for such
breakages in place.

We do use the chance to run the static executable in this test, which
was previously not tested at all.

It also gives us a first test whether NixOS VM tests work well in GitHub
Actions.
2026-05-07 07:10:56 +00:00
Taimoor ZaeemandWolfgang Walther 96f12f83f0 ci: cancel previous runs for tag pipelines
Reverts the change in ad5bb38d70.

https://github.com/orgs/community/discussions/13015 is now solved.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-07 07:08:41 +00:00
steve-chavez 02a1de2afe test: OPTIONS produce logs but not CORS preflight 2026-05-06 17:21:31 -05:00
renovate[bot]andWolfgang Walther beafd0b53f chore(deps): update korthout/backport-action digest to bf97bcf 2026-05-06 19:47:13 +00:00
Taimoor Zaeem 4c04572783 nix(chore): add commit prefix in release commit message
We should follow our commit convention.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-06 23:26:06 +05:00
steve-chavez 6b3ec5d9e0 test: empty db-anon-role lacks role in logs 2026-05-05 19:29:59 -05:00
steve-chavez 6804ce3cbe test: empty schema cache error has role in logs 2026-05-05 19:29:59 -05:00
steve-chavez 42e53e84e1 test: invalid rpc call contains role in logs 2026-05-05 19:29:59 -05:00
steve-chavez 8105375b64 test: expired JWT shows empty role in logs 2026-05-05 19:29:59 -05:00
steve-chavez e39740cca3 nix: allow empty PGRST_DB_ANON_ROLE on postgrest-run
So we can do `PGRST_DB_ANON_ROLE="" postgrest-run` since empty string has
meaning. argbash conflates both unset and empty string so this requires
a workaround.
2026-05-05 17:32:50 -05:00
Taimoor ZaeemandWolfgang Walther c8e7f11688 chore: add changelog entry for v14.11
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-05 15:53:15 +00:00
Taimoor Zaeem fd3f9378ac fix: functions count in startup logs not counting overloaded functions
Renames the "Functions" in the logs to "RPCs". This clarifies that
we log number of callable functions and not the number of SQL objects.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-05-04 20:42:20 +05:00
Michał KłeczekandWolfgang Walther 85d91e5c0a add: Publish GHC runtime metrics to Prometheus metrics endpoint 2026-05-03 11:41:02 +00:00
Laurence IslaandGitHub c09394517d fix: call pg_notification_queue_usage() automatically when the LISTEN channel bug is detected (#4858)
There's a PostgreSQL bug that doesn't let any listener to register in the DB:
https://www.postgresql.org/message-id/flat/CAK98qZ3wZLE-RZJN_Y%2BTFjiTRPPFPBwNBpBi5K5CU8hUHkzDpw%40mail.gmail.com
The only workaround is to advance the async notification queue tail,
which can be done by executing:  "SELECT pg_notification_queue_usage();".
Before we just logged a HINT with this suggestion, but now we call that function directly
and then let the listener to automatically recover.

No automated tests were added here as it would be too complex and this is a PostgreSQL bug. But this was manually tested following the steps on https://github.com/PostgREST/postgrest/pull/4581#issuecomment-3690610592
2026-04-30 18:22:59 -05:00
Michal KleczekandGitHub 04a0e041e4 fix: Restore showing LISTEN pgrst in pg_stat_activity 2026-04-30 17:21:12 -05:00
Laurence IslaandWolfgang Walther 5f6f7dca44 fix: unexpected results when embedding the same table twice 2026-04-29 09:10:48 +00:00
Wolfgang Walther ea08a4d767 ci: test build of dev shell on darwin 2026-04-28 12:56:53 +00:00
Wolfgang Walther 822f8ef802 nix: remove unused tools from dev shell
I doubt that anyone would use cabal2nix or update-nix-fetchgit manually
- and at the same time everyone surely has git installed through their
OS anyway, otherwise they can't even get that far.
2026-04-28 12:56:53 +00:00
Wolfgang Walther c8439e437c ci: use nix-build-uncached to save useless downloads
There is no point to actually realize the derivations to disk, when they
are in cache. `nix-build-uncached` checks the cache and stops before
downloading dependencies if they are available. It will however build as
normal on a cache-miss. Should speed up CI when no Nix dependencies
change a fair bit.

We can only do this for the MacOS job, which only tests the builds. We
can not do this for other jobs which require the build results to be
present before uploading parts of them as artifacts.
2026-04-28 12:56:53 +00:00
Laurence Isla dc179665dc chore: fix incorrect username in changelog entry 2026-04-27 18:22:58 -05:00
Wolfgang Walther 3ec9b94c79 ci: avoid useless download of GHC during stack builds
Forgot to adjust when updating stackage.
2026-04-27 19:52:57 +00:00
renovate[bot]andWolfgang Walther 9b9521b75a chore(deps): update korthout/backport-action action to v4.5 2026-04-27 19:51:37 +00:00
Laurence Isla 41b86fffa5 fix: no longer retries the transaction on 40001 errors 2026-04-27 14:30:10 -05:00
Taimoor ZaeemandWolfgang Walther bf758698b3 fix: login with uppercase and mixed case role names
PostgREST failed when querying role settings where current
role name contained uppercase letters. This commit resolves
it by quoting the CURRENT_USER.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-04-27 09:19:26 +00:00
Wolfgang WaltherandMichal Kleczek a5cc457875 chore(deps): update hasql to 1.9.3.1
Michael and I arrived at mostly the same things independently. Took some
of his, some of mine.

Co-authored-by: Michal Kleczek <michal@kleczek.org>
2026-04-27 09:12:31 +00:00
Wolfgang Walther f80122e12b ci: trigger workflows when nixpkgs pin is updated
Otherwise we might break things...
2026-04-27 08:29:41 +00:00
Wolfgang Walther 2e5baaf296 chore(deps): update nixpkgs pin 2026-04-27 08:29:41 +00:00
Wolfgang Walther 0e0bc6a2c1 chore(deps): fix docs deps
These should have been updated by postgrest-nixpkgs-upgrade, when I
upgraded the Nixpkgs pin, but I did not follow my own readme, and just
ran nix flake update...
2026-04-27 08:29:41 +00:00
Taimoor ZaeemandSteve Chavez 38829191a7 test(load): reduce num of tables in errors loadtest
The table not found error only generates hint when the total number of
tables in a schema are less than 500.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-04-26 12:32:47 -05:00
Taimoor ZaeemandWolfgang Walther c9253ed056 test(load): update error loadtest endpoint to generate hint
This endpoint didn't generate error hint which is not desired
for loadtest.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-04-24 10:30:30 +00:00
Taimoor ZaeemandWolfgang Walther a1ab388ab6 test(doctests): remove redundant doctests in Error.hs
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-04-23 08:23:28 +00:00
renovate[bot]andWolfgang Walther de4ad62562 chore(deps): update korthout/backport-action action to v4.4.0 2026-04-21 09:05:36 +00:00
Wolfgang Walther e2a677e21d chore: try to re-enable codecov annotations
Not sure whether this works, but it likely can't hurt.
2026-04-20 23:24:49 +02:00
Michał KłeczekandSteve Chavez baebacf3db fix: shutdown should wait for in flight requests
Upgraded warp to 3.4.13 which fixed https://github.com/yesodweb/wai/issues/853
Changed interrupt handling so that instead of killing the main thread, listening sockets are closed which triggers warp graceful shutdown.
2026-04-20 16:19:13 -05:00
Wolfgang Walther 3b1373ec02 chore: update the nix upgrade docs
These have been outdated for a while, not only because we're now using
flake.nix, but also in various other places.

Resolves #4816
2026-04-20 19:29:26 +00:00
Wolfgang Walther 695678ad40 chore: add changelog for static aarch64 builds and remove left-overs
I guess I forgot to update a few places when doing this.
2026-04-20 19:29:26 +00:00
Wolfgang Walther 92ba6d5caa ci: fix actions/cache-on-main
My recent commit to fix the stack cache on macOS only changed the
restore keys, but no cache would ever be saved that way. I effectively
disabled all caching...
2026-04-19 23:03:07 +02:00
Wolfgang Walther 6b4022b0c1 ci: fix docker push for release job
Unfortunately, `docker manifest create` does not support image digests,
so we need to push a tag for each version of the image. Furthermore, the
docker CLI does not provide a way to delete that tag afterwards, so we
will end up with those temporary tags.
2026-04-19 22:20:46 +02:00
Wolfgang Walther 0cb354be7a ci: fix stack cache for macOS
We now have two stack jobs for macOS, but they use the same cache, which
is very inefficient. Adding the architecture to the cache key will give
each of these jobs a separate cache.
2026-04-19 21:48:06 +02:00
Wolfgang Walther 1c869bf3a6 ci: remove builds for GHC 9.6.7 and 9.8.4
Now that we can build with at least GHC 9.10 on all platforms, there is
no need to keep testing these older versions anymore.
2026-04-19 20:17:48 +02:00
Wolfgang Walther fee820bac2 chore: release static executable for aarch64-linux
Also makes the aarch64 variant of the docker image consist of a single
static executable, similar to the x86_64 variant.
2026-04-19 17:53:48 +00:00
Wolfgang WaltherandMichal Kleczek 1676314fc3 nix: build with GHC 9.12.3
Jumping past GHC 9.6, 9.8 and 9.10 immediately, because these versions
do not successfully build the static binary with TemplateHaskell and
iserv-proxy, yet.

Michal wrote the changes to hsie.

Co-authored-by: Michal Kleczek <michal@kleczek.org>
2026-04-19 17:53:48 +00:00
Wolfgang Walther 49a16fe505 ci: build with cabal and GHC 9.12.3 2026-04-19 17:50:23 +00:00
Wolfgang Walther 43447ed711 test: fix two doctests
Spotted them randomly while debugging something else.
2026-04-19 19:10:56 +02:00
Wolfgang Walther edbd2a3dca chore: remove renovate haskell updates
These don't really make sense, because it's non-trivial to construct a
build plan that *ensures* the dependency that renovate creates a PR for
is tested with the latest version. Cabal might just select a build plan,
that doesn't include the dependency, in which case we could just as well
have no upper bounds at all.

Since there is no good way to test these changes (I tried, a while
ago!), let's just disable these PRs. They are more annoying than
anything else.
2026-04-19 18:38:51 +02:00
Wolfgang Walther 03f7234360 ci: test against PostgreSQL 18 2026-04-19 13:16:55 +00:00
Wolfgang Walther d79f28fd75 chore: fix stack.yml's nix integration once more
We don't need an impure environment anymore, because we can't run the
test suite via stack anyway.
2026-04-19 13:16:55 +00:00
Wolfgang Walther 051b4799de chore(deps): update nixpkgs to latest nixpkgs-unstable
This allows building TemplateHaskell with GHC 9.6+ via iserv-proxy.
2026-04-19 13:16:55 +00:00
Wolfgang Walther 1b74b93466 remove: drop support for PostgreSQL 13
PostgreSQL 13 has been EOL since November 2025 and will not be available
in the next Nixpkgs bump anymore.
2026-04-19 13:16:55 +00:00
Wolfgang Walther 4ccb125e85 chore: remove outdated comment
We started actually depending on postgresql-libpq in
5356f4e973.
2026-04-19 13:52:19 +02:00
steve-chavezandWolfgang Walther b6c6bb27a8 ci: reinstate macos x86 releases
Closes https://github.com/PostgREST/postgrest/issues/4755
2026-04-19 11:26:06 +00:00
Wolfgang Walther 1f70e42d3b ci: build with cabal and GHC 9.10.3 2026-04-19 10:07:14 +00:00
Wolfgang Walther be7a42ff30 chore: stackage 23.28 -> 24.37
This updates stack builds to GHC 9.10.3.
2026-04-19 10:07:14 +00:00
Wolfgang Walther 9ed7f52685 nix: put tmpdirs directly into /tmp to fix long unix socket paths
I'm not sure whether that's Lix specific, but when I enter nix-shell, I
get quite a long TMPDIR, which results in all postgrest-with-pg-XX
scripts failing, because the unix socket path is too long. If TMPDIR was
not set, mktemp would fall back to /tmp, so we can just do that in any
case, which guarantees a short path in every case.
2026-04-18 20:53:21 +02:00
Wolfgang Walther dfb3602318 nix: remove left-over comment 2026-04-18 20:53:20 +02:00
Wolfgang Walther f1cd9a2136 chore: stackage 22.44 -> 23.28
This updates stackage builds to GHC 9.8.4.
2026-04-18 18:38:32 +00:00
Taimoor ZaeemandWolfgang Walther 759917772c chore: add changelog entry for v14.10
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-04-17 10:20:30 +00:00
steve-chavez bcc8998e5e add: log schema cache queries' timings
This adds a new log line that shows each schema cache query time individually, only on
`log-level=debug`. Like so:

```
$ PGRST_LOG_LEVEL=debug postgrest-with-pg-17 -f test/spec/fixtures/load.sql postgrest-run

....
10/Apr/2026:21:48:45 -0500: Schema cache queried in 192.2 milliseconds
10/Apr/2026:21:48:45 -0500: tables: 72.027 ms, keydeps: 20.118 ms, rels: 6.189 ms, funcs: 35.010 ms, comprels: 4.319 ms, dreps: 1.614 ms, mhandlers: 7.419 ms, tzones: 43.025 ms
```

This helps debug specific schema cache queries being slow like on
https://github.com/PostgREST/postgrest/issues/4613#issuecomment-4210191065 and
https://github.com/PostgREST/postgrest/issues/3046#issuecomment-3469059948.
It also closes https://github.com/PostgREST/postgrest/issues/3215,
which main motivation was to find out which query is slow.

Implementation details
---------------------

To time each query inside a transaction in pure SQL, we do:

```sql
-- start timer
select set_config('pgrst.tmp_x', clock_timestamp()::text, false);
-- run the query
select <query>
-- end timer
select set_config('pgrst.tmp_x', (clock_timestamp() - current_setting('pgrst.tmp_x', false)::timestamptz)::text, false);

-- .... repeated for every query

-- at the end we capture all the timings with
select extract('milliseconds' from current_setting('pgrst.tmp_x', false)::interval), extract(..;
```

Considerations
--------------

Only added this on `log-level=debug` because while the queries are fast
and the data is valuable, it triples the amount of queries we run during schema cache refresh,
which could be troublesome on slow networks. It's possible to reduce the
amount of queries by starting and stopping timers in one statement, but
this would still double the amount of queries and makes the code messy,
doesn't seem worth it.

Also it would pollute pg_stat_statements, it's only required to debug certain
extreme cases anyway.
2026-04-16 13:10:24 -05:00
steve-chavez 6af77360d3 docs: add note about schema cache queries 2026-04-16 13:10:24 -05:00
renovate[bot]andWolfgang Walther f939a81ad6 chore(deps): update ubuntu:noble docker digest to c4a8d55 2026-04-16 10:44:13 +00:00
renovate[bot]andWolfgang Walther bf8a52694a chore(deps): update haskell-actions/setup action to v2.11.0 2026-04-15 18:05:32 +00:00
Wolfgang Walther 4fbde347f8 docs: fix broken link 2026-04-15 09:29:35 +02:00
Michal KleczekandGitHub aca58c8571 fix: Flush pool as late as possible during schema cache reloading (#4645)
retryingSchemaCacheLoad flushes the pool upon every retry before it starts reloading the schema. This is too early as schema reloading might take some time during which new connections might be acquired. The consequence is that:
* upon successful schema cache reload we might have some connections created with the old schema cache
* we close connections upon each retry and under load we will keep closing and re-opening connections until schema cache load succeeds

This change is to make sure we flush the pool only after successful schema cache querying but before loading (so that connections acquired during loading wait for it and do not interfere with timing the loading process).
2026-04-14 17:09:25 -05:00
Taimoor ZaeemandSteve Chavez 4ca34c0695 chore: update cabal spec version in postgrest.cabal
The current version is too old and it prevented using latest features
like common stanzas and visibility fields.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-04-14 12:16:48 -05:00
renovate[bot]andWolfgang Walther 26ce39bb46 chore(deps): update actions/cache action to v5.0.5 2026-04-13 21:04:11 +00:00
Taimoor ZaeemandSteve Chavez 4e8c4f218f chore: add changelog entry for v14.9
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-04-13 11:15:18 -05:00
renovate[bot]andWolfgang Walther 070eb2fc3a chore(deps): update actions/create-github-app-token action to v3.1.1 2026-04-12 15:51:35 +00:00
renovate[bot]andWolfgang Walther 431f8586f5 chore(deps): update actions/upload-artifact action to v7.0.1 2026-04-12 15:25:15 +00:00
steve-chavez c9adaed79d docs: target_session_attrs=read-only in libpq>=14 2026-04-10 15:14:41 -05:00
Michał KłeczekandSteve Chavez 381e12efa2 refactor: remove usage of Control.Debounce
This change replaces usage of Control.Debounce in Logger with makeDebouncer function extracted from AppState to a new module PostgREST.Debounce.
2026-04-10 12:22:08 -05:00
Michał KłeczekandSteve Chavez f4fc29855e refactor: remove leftover comment in AppState 2026-04-10 12:22:08 -05:00
steve-chavez 2fb2e86473 fix: remove red herring warp logs on default log-level
The logs added on e95e815483 are red
herrings under normal operation.

This moves them to `log-level=debug` and removes "error" from the
message prefix.

Fixes https://github.com/PostgREST/postgrest/issues/4799
2026-04-09 16:44:25 -05:00
Michał KłeczekandSteve Chavez c31aeecdcf test: add test_pool_acquisition_timeout_logs_are_debounced to io tests 2026-04-09 13:59:17 -05:00
Michał KłeczekandSteve Chavez 5d4f82d70f add: log pool flushes
Emit a dedicated PoolFlushed observation when the DB pool is released during schema cache reload.
2026-04-09 10:41:29 -05:00
Michał KłeczekandSteve Chavez ad907219be test(refactor): make debug output testing more robust
This change introduces a match_log function, that implements lenient matching of log output lines. Tests of log output are retrofited to use the function.
2026-04-09 10:41:29 -05:00
renovate[bot]andWolfgang Walther 463b701e9f chore(deps): update haskell-actions/setup action to v2.10.4 2026-04-09 08:48:59 +00:00
Steve Chavez 3d98f8d65b refactor: clarify debouncer function 2026-04-07 13:44:03 -05:00
Michał KłeczekandSteve Chavez a4c1d945ee refactor: Simplify schema cache loading triggering logic
DISCLAIMER:
This commit was authored entirely by a human without the assistance of LLMs.

Using debouncer to trigger schema cache loading makes it difficult to understand when exactly it is triggered.
2026-04-07 13:44:03 -05:00
renovate[bot]andWolfgang Walther ff068aa1b7 chore(deps): update ubuntu:noble docker digest to 84e77de 2026-04-07 08:10:36 +00:00
Taimoor ZaeemandSteve Chavez 9e48a338c3 chore: add changelog entry for v14.8
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-04-06 10:50:16 -05:00
steve-chavez b610b1ea51 docs: clarify set operators need views/functions
Closes https://github.com/PostgREST/postgrest/issues/4780.
2026-04-03 17:02:21 -05:00
Artur Bento de CarvalhoandGitHub 69c6a0aa36 fix: use int32/int64 formats for integer types
Fixed integer type mapping in OpenAPI 2.0: replaced the invalid integer format with int32/int64 and added the toSwaggerFormat function to map PostgreSQL types to valid OpenAPI 2.0 formats:

smallint -> int32
integer -> int32
bigint -> int64
2026-04-03 14:21:32 -05:00
renovate[bot]andWolfgang Walther a5e66a4448 chore(deps): update docker/login-action action to v4.1.0 2026-04-02 20:07:13 +00:00
Michał KłeczekandSteve Chavez 328598eaed test: Schema cache load debouncing
test: adjust replicateM to 100
2026-04-02 09:02:18 -05:00
Taimoor ZaeemandSteve Chavez d6c338d588 add: config db-timezone-enabled for optional querying of timezones
To avoid repeated querying of `pg_timezone_names` every time schema
cache is reset, `Prefer: timezone` can be disabled by setting
`db-timezone-enabled = false`.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-04-01 12:48:38 -05:00
Michał KłeczekandSteve Chavez bd5de884fa test(io): add test_so_reuseport_zero_downtime_handover 2026-03-31 13:33:33 -05:00
Michal KleczekandGitHub 8d5b1018d3 refactor(test): provide means to validate metrics and observations
Some helpers are provided for introspecting metrics already (used in JWT cache tests). This change provides facilities to additionally validate emited Observation events.
A new Spec module is also implemented, adding basic tests of schema cache reloading - their main goal is to excercise the new infrastructure.
2026-03-31 13:15:56 -05:00
renovate[bot]andWolfgang Walther ae7d4d2553 chore(deps): update codecov/codecov-action action to v6 2026-03-27 10:34:37 +00:00
renovate[bot]andWolfgang Walther 88afa05cf8 chore(deps): update korthout/backport-action action to v4.3.0 2026-03-27 08:45:54 +00:00
Taimoor ZaeemandWolfgang Walther 748ba7f02b chore: remove blank line in CHANGELOG.md
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-27 08:44:09 +00:00
Taimoor ZaeemandWolfgang Walther 0b665ecd10 chore: correct config name in CHANGELOG.md
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-27 08:44:09 +00:00
Taimoor ZaeemandSteve Chavez 1b402d16ad test(spec): move preference tests into separate modules
Towards #4751.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-26 12:36:47 -05:00
Taimoor ZaeemandSteve Chavez 135b77d12b test(io): rename jwt-role-claim-key*.config dump files
These files were added to cover the config dumping of
string comparison operators in `JSPath`. Renaming and
adding a comment in these to make it clear.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-24 11:11:43 -05:00
steve-chavez effcc080e5 chore: put ai policy first in CONTRIBUTING
Otherwise it's not that visible.
2026-03-23 10:10:58 -05:00
arturbent0andWolfgang Walther 5e504a77fa docs: clarify debounce time window in schema cache reloads 2026-03-23 12:04:01 +00:00
arturbent0andWolfgang Walther c9c6ebb53a docs: add new terms to spellcheck dictionary 2026-03-23 12:04:01 +00:00
Artur Bento de CarvalhoandWolfgang Walther 941c2f2d17 docs: explain schema cache reload behavior with NOTIFY debouncing
Add a "debouncing" section to explain how PostgREST handles multiple NOTIFY events.
2026-03-23 12:04:01 +00:00
Taimoor ZaeemandWolfgang Walther bf7dc485b3 chore: add changelog entry for v14.7
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-23 11:22:29 +00:00
steve-chavez 622c6d3f19 fix: not logging termination unix signals
Under container environments like ECS, it's hard to know when PostgREST
is being terminated.
2026-03-20 13:09:10 -05:00
Taimoor ZaeemandSteve Chavez 12ef63370b nix(test): add test suite for observability tests
- Create separate test suite for observability tests

- Create wrapper script `postgrest-test-observability`

- Add to CI and `postgrest-check`

- Move JWT cache tests under observability tests

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-19 10:38:42 -05:00
renovate[bot]andWolfgang Walther 796339172c chore(deps): update ubuntu:noble docker digest to 186072b 2026-03-19 09:16:11 +00:00
renovate[bot]andWolfgang Walther 238e9078bf chore(deps): update cachix/cachix-action action to v17 2026-03-18 21:19:20 +00:00
renovate[bot]andWolfgang Walther 1934179c64 chore(deps): update codecov/codecov-action action to v5.5.3 2026-03-18 20:02:06 +00:00
renovate[bot]andWolfgang Walther 88c365fb52 chore(deps): update actions/cache action to v5.0.4 2026-03-18 19:08:42 +00:00
Taimoor ZaeemandSteve Chavez 0ff8a1a37a refactor(auth): use wildcard type constructor import
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-18 13:09:27 -05:00
renovate[bot]andWolfgang Walther c448712e88 chore(deps): update ubuntu:noble docker digest to 0d39fcc 2026-03-17 10:19:37 +00:00
Taimoor ZaeemandSteve Chavez d4ba41bd43 nix(shell): bypass proxy variables using NO_PROXY=*
Tested using:

```sh
export HTTP_PROXY=http://localhost:7890
nix-shell
postgrest-test-io
```

Closes #4633.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-16 10:53:10 -05:00
Taimoor ZaeemandSteve Chavez 40a1fea916 refactor: move role insertion in claims to PreQuery.hs
Also add a comment on `AuthResult` type to explain that
we can also access the claims in the db using GUCs.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-16 10:24:24 -05:00
Andrei DziahelandSteve Chavez 1f54e2accc add: add "Vary" header to responses
Co-authored-by: Steve Chavez <stevechavezast@gmail.com>
2026-03-16 09:55:20 -05:00
renovate[bot]andWolfgang Walther d5df12c9c2 chore(deps): update actions/create-github-app-token action to v3 2026-03-14 18:04:36 +00:00
steve-chavezandWolfgang Walther af4c4157f5 nix: remove uneeded slocat
slocat was introduced to test pipeline mode (see https://github.com/PostgREST/postgrest/pull/2682),
but pipeline mode was not merged https://github.com/PostgREST/postgrest/pull/2707.

So it's really not needed on the loadtests.
2026-03-13 18:56:31 +00:00
renovate[bot]andWolfgang Walther 5ff271592c chore(deps): update ubuntu:noble docker digest to d1e2e92 2026-03-13 14:06:41 +00:00
renovate[bot]andWolfgang Walther bb4358c8bb chore(deps): update cachix/cachix-action digest to 3ba601f 2026-03-13 14:06:12 +00:00
renovate[bot]andWolfgang Walther 2ab2706c8e chore(deps): update actions/cache action to v5.0.3 2026-03-13 14:05:23 +00:00
renovate[bot]andWolfgang Walther b83fbb76a9 chore(deps): update haskell-actions/setup action to v2.10.3 2026-03-13 14:04:41 +00:00
renovate[bot]andWolfgang Walther fb54db143d chore(deps): update actions/checkout action to v6.0.2 2026-03-13 14:03:45 +00:00
renovate[bot]andWolfgang Walther 8ee8cdd7f8 chore(deps): update github artifact actions 2026-03-12 21:08:47 +00:00
Michał KłeczekandSteve Chavez bee2ea6fa6 test(io): add test_graceful_shutdown_waits_for_in_flight_request 2026-03-12 15:24:46 -05:00
Michał KłeczekandSteve Chavez a4aa58b4de refactor: move socket creation and management to App module
Right now listening sockets initialization, management and usage is split between App, AppState and Admin modules: they are created in AppState.init and remembered in AppState but used only in App and Admin.

It has several negative consequences:
- sockets are initialized even if not needed (eg. command line invocations like dump-config or dump-schema)
- it is impossible to start listening on a socket after initial schema cache load because it requires AppState

This change decouples listen socket management from AppState. Sockets are created only when needed (ie. not in command line tools invocation) and passed to admin application and to Warp by the App module.
2026-03-12 10:03:47 -05:00
Michał KłeczekandSteve Chavez e741c1bca7 refactor: provide AppState infrastructure to wait for schema cache load
This commit replaces ioRef based implementation of schema cache status tracking to MVar based, so that it is possible to wait for schema cache loading.

Waiting for schema cache loading is necessary to implement zero-downtime upgrades  with SO_REUSEPORT, where listening on a socket must wait for schema cache loading.
2026-03-12 09:04:56 -05:00
Wolfgang WaltherandAndrei Dziahel d5694672a4 nix(flake): add devShell
Adds a devShell to the flake for use with `nix develop`.

Co-authored-by: Andrei Dziahel <develop7@develop7.info>
2026-03-11 19:50:01 +00:00
Wolfgang Walther ca858f7544 docs: fix prometheus text format link
Reported by linkcheck.
2026-03-11 08:48:49 +01:00
renovate[bot]andWolfgang Walther cab60b9ee8 chore(deps): update docker/setup-buildx-action action to v4 2026-03-10 18:40:00 +00:00
steve-chavezandWolfgang Walther 0b7f0197fc chore: update license 2026-03-10 16:41:42 +00:00
Taimoor ZaeemandSteve Chavez 93db1a5a47 refactor: remove redundant parameters from actionResponse
Towards #4608.

We already have access to these parameters via `ApiRequest` type,
hence no need to pass them separately.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-10 09:55:42 -05:00
renovate[bot]andWolfgang Walther f657ac4e0b chore(deps): update docker/login-action action to v4 2026-03-10 14:15:16 +00:00
Taimoor ZaeemandSteve Chavez f5b9ed28db nix(chore): remove defunct config in loadtest.nix
`PGRST_JWT_CACHE_MAX_LIFETIME` is defunct since #4084 is merged.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-09 15:46:23 -05:00
renovate[bot]andWolfgang Walther 0eb2bdd8e1 chore(deps): update korthout/backport-action action to v4.2.0 2026-03-09 13:13:13 +00:00
Taimoor ZaeemandSteve Chavez 58368513ab chore: add changelog entry for v14.6
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-08 21:01:30 -05:00
Taimoor ZaeemandSteve Chavez 5abacba0d8 fix(error): leaking table and function names when calculating hint
Increase similarity score to 0.75 from 0.33 for table and functions
error hint.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-06 16:03:21 -05:00
Laurence Isla 2861b35f41 chore: update sponsors
* The list of sponsors is now ordered by total contribution value (highest first)
* Removed former backers from sponsor pages
2026-03-03 22:02:18 -05:00
steve-chavez 4e4ae3c7a2 chore: add AI policy in CONTRIBUTING
To preserve our scarce review capacity, we reserve the right to reject
seemingly AI generated low-value contributions.

---

Also delete the first paragraph since now it can be interpreted as
low-effort contributions.
2026-03-03 19:56:08 -05:00
steve-chavez 1c6d0816a5 chore: mention tests are required in CONTRIBUTING
Also remove the Haskell conventions subtitle since these are general
recommendations.
2026-03-02 17:57:01 -05:00
Laurence Isla 341e40196d chore: update sponsor 2026-03-02 17:36:04 -05:00
Michał KłeczekandSteve Chavez 8e1f0a0471 refactor: standardize query logging to follow code logging other observations
This commit addresses the following TODO in observationMessage function:
"TODO pending refactor: The logic for printing the query cannot be done here. Join the observationMessages function into observationLogger to avoid this mempty."

Changing observationMessage to observationMessages (that returns [Text] instead of Text) made it possible to address issue documented as TODO and harmonize query logging with other observations.
2026-03-02 15:46:17 -05:00
Michał KłeczekandSteve Chavez 2408cd332d refactor: get rid of redundant SchemaCacheSummaryObs
There is unnecessary coupling between observation messages and emited log entries. This causes schema loading logic to emit redundant events: SchemaCacheSummaryObs and SchemaCacheLoadedObs.

Logically - we want to emit a single event containing both summary and timing information. How it is logged is a different matter and should be decoupled.

This commit
* changes observationMessage function returning Text to observationMessages returning [Text] so that it is possible to return multiple (or zero) messages to log based on an observation event
* Removes SchemaCacheSummaryObs constructor from Observation type and adds summary text to SchemaCacheLoadedObs
2026-03-02 14:45:31 -05:00
Michał KłeczekandSteve Chavez 58a973e664 refactor: make Observation Logger and Listener modules more cohesive
Observation module exports observationMessage and isDbListenerBug functions. The first one is used only in Logger module and the second one - only in Listener.

This change moves observationMessage function to Logger module and isDbListenerBug function to Listener module, making all three modules more cohesive and lessening dependencies.
It also gives the compiler more opportunities for intra-module inlining and optimization.
2026-03-02 12:26:22 -05:00
Taimoor ZaeemandSteve Chavez a36963026e docs: add client error verbosity section to errors page
Add a new section "Client Error Verbosity". Move the details from
the config page to this section and link it.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-03-02 12:07:55 -05:00
Michał KłeczekandSteve Chavez 78f231c87a refactor: failure in schema cache load should not set listener delay
Listener has its own backoff logic independent of schema cache loading. It also uses listener delay value to identify its retries and to trigger schema cache reloads upon failures.
If schema cache reloading sets listener delay it might lead to some unexpected reloading loops.
That's especially important in set ups where listener connection points to master and the pool to replica(s).
2026-02-25 17:34:38 -05:00
Taimoor ZaeemandSteve Chavez 5992430e9f chore: update CHANGELOG.md
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-25 15:24:27 -05:00
Taimoor ZaeemandSteve Chavez 83dc082acf add: config client-error-verbosity to set error verbosity
Set error verbosity using this config. The verbosity can
be set to `verbose` or `minimal` for client error responses.

This only affects client side HTTP responses, server side logs
are not affected by this config.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-25 15:24:27 -05:00
Taimoor ZaeemandSteve Chavez 2edc44c352 refactor: remove ToJSON instance definition on error types
Towards #4088.

- Some of these instances are not used. Reduces number of lines
  significantly.

- Removing this gives us more flexibility for cases like conditional
  encoding based on some outside parameter, without needing to
  add the conditional at type level.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-23 12:10:51 -05:00
Taimoor ZaeemandSteve Chavez 02feaf087e refactor: remove commented out code from Auth/Jwt.hs
Some commented out code got left in the codebase in
5196823365.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-20 11:07:30 -05:00
Taimoor ZaeemandSteve Chavez c52fd3fd2b refactor: rename ApiRequestError type constructor to ApiRequestErr
Rename to keep the naming convention consistent with other
type constructor names.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-19 11:29:51 -05:00
Taimoor ZaeemandSteve Chavez 248b777117 refactor: move error data types to Error/Types.hs
The `Error.hs` module is getting bloated. This moves types
to another module for better separation of concerns.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-19 11:29:51 -05:00
steve-chavez 66fda76644 docs: missing link to prefer:tx in config page 2026-02-19 10:23:22 -05:00
dshukertjrandWolfgang Walther 23692ab8e7 docs: Update the Supabase logo to a correct one 2026-02-19 09:52:46 +00:00
Taimoor ZaeemandWolfgang Walther b977ffbb90 test(io): reset statement_timeout of anon role to correct value
Forgot to do this in bfb4f900e7.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-18 12:46:30 +00:00
Michał KłeczekandSteve Chavez 85a313a8cc test(spec): Move metrics state helpers from JwtCacheSpec to SpecHelpers
Refactoring: State validation helpers used in JwtCacheSpec moved to SpecHelper
to make them available in other Spec modules.
2026-02-17 17:55:16 -05:00
Taimoor ZaeemandSteve Chavez bfb4f900e7 test(io): increase statement_timeout value of anonymous role
It is too low which leaves a small window of values to
use when testing other features.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-17 11:35:12 -05:00
Taimoor ZaeemandSteve Chavez 055efe7abd refactor: move Proxy-Status header to error type class
Headers that apply to all errors should all be in one place.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-16 13:09:01 -05:00
Michał KłeczekandSteve Chavez ca96328142 refactor: Remove unnecessary lazy initialization of stateLogDebouncePoolTimeout
stateLogDebouncePoolTimeout is an MVar initialized on the first logging of PoolAcqTimeoutObs. The code in logWithDebounce has race condition that could lead to creation of multiple debouncers.

This change simplifies logic by getting rid of lazy initialization of debouncer.
2026-02-15 13:14:10 -05:00
Michał KłeczekandSteve Chavez d6816d8d2a refactor: Remove unnecessary SQL.UsageError parameter from PoolAcqTimeoutObs
PoolAcqTimeoutObs is always created with SQL.AcquisitionTimeoutUsageError so the parameter does not provide any useful information.
On the other hand, it complicates debouncing logic in Logger as logWithDebounce has to lazily create the debouncer (which is error prone and hence the logic has a race condition).

This change removes the parameter from PoolAcqTimeoutObs. It is a prerequisite to simplifying logWithDebounce.
2026-02-15 13:14:10 -05:00
Taimoor ZaeemandSteve Chavez c5d495006a chore: add changelog entry for v14.5
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-13 08:58:57 -05:00
steve-chavez abe414ad88 nix: print how to tail the server logs
For easier debugging during loadtests.

Also prefix the echo traces for the postgrest-with-pgrst command.
2026-02-12 20:18:08 -05:00
steve-chavez e95e815483 fix: don't hide async exceptions in logs
Fixes #4646. Using the repro on #4646, this now produces the log:

```
11/Feb/2026:09:40:08 -0500: Warp server error: stack overflow
```

When:
```
$ curl localhost:3000/
curl: (52) Empty reply from server
```
2026-02-12 19:43:50 -05:00
Taimoor ZaeemandWolfgang Walther c3500bd3af docs(install): update postgresql minimum supported version
PostgREST dropped support for PostgreSQL version 12 however,
it was not reflected in the docs.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-12 09:22:01 +00:00
Taimoor ZaeemandSteve Chavez 99984d328e docs: add section on docker resource constraints
Closes #4623.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-06 16:30:47 -05:00
Taimoor ZaeemandSteve Chavez aaaf7e96bd test(io): set client_min_messages to warning in SQL fixtures
To suppress unwanted "NOTICE: ..." messages which pollute the log.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-04 09:20:03 -05:00
Taimoor ZaeemandSteve Chavez 37fc0a211b test(io): refactor SQL fixtures
- Move privileges and grants to privileges.sql
- Move schema, tables and functions to schema.sql
- Move global database settings to database.sql

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-04 09:20:03 -05:00
Taimoor ZaeemandSteve Chavez c35eb7a6c2 test(io): move role fixtures to roles.sql
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-02 17:38:32 -05:00
Taimoor ZaeemandSteve Chavez 2bd4b07418 chore: add entry for a feature which was a breaking change
Adds the associated breaking change entry in CHANGELOG.md
for #4517.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-02-02 09:14:58 -05:00
steve-chavez 55cd5a23ff chore: move fixed entry to feature in changelog 2026-01-30 16:23:30 -05:00
Taimoor ZaeemandSteve Chavez 72a9d70f09 chore: add changelog entry for v14.4
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-01-30 08:46:47 -05:00
Laurence Isla d10c779fc6 perf: optimize count=exact when there's no limits, offsets or db-max-rows 2026-01-28 18:43:44 -05:00
Laurence Isla d031bb2df5 refactor: use a single function to get the page_total count 2026-01-28 18:43:44 -05:00
Michal KleczekandGitHub 73a465501e fix: listener running with exception masked after first failure 2026-01-28 17:13:59 -05:00
Wolfgang Walther 6f1fe69433 docs: add Wolfgang Walther as author
This was done in other places in
ffa393850a, but missed in the docs.
2026-01-28 09:59:42 +01:00
Wolfgang Walther 246239f42a docs: fix links
datrium.com doesn't exist anymore, while euronodes.com seems to only
fail SSL in CI.
2026-01-28 09:57:09 +01:00
Laurence Isla 1682677297 fix: filtering the returned representation whenn using or/and filters on mutations 2026-01-27 21:24:19 +00:00
Michał KłeczekandSteve Chavez 21b8c3458a add: Log pg version details of listener connection
Follow-up to #4617 adding more information to log entry produced upon successful listener connection establishement.
2026-01-27 15:44:05 -05:00
Michał KłeczekandSteve Chavez 5356f4e973 add: Log actual host and port of listener connection
Diagnosing problems with listener channel notifications not being handled properly by PostgREST connected to read replicas is difficult. Issues might be related to lost connections and listener not being connected to the right host after failover or database server restarts.
This patch adds logging of actual host:port used by libpq connection opened by the listener. It should make it easier to find out if PostgREST is connected to the right host.
2026-01-27 11:00:30 -05:00
Michal KleczekandGitHub 00c7cb1a22 fix: ensure Listener connections are released
retryingListen function potentially leaks database connections. This patch ensures the connections are released in case of listen/notify errors.
2026-01-26 16:26:24 -05:00
Taimoor ZaeemandSteve Chavez d75b1a7e96 docs: using default parameters on RPC with GET request
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-01-26 08:06:48 -05:00
Taimoor ZaeemandGitHub fe0386e9c4 add: string slicing operator for jwt-role-claim-key (#4603) 2026-01-24 05:40:21 -05:00
Taimoor ZaeemandWolfgang Walther 5a4c11c72a chore(ci): correct slightly misleading name of coverage workflow
The coverage workflow mentions that we run the tests against PG 15,
however that is incorrect and misleading. We actually run it against the
latest supported PostgreSQL version according to the nix scripts.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-01-18 14:53:52 +00:00
Laurence IslaandGitHub c15e908266 add(logs): Include a HINT when the LISTEN channel breaks due to a Postgres bug
The HINT shows a SQL command that solves the issue.
2026-01-18 18:47:53 +07:00
Taimoor ZaeemandSteve Chavez c18727ff43 refactor: move walkJSPath to Config/JSPath.hs module
The logic to evaluate `JSPath` belongs to `JSPath.hs` module.
Hence, moving this logic from `Auth/Jwt.hs` to here.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-01-14 22:34:08 +07:00
Michał KłeczekandSteve Chavez 886df84e87 test: Fix flakiness of test_second_request_for_non_existent_table_should_be_quick
Changed divider in assertion (response.elapsed.total_seconds() < first_duration / divider) to 2 (from 10).
2026-01-13 11:37:50 +07:00
renovate[bot]andWolfgang Walther 8f5fe3f46b chore(deps): update haskell-actions/setup action to v2.10.2 2026-01-11 17:31:31 +00:00
steve-chavez ffa393850a chore: update LICENSE 2026-01-11 11:59:31 +08:00
Kian-Meng AngandSteve Chavez f5c7cde9b7 docs: fix typos
Found via `codespell -S test -L notin,nam,plantext,cryto,corelation,inout,ot`
2026-01-06 05:24:51 +08:00
renovate[bot]andWolfgang Walther 0018c38f79 chore(deps): update haskell-actions/setup action to v2.10.1 2026-01-05 19:02:21 +00:00
Taimoor ZaeemandSteve Chavez 9042457326 chore: add changelog entry for v14.3
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-01-04 13:13:00 +08:00
Taimoor ZaeemandSteve Chavez e988fe49ed chore: move changelog entry to unreleased section
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-01-03 15:18:51 +08:00
Michał KłeczekandSteve Chavez e592d568c6 fix: Performance and high memory usage of relation hint calculation
* Calculation of hint message when requested relation is not present in schema cache requires creation of a FuzzySet (to use fuzzy search to find candidate tables). For schemas with many tables it is costly.
This patch introduces dbTablesFuzzyIndex in SchemaCache to memoize the FuzzySet creation.

* Additionally, because of FuzzySet large memory requirements, this patch introduces a limit of 500 relations per schema, above which FuzzySet is not created and hint calculation disabled.
2026-01-03 07:56:12 +08:00
Taimoor ZaeemandSteve Chavez 9ec5b030ce refactor: move content negotiation logic to Negotiate.hs
Moves the `negotiateContent` function to `Plan/Negotiate.hs`
module. This also adds comments to describe the current approach
we have for negotiation.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2026-01-02 06:32:17 +08:00
steve-chavez 76e0e1fbff nix: fix sleep default on postgrest-with-pgrst
The sleep was always applying since it had a default.
2026-01-02 06:11:07 +08:00
steve-chavez b8ca1bb04c nix: add new loadtest for errors
Also included it in CI.
2025-12-23 15:00:24 -05:00
steve-chavez 57c5ff5db3 nix: show wait time for fixture loading 2025-12-23 15:00:24 -05:00
steve-chavez 2b83dcc794 nix: sleep param for withPgrst 2025-12-23 15:00:24 -05:00
steve-chavez 7c3bccf4ba nix: show wait time on withPgrst 2025-12-23 15:00:24 -05:00
steve-chavez d2aa6c1b8a nix: timeout param for withPgrst 2025-12-23 15:00:24 -05:00
Taimoor ZaeemandSteve Chavez a329bca16b nix: add config file for hlint
Adds a config file for hlint containing arguments and
custom warnings.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-12-22 14:19:46 -05:00
388930336d refactor: fix definition of Ord instance for Routine type (#4577)
The `Ord` instance definition for type `Routine` had a logical
error when comparing two routines. The error did not affect any
end users. However, for correctness and completeness reasons, this
commit fixes the error.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Co-authored-by: Joel Jacobson <joel@compiler.org>
2025-12-22 08:34:29 +00:00
renovate[bot]andWolfgang Walther d06377a37b chore(deps): update docker/setup-buildx-action action to v3.12.0 2025-12-20 20:25:19 +00:00
Taimoor ZaeemandSteve Chavez eed018fe1d chore: add changelog entry for v14.2
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-12-19 12:29:57 -05:00
steve-chavez 013f078bc4 fix: inaccurate Server-Timing durations
The transaction duration was notably off, doing:

```
curl localhost:3000/rpc/sleep?seconds=5 -i
```

Shows `46.1` for the `transaction;dur`, with this fix we obtain
`5007.3`.

Fixes https://github.com/PostgREST/postgrest/issues/4522

This also fixes inaccurate "schema cache queried" logs,
see https://github.com/PostgREST/postgrest/issues/4551.
2025-12-18 17:35:18 -05:00
renovate[bot]andWolfgang Walther 2bcd336400 chore(deps): update haskell-actions/setup action to v2.9.1 2025-12-17 18:01:04 +00:00
steve-chavez 8f34afd66e nix: jwt loadtest chain commands and split rsa gen
Chaining the target generation command after the build is done ensures
that if a build takes too long, the dynamic target generation for vegeta
won't cause 401 Unauthorized errors due to already expired JWTs.

Also split the rsa materials writing to another python program for
easier maintenance.
2025-12-16 21:15:28 -05:00
steve-chavez 4cdc4c4861 nix: show build time on postgrest-with-pgrst 2025-12-16 21:15:28 -05:00
steve-chavez 050554e2bb nix: fail jwt loadtest in case of 401
Also print last unauthorized body
2025-12-16 21:15:28 -05:00
Taimoor ZaeemandSteve Chavez b3b7bd9327 docs: add link on how to start docker service
Closes #4552.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-12-16 11:34:04 -05:00
renovate[bot]andWolfgang Walther dac8a5fb86 chore(deps): update haskell-actions/setup action to v2.9.0 2025-12-15 21:01:22 +00:00
Joel JakobssonandGitHub fd6a3bdccf fix: hasSingleUnnamedParam incorrectly matching named parameters
The hasSingleUnnamedParam function was only checking the parameter type
but not whether the parameter actually had no name. This caused functions
with a single NAMED parameter (e.g., `foo(data json)`) to incorrectly
match the single-param fallback mode.

The result was a confusing PostgreSQL error 42883 "function does not exist"
instead of a clean PGRST202 error explaining that no matching function
was found.

Added ppName == mempty check so functions with named parameters don't
incorrectly match the single-param fallback.
2025-12-15 15:34:48 -05:00
renovate[bot]andWolfgang Walther 60c70c5b37 chore(deps): update korthout/backport-action action to v4.0.1 2025-12-15 20:18:02 +00:00
renovate[bot]andWolfgang Walther c9f9b93ad7 chore(deps): update github artifact actions 2025-12-13 14:36:30 +00:00
renovate[bot]andWolfgang Walther e39b91c27b chore(deps): update korthout/backport-action action to v4 2025-12-13 14:36:12 +00:00
renovate[bot]andWolfgang Walther d694ad90e0 chore(deps): update actions/cache action to v5.0.1 2025-12-13 11:31:32 +00:00
steve-chavez 99b8246d6e nix: add --method for postgrest-loadtest
Currently the `postgrest-loadtest -k jwt-*` loadtests use the OPTIONS method,
which is good for isolating the perf changes for JWT auth.

But using GET is more accurate for actual perf improvements,
since it actually hits the database.

This adds a method argument for postgrest-loadtest, so GET can be used like:

```
postgrest-loadtest -k jwt-hs --method GET
```

CI is not changed, this is mostly useful for manually checking variance.
2025-12-12 14:46:46 -05:00
steve-chavez 90bf1762ae nix: rename genTargetsHs to genTargets
Previous name wasn't accurate as RSA is used as well.
2025-12-12 14:46:46 -05:00
renovate[bot]andWolfgang Walther c9c975ce49 chore(deps): update actions/cache action to v5 2025-12-12 08:29:32 +00:00
Taimoor ZaeemandSteve Chavez 29c2aa015f refactor: create function to handle response preferences
Centralizes handling of preferences that are used to
create `Preference-Applied` header which is returned on
responses.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-12-11 17:55:45 -05:00
Taimoor ZaeemandSteve Chavez abad47ff1b add: log error when db-schemas config contain restricted schemas
We don't allow accessing schemas "pg_catalog" and "information_schema".
Instead of silently failing requests on these schemas, this now adds
logging an error on startup when these schemas are found in `db-schemas`
config.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-12-11 14:28:08 -05:00
Laurence IslaandSteve Chavez c232e97f80 chore(changelog): move incorrectly placed fixes 2025-12-10 20:00:31 -05:00
Laurence Isla ab9b00ce88 refactor: use only Lazy.ByteString to calculate the response body length 2025-12-10 14:18:19 +00:00
Laurence Isla 9200a98087 fix: add missing Content-Length to empty HTTP 201 responses
For when a preference other than return=representation is requested.
2025-12-10 14:18:19 +00:00
Laurence Isla 3b27b2220d fix: regression that truncates error message when offset is out of bounds
- Happens when offset > the number of rows and when "Prefer: count=exact" header is sent
- Regression introduced in commit 57ef998
2025-12-10 14:18:19 +00:00
renovate[bot]andWolfgang Walther 6573953ebd chore(deps): update codecov/codecov-action action to v5.5.2 2025-12-10 12:24:43 +00:00
Taimoor ZaeemandSteve Chavez 0fa2f569a7 fix: misleading logs on unsupported postgresql versions
Postgrest fails on unsupported pg versions. However before killing
the thread, it continues to print a few more log messages which
were misleading. This commit fixes this by making sure that the
no log message should be printed after the unsupported pg version
observation and kill the thread immediately.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-12-09 13:46:34 -05:00
renovate[bot]andWolfgang Walther 24a27d17d0 chore(deps): update actions/create-github-app-token action to v2.2.1 2025-12-06 11:00:06 +00:00
renovate[bot]andGitHub 1724e4551d chore(deps): update actions/checkout digest to 8e8c483 (main) (#4511) 2025-12-03 19:32:01 +00:00
renovate[bot]andWolfgang Walther 457f20f962 chore(deps): update actions/checkout action to v6.0.1 2025-12-03 19:30:42 +00:00
renovate[bot]andWolfgang Walther 70240b0d08 chore(deps): update actions/create-github-app-token action to v2.2.0 2025-12-03 19:29:54 +00:00
Taimoor ZaeemandSteve Chavez 50eec773bf test(io): fix freeport function to prevent failures
Sometimes, a healthcheck related test fails as occurred in
https://github.com/PostgREST/postgrest/actions/runs/19771357953/job/56655949002.
This happens due to freeport function accidently picking up a used port.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-29 18:17:00 -05:00
Taimoor ZaeemandSteve Chavez 35de13eebd test(io): move fixtures to fixtures/ directory
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-27 17:08:03 -05:00
Taimoor ZaeemandSteve Chavez 3baa6b0063 test(io): rename fixtures.sql to load.sql
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-27 17:08:03 -05:00
Andrei DziahelandGitHub 6d462f762e nix: override PGRST_CMD for postgrest-loadtest from env
Enables `PGRST_CMD=postgrest-profiled-run postgrest-loadtest` running
loadtest against profiled executable, which wasn't available before
2025-11-25 19:46:23 +00:00
Taimoor ZaeemandGitHub 802cce9a28 nix(shell): remove postgrest/ directory prefix when running pg (#4502)
When running postgres from nix-shell, nix creates a directory
structure like `postgrest/postgrest-with-pg-17-XXX` in the `/tmp`
directory. This commit removes the extra `postgrest/` prefix to
shorten length of absolute path length of filenames.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-25 15:54:25 +00:00
Marek BriegerandGitHub ee334cbcf8 chore: improve grammar in README.md 2025-11-25 12:20:26 +00:00
Taimoor ZaeemandSteve Chavez c561a3749c test(io): move authentication related tests to test_auth.py
The `test_io.py` module is too bloated (2100+ lines). To
logically group related tests, as a first step, this commit
separates authentication related IO tests into `test_auth.py`
module.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-21 13:21:15 -05:00
renovate[bot]andWolfgang Walther f8b56736d7 chore(deps): update actions/checkout digest to 1af3b93 2025-11-21 17:29:18 +00:00
renovate[bot]andWolfgang Walther 7bcd46b145 chore(deps): update actions/checkout action to v6 2025-11-21 17:29:09 +00:00
Taimoor ZaeemandSteve Chavez d905592115 docs: add asterisk character to reserved characters list
The `*` character is a reserved character in the postgrest
url grammar but we didn't mention it in the docs. Now added.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-20 16:20:28 -05:00
Taimoor ZaeemandSteve Chavez 4b8dec8901 docs: clarify postgrest reserved characters in url grammar
Fixes a typo in the documentation where it said that `()`
is a reserved character, when we actually meant to say that
`(` and `)` are reserved characters.

Related discussion in #4254.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-20 16:20:28 -05:00
Taimoor ZaeemandSteve Chavez 379eaec8e0 test(io): remove stale jwt cache test
Removes a test related to jwt cache which is stale
since #4084.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-19 11:17:18 -05:00
Andrei DziahelandWolfgang Walther ae2d650516 fix: hashtables <1.4.1 fail to build with GCC 15
Introduces hashtables >= 1.4.1 requirement
2025-11-18 19:26:16 +00:00
Wolfgang Walther 4a83e79923 ci/test: remove macos x86 flake check
See previous commit.
2025-11-18 11:05:23 +01:00
Wolfgang Walther eb00c1991c ci: remove macos x86 builds
The x86 GitHub runner will not be available anymore, soon.

We might be able to re-introduce this, once we can build a static
executable via Nix on darwin, too.
2025-11-18 11:01:05 +01:00
Taimoor ZaeemandWolfgang Walther 462798dc2d nix: shorten postgrest-with-postgresql-xx scripts
Renames these scripts to `postgrest-with-pg-xx`. The renaming
helps reduce the length of temporary filenames. This is needed
to ensure that socket file names remain under the maximum
allowed length of 107 chars.

Closes #4461.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-18 09:40:16 +00:00
renovate[bot]andWolfgang Walther 1efd5781f8 chore(deps): update actions/checkout action to v5.0.1 2025-11-18 09:38:41 +00:00
Taimoor ZaeemandWolfgang Walther 08c6deba12 docs: update server-host config in docker-compose example
The docker-compose example did not work with multiple containers
when PGRST_SERVER_HOST is set to `localhost`. This updates the
value to `0.0.0.0` allowing other containers to connect.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-18 09:36:37 +00:00
renovate[bot]andWolfgang Walther 531848366d chore(deps): update ubuntu:noble docker digest to c35e29c 2025-11-15 13:52:00 +00:00
Taimoor ZaeemandSteve Chavez 91abcd49e1 test(io): move resource embedding tests to test_io.py
- Adds fixtures to `test/io/fixtures.sql` to test resource
  embedding related queries.

- Moves the resource embedding related tests that no longer
  require big schema from `test_big_schema.py` to `test_io.py`.

Closes #4417.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-11 13:43:10 -05:00
Jens TroegerandSteve Chavez 770c404ebe docs: add another project to the list of Example Apps 2025-11-11 13:25:55 -05:00
Taimoor ZaeemandSteve Chavez 99217433b1 chore(cabal): remove unused haskell dependencies
This should reduce setup time for build process.

- cache: introduced in #2928, defunct since #4084
- clock: introduced in #2928, defunct since #4084
- heredoc: introduced in #714, defunct since #4390
- iproute: introduced in #3560, defunct since #4288

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-09 13:49:18 -05:00
Taimoor ZaeemandWolfgang Walther bccf7ac97a Revert "chore: remove unused haskell dependencies"
This reverts commit b6bb8382e7.
2025-11-06 08:11:44 +00:00
Taimoor ZaeemandWolfgang Walther ed8be34b20 chore: add changelog entry for v14.1
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-06 08:11:32 +00:00
Taimoor ZaeemandSteve Chavez b6bb8382e7 chore: remove unused haskell dependencies
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-04 14:12:08 -05:00
renovate[bot]andWolfgang Walther 893e0061a3 chore(deps): update actions/checkout digest to 71cf226 2025-11-03 20:47:50 +00:00
Taimoor ZaeemandSteve Chavez 58efc2680e fix: server-host !6 incorrectly binds to IPv4 address
Updates streaming-commons to version 0.2.3.1. This resolves #3202.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-02 12:24:14 -05:00
Taimoor ZaeemandWolfgang Walther 8180905890 chore(changelog): fix typo in changelog entry
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-11-01 09:00:31 +00:00
Taimoor ZaeemandSteve Chavez db2be093b5 refactor: move dumpQi to Config.hs and clarify quoteQi
The `dumpQi` function is only used in the `Config.hs` module, so
it is moved there. This also adds a doctest for `quoteQi` and
clarifies its usage.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-10-30 12:26:29 -05:00
Taimoor ZaeemandSteve Chavez a688878236 fix: db-pre-config function failing with pg reserved words
When db-pre-config is accidentally set to a pg reserved word
like "true", it fails with a confusing error. The function
names should be properly quoted to avoid such errors. This commit
resolves this by quoting the pre-config function name.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-10-29 16:54:24 -05:00
Michał KłeczekandSteve Chavez 75d4131aa6 chore: Add commit structuring guidelines to CONTRIBUTING.md 2025-10-28 10:02:56 -05:00
Taimoor ZaeemandSteve Chavez 66f84c5903 refactor: move escapeIdent function to Identifiers.hs
Moves the functions `escapeIdent` and `trimNullChars` to
SchemaCache/Identifiers.hs module.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-10-28 09:47:37 -05:00
Taimoor ZaeemandSteve Chavez 5d9b169380 refactor: sort exports of Identifiers.hs and SqlFragments.hs
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-10-28 09:47:37 -05:00
Taimoor ZaeemandSteve Chavez eb908c696f chore(changelog): update versioning scheme description
The changelog description mentioned that we follow semantic
versioning but from now on we don't. Hence updated the description
to reflect new versioning policy.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-10-27 10:00:40 -05:00
renovate[bot]andWolfgang Walther 21edcc86ce chore(deps): update github artifact actions 2025-10-25 08:23:40 +00:00
Wolfgang Walther 6f46233ad7 ci: fix tag job with new release workflow
A single component version is the development version, everything with
more components is not. Thus, we only need to check for a single dot.
2025-10-25 10:20:32 +02:00
steve-chavez 018803473c bump version to 15 2025-10-24 17:39:04 -05:00
298 changed files with 10319 additions and 8989 deletions
-42
View File
@@ -1,42 +0,0 @@
freebsd_instance:
image_family: freebsd-14-3
build_task:
# Don't change this name without adjusting .github/workflows/build.yaml
name: Build FreeBSD (Stack)
install_script: pkg install -y postgresql16-client hs-stack git
only_if: |
$CIRRUS_TAG != '' || $CIRRUS_BRANCH == 'main' || $CIRRUS_BRANCH =~ 'v*' ||
changesInclude(
'.github/workflows/build.yaml',
'.github/actions/artifact-from-cirrus/**',
'.cirrus.yml',
'postgrest.cabal',
'stack.yaml*',
'**.hs'
)
stack_cache:
folders: /.stack
fingerprint_script:
- echo $CIRRUS_OS
- stack --version
- md5sum postgrest.cabal
- md5sum stack.yaml.lock
stack_work_cache:
folders: .stack-work
fingerprint_script:
- echo $CIRRUS_OS
- stack --version
- md5sum postgrest.cabal
- md5sum stack.yaml.lock
- find main src -type f -iname '*.hs' -exec md5sum "{}" +
build_script: |
stack build -j 1 --local-bin-path . --copy-bins
strip postgrest
bin_artifacts:
path: postgrest
-5
View File
@@ -1,5 +0,0 @@
# TODO: Remove this once a new actionlint release has been cut
# and made its way to us through nixpkgs.
self-hosted-runner:
labels:
- ubuntu-24.04-arm
@@ -1,119 +0,0 @@
name: Artifact from Cirrus
description: Waits for a specific Cirrus CI run to complete, then downloads the artifact and uploads it to the current workflow. This will silently succeed if Cirrus CI did not schedule a task within 2 minutes.
inputs:
download:
description: Name of Artifact to download from Cirrus CI
required: true
task:
description: Name of Cirrus Task
required: true
token:
description: GitHub Token
required: true
upload:
description: Name of Artifact to upload on GitHub Actions
required: true
runs:
using: composite
steps:
- shell: bash
run: echo "GH_TOKEN=${{ inputs.token }}" >> "$GITHUB_ENV"
- name: Wait for Check Suite to be created
id: check-suite
env:
# GITHUB_SHA does weird things for pull request, so we roll our own:
COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
shell: bash
run: |
get_check_runs_url() {
gh api "repos/{owner}/{repo}/commits/${COMMIT}/check-suites" \
| jq -r '.check_suites[] | select(.app.slug == "cirrus-ci") | .check_runs_url'
}
for _ in $(seq 1 12); do
check_runs_url="$(get_check_runs_url)"
if [ -z "$check_runs_url" ]; then
echo "Cirrus CI task has not started, yet. Waiting..."
sleep 10
else
echo "check_runs_url=$check_runs_url" >> "$GITHUB_OUTPUT"
exit 0
fi
done
>&2 echo "Cirrus CI check suite not found. Is Cirrus CI enabled for this repo?"
- name: Find task by name
id: find-task
if: steps.check-suite.outputs.check_runs_url
shell: bash
run: |
get_number_of_tasks() {
gh api "${{ steps.check-suite.outputs.check_runs_url }}" \
| jq -r '.check_runs | map(select(.name == "${{ inputs.task }}")) | length'
}
tasks="$(get_number_of_tasks)"
case "$tasks" in
0)
echo "Task not found, assuming it's skipped intentionally..."
exit 0
;;
1)
echo "task_found=1" >> "$GITHUB_OUTPUT"
exit 0
;;
*)
>&2 echo "More than 1 task with the same name found. Don't know what to do..."
exit 1
;;
esac
- name: Wait for Cirrus CI to complete task
if: steps.find-task.outputs.task_found
shell: bash
run: |
get_conclusion() {
gh api "${{ steps.check-suite.outputs.check_runs_url }}" \
| jq -r '.check_runs[] | select(.name == "${{ inputs.task }}" and .status == "completed") | .conclusion'
}
while true; do
conclusion="$(get_conclusion)"
if [ -z "$conclusion" ]; then
echo "Cirrus CI task has not completed, yet. Waiting..."
sleep 30
else
if [ "$conclusion" == "success" ]; then
break
else
exit 1
fi
fi
done
- name: Download artifact from Cirrus CI
if: steps.find-task.outputs.task_found
id: download
shell: bash
run: |
get_external_id() {
gh api "${{ steps.check-suite.outputs.check_runs_url }}" \
| jq -er '.check_runs[] | select(.name == "${{ inputs.task }}") | .external_id'
}
archive="$(mktemp)"
artifacts="$(mktemp -d)"
until curl --no-progress-meter --fail -o "${archive}" \
"https://api.cirrus-ci.com/v1/artifact/task/$(get_external_id)/${{ inputs.download }}.zip"
do
# This happens when a tag is pushed on the same commit. In this case the
# job is immediately marked as "completed" for us, so we end up here after a few
# seconds - but the actual Cirrus CI task is still running and didn't produce its artifact, yet.
echo "Artifact not found on Cirrus CI, yet. Waiting..."
sleep 30
done
unzip "${archive}" -d "${artifacts}"
echo "artifacts=${artifacts}" >> "$GITHUB_OUTPUT"
- name: Save artifact to GitHub Actions
if: steps.find-task.outputs.task_found
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: ${{ inputs.upload }}
path: ${{ steps.download.outputs.artifacts }}
if-no-files-found: error
+6 -7
View File
@@ -8,7 +8,6 @@ inputs:
required: true required: true
save-prs: save-prs:
description: Whether to additionally store the cache in a pull request, too. Should only be used for very small caches. description: Whether to additionally store the cache in a pull request, too. Should only be used for very small caches.
type: boolean
prefix: prefix:
description: Cache key prefix to be used in both primary key and restore-keys. description: Cache key prefix to be used in both primary key and restore-keys.
required: true required: true
@@ -19,17 +18,17 @@ inputs:
runs: runs:
using: composite using: composite
steps: steps:
- uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1 - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
if: ${{ startsWith(github.ref, 'refs/heads/') || (inputs.save-prs && startsWith(github.ref, 'refs/pull/')) }} if: ${{ startsWith(github.ref, 'refs/heads/') || (inputs.save-prs && startsWith(github.ref, 'refs/pull/')) }}
with: with:
path: ${{ inputs.path }} path: ${{ inputs.path }}
key: ${{ runner.os }}-${{ inputs.prefix }}-${{ inputs.suffix }} key: ${{ runner.os }}-${{ runner.arch }}-${{ inputs.prefix }}-${{ inputs.suffix }}
restore-keys: | restore-keys: |
${{ runner.os }}-${{ inputs.prefix }}- ${{ runner.os }}-${{ runner.arch }}-${{ inputs.prefix }}-
- uses: actions/cache/restore@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1 - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
if: ${{ !startsWith(github.ref, 'refs/heads/') && !(inputs.save-prs && startsWith(github.ref, 'refs/pull/')) }} if: ${{ !startsWith(github.ref, 'refs/heads/') && !(inputs.save-prs && startsWith(github.ref, 'refs/pull/')) }}
with: with:
path: ${{ inputs.path }} path: ${{ inputs.path }}
key: ${{ runner.os }}-${{ inputs.prefix }}-${{ inputs.suffix }} key: ${{ runner.os }}-${{ runner.arch }}-${{ inputs.prefix }}-${{ inputs.suffix }}
restore-keys: | restore-keys: |
${{ runner.os }}-${{ inputs.prefix }}- ${{ runner.os }}-${{ runner.arch }}-${{ inputs.prefix }}-
+35
View File
@@ -0,0 +1,35 @@
name: Run anywhere
description: Runs the same code either in a VM or on the bare machine
inputs:
vm:
description: Which VM to run on.
envs:
description: List of relevant environment variables, which might need to be copied into the VM.
prepare:
description: Code to run in a prepare step, e.g. installing dependencies.
run:
description: Code to run as the main action.
required: true
runs:
using: composite
steps:
- if: ${{ inputs.vm == 'freebsd' }}
uses: vmactions/freebsd-vm@83b151f58c6047089f4c80eb5ba2039d158ce093 # v1.5.3
with:
envs: ${{ inputs.envs }}
prepare: ${{ inputs.prepare }}
# Work around https://github.com/vmactions/freebsd-vm/issues/59
run: |
pw user add -n action -m
su action -c '${{ inputs.run }}'
- if: ${{ inputs.vm == '' }}
name: Prepare
shell: ${{ runner.os == 'Windows' && 'pwsh' || 'bash' }}
run: ${{ inputs.prepare }}
- if: ${{ inputs.vm == '' }}
name: Run
shell: ${{ runner.os == 'Windows' && 'pwsh' || 'bash' }}
run: ${{ inputs.run }}
+2 -2
View File
@@ -11,12 +11,12 @@ inputs:
runs: runs:
using: composite using: composite
steps: steps:
- uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34 - uses: nixbuild/nix-quick-install-action@9f63be77f412a248c9d9a65a4c82cf066cdf8f0c # v35
with: with:
nix_conf: |- nix_conf: |-
always-allow-substitutes = true always-allow-substitutes = true
max-jobs = auto max-jobs = auto
- uses: cachix/cachix-action@0fc020193b5a1fa3ac4575aa3a7d3aa6a35435ad # v16 - uses: cachix/cachix-action@5f2d7c5294214f71b873db4b969586b980625e71 # v17
with: with:
name: postgrest name: postgrest
authToken: ${{ inputs.authToken }} authToken: ${{ inputs.authToken }}
+3
View File
@@ -4,6 +4,9 @@ codecov:
comment: false comment: false
github_checks:
annotations: true
coverage: coverage:
status: status:
project: project:
-41
View File
@@ -13,9 +13,6 @@
}, },
"packageRules": [ "packageRules": [
{ {
"matchBaseBranches": [
"/^v[0-9]+/"
],
"matchManagers": [ "matchManagers": [
"haskell-cabal" "haskell-cabal"
], ],
@@ -26,44 +23,6 @@
"/^v[0-9]+/" "/^v[0-9]+/"
], ],
"groupName": "all dependencies" "groupName": "all dependencies"
},
{
"matchManagers": [
"haskell-cabal"
],
"matchPackageNames": [
"base",
"bytestring",
"containers",
"directory",
"mtl",
"parsec",
"process",
"text"
],
"groupName": "GHC dependencies"
},
{
"matchManagers": [
"haskell-cabal"
],
"matchPackageNames": [
"hasql",
"hasql-dynamic-statements",
"hasql-notifications",
"hasql-transaction",
"hasql-pool"
],
"groupName": "hasql"
},
{
"matchManagers": [
"haskell-cabal"
],
"matchPackageNames": [
"fuzzyset"
],
"allowedVersions": "<0.3"
} }
] ]
} }
+6 -5
View File
@@ -9,7 +9,7 @@ on:
jobs: jobs:
backport: backport:
name: Backport name: Backport
runs-on: ubuntu-24.04 runs-on: ubuntu-slim
# It triggers only when PR is already merged on either: # It triggers only when PR is already merged on either:
# #
# - The merge event itself (action != labeled) or # - The merge event itself (action != labeled) or
@@ -28,9 +28,9 @@ jobs:
# This actions creates the github token using the postgrest app secrets # This actions creates the github token using the postgrest app secrets
- name: Create Github App Token - name: Create Github App Token
id: app-token id: app-token
uses: actions/create-github-app-token@29824e69f54612133e76f7eaac726eef6c875baf # v2.2.1 uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with: with:
app-id: ${{ vars.POSTGREST_CI_APP_ID }} client-id: ${{ vars.POSTGREST_CI_APP_ID }}
private-key: ${{ secrets.POSTGREST_CI_PRIVATE_KEY }} private-key: ${{ secrets.POSTGREST_CI_PRIVATE_KEY }}
permission-contents: write permission-contents: write
permission-pull-requests: write permission-pull-requests: write
@@ -38,14 +38,15 @@ jobs:
# This is required for backport action to cherry-pick the PR # This is required for backport action to cherry-pick the PR
- name: Fetch PR ref - name: Fetch PR ref
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with: with:
allow-unsafe-pr-checkout: true
ref: ${{ github.event.pull_request.head.sha }} ref: ${{ github.event.pull_request.head.sha }}
token: ${{ steps.app-token.outputs.token }} token: ${{ steps.app-token.outputs.token }}
# Backport action that creates the PR with given settings # Backport action that creates the PR with given settings
- name: Create backport PR - name: Create backport PR
uses: korthout/backport-action@c656f5d5851037b2b38fb5db2691a03fa229e3b2 # v4.0.1 uses: korthout/backport-action@2e830a1d0b8269505846ddd407a70876913ad1f8 # v4.6
with: with:
github_token: ${{ steps.app-token.outputs.token }} github_token: ${{ steps.app-token.outputs.token }}
pull_description: 'Backport for #${pull_number}.' pull_description: 'Backport for #${pull_number}.'
+67 -69
View File
@@ -16,6 +16,7 @@ on:
- .github/* - .github/*
- '*.nix' - '*.nix'
- nix/** - nix/**
- flake.lock
- .cirrus.yml - .cirrus.yml
- cabal.project* - cabal.project*
- postgrest.cabal - postgrest.cabal
@@ -30,10 +31,20 @@ concurrency:
jobs: jobs:
static: static:
name: Nix - Linux x86-64 static strategy:
runs-on: ubuntu-24.04 fail-fast: false
matrix:
include:
- name: Linux aarch64
runs-on: ubuntu-24.04-arm
artifact: aarch64
- name: Linux x86-64
runs-on: ubuntu-24.04
artifact: x86-64
name: Nix - ${{ matrix.name }} static
runs-on: ${{ matrix.runs-on }}
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment - name: Setup Nix Environment
uses: ./.github/actions/setup-nix uses: ./.github/actions/setup-nix
with: with:
@@ -42,44 +53,44 @@ jobs:
- name: Build static executable - name: Build static executable
run: nix-build -A postgrestStatic -A postgrestStatic.tests run: nix-build -A postgrestStatic -A postgrestStatic.tests
- name: Save built executable as artifact - name: Save built executable as artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with: with:
name: postgrest-linux-static-x86-64 name: postgrest-linux-static-${{ matrix.artifact }}
path: result/bin/postgrest path: result/bin/postgrest
if-no-files-found: error if-no-files-found: error
- name: Build Docker image - name: Build Docker image
run: nix-build -A docker.image --out-link postgrest-docker.tar.gz run: nix-build -A docker.image --out-link postgrest-docker-${{ matrix.artifact }}.tar.gz
- name: Save built Docker image as artifact - name: Save built Docker image as artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with: with:
name: postgrest-docker-x86-64 name: postgrest-docker-${{ matrix.artifact }}
path: postgrest-docker.tar.gz path: postgrest-docker-${{ matrix.artifact }}.tar.gz
if-no-files-found: error if-no-files-found: error
- name: Test static executable with NixOS' VM test
# GHA's ARM runner does not support KVM
if: runner.arch == 'X64'
run: nix-build -A nixpkgs-nixos-test
macos: macos:
name: Nix - MacOS name: Nix - MacOS
runs-on: macos-15 runs-on: macos-26
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment - name: Setup Nix Environment
uses: ./.github/actions/setup-nix uses: ./.github/actions/setup-nix
with: with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
- name: Install gnu sed - name: Install nix-build-uncached
run: brew install gnu-sed run: nix-env -f default.nix -iA nix-build-uncached
- name: Build everything - name: Build everything (default.nix)
run: | run: nix-build-uncached
# The --dry-run will give us a list of derivations to download from cachix and
# derivations to build. We only take those that would have to be built and then build - name: Build everything (shell.nix)
# those explicitly. This has the advantage that pure verification will not include run: nix-build-uncached shell.nix
# a download anymore, making it much faster. If something needs to be built, only
# the dependencies required to do so will be downloaded, but not everything.
nix-build --dry-run 2>&1 \
| gsed -e '1,/derivations will be built:$/d' -e '/paths will be fetched/Q' \
| xargs nix-build
stack: stack:
@@ -87,66 +98,66 @@ jobs:
fail-fast: false fail-fast: false
matrix: matrix:
include: include:
- name: Linux aarch64 - name: FreeBSD x86-64
runs-on: ubuntu-24.04-arm runs-on: ubuntu-24.04
cache: | vm: freebsd
~/.stack/pantry artifact: postgrest-freebsd-x86-64
~/.stack/snapshots deps: pkg install -y git postgresql16-client hs-stack
~/.stack/stack.sqlite3
artifact: postgrest-ubuntu-aarch64
deps: sudo apt-get update && sudo apt-get install libpq-dev
- name: MacOS aarch64 - name: MacOS aarch64
runs-on: macos-14 runs-on: macos-14
cache: |
~/.stack/pantry
~/.stack/snapshots
~/.stack/stack.sqlite3
artifact: postgrest-macos-aarch64 artifact: postgrest-macos-aarch64
deps: brew link --force libpq deps: brew link --force libpq
- name: MacOS x86-64
runs-on: macos-15-intel
artifact: postgrest-macos-x86-64
deps: brew link --force libpq
- name: Windows - name: Windows
runs-on: windows-2022 runs-on: windows-2022
cache: |
C:\sr\pantry
C:\sr\snapshots
C:\sr\stack.sqlite3
deps: Add-Content $env:GITHUB_PATH $env:PGBIN deps: Add-Content $env:GITHUB_PATH $env:PGBIN
artifact: postgrest-windows-x86-64 artifact: postgrest-windows-x86-64
name: Stack - ${{ matrix.name }} name: Stack - ${{ matrix.name }}
runs-on: ${{ matrix.runs-on }} runs-on: ${{ matrix.runs-on }}
env:
# Putting .stack in the working directory helps with moving this in and out of the FreeBSD VM.
STACK_ROOT: ${{ github.workspace }}/.stack
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: haskell-actions/setup@0512451d82f3ca8c147db62e30464e7c4ca63d30 # v2.9.1 - if: ${{ !matrix.vm }}
uses: haskell-actions/setup@6037f33647c3f17758a2356c80fc4a53d7e0685d # v2.12.0
with: with:
# This must match the version in stack.yaml's resolver # This must match the version in stack.yaml's resolver
ghc-version: 9.6.7 ghc-version: 9.10.3
enable-stack: true enable-stack: true
stack-no-global: true stack-no-global: true
stack-setup-ghc: true stack-setup-ghc: true
- name: Cache ~/.stack - name: Cache .stack
uses: ./.github/actions/cache-on-main uses: ./.github/actions/cache-on-main
with: with:
path: ${{ matrix.cache }} path: .stack
prefix: stack prefix: ${{ matrix.vm }}${{ matrix.vm && '-' }}stack
suffix: ${{ hashFiles('postgrest.cabal', 'stack.yaml.lock') }} suffix: ${{ hashFiles('postgrest.cabal', 'stack.yaml.lock') }}
- name: Cache .stack-work - name: Cache .stack-work
uses: ./.github/actions/cache-on-main uses: ./.github/actions/cache-on-main
with: with:
path: .stack-work path: .stack-work
save-prs: true save-prs: true
prefix: stack-work-${{ hashFiles('postgrest.cabal', 'stack.yaml.lock') }} prefix: ${{ matrix.vm }}${{ matrix.vm && '-' }}stack-work-${{ hashFiles('postgrest.cabal', 'stack.yaml.lock') }}
suffix: ${{ hashFiles('main/**/*.hs', 'src/**/*.hs') }} suffix: ${{ hashFiles('main/**/*.hs', 'src/**/*.hs') }}
- name: Install dependencies
if: matrix.deps
run: ${{ matrix.deps }}
- name: Build with Stack - name: Build with Stack
run: stack build --lock-file error-on-write --local-bin-path result --copy-bins uses: ./.github/actions/run-anywhere
- name: Strip Executable with:
run: strip result/postgrest* vm: ${{ matrix.vm }}
envs: STACK_ROOT
prepare: ${{ matrix.deps }}
run: |
stack build --lock-file error-on-write --local-bin-path result --copy-bins
strip result/postgrest*
- name: Save built executable as artifact - name: Save built executable as artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with: with:
name: ${{ matrix.artifact }} name: ${{ matrix.artifact }}
path: | path: |
@@ -155,29 +166,16 @@ jobs:
if-no-files-found: error if-no-files-found: error
freebsd:
name: Stack - FreeBSD from CirrusCI
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- uses: ./.github/actions/artifact-from-cirrus
with:
token: ${{ github.token }}
task: Build FreeBSD (Stack)
download: bin
upload: postgrest-freebsd-x86-64
cabal: cabal:
strategy: strategy:
matrix: matrix:
ghc: ['9.6.7', '9.8.4'] ghc: ['9.10.3', '9.12.3']
fail-fast: false fail-fast: false
name: Cabal - Linux x86-64 - GHC ${{ matrix.ghc }} name: Cabal - Linux x86-64 - GHC ${{ matrix.ghc }}
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: haskell-actions/setup@0512451d82f3ca8c147db62e30464e7c4ca63d30 # v2.9.1 - uses: haskell-actions/setup@6037f33647c3f17758a2356c80fc4a53d7e0685d # v2.12.0
with: with:
ghc-version: ${{ matrix.ghc }} ghc-version: ${{ matrix.ghc }}
- name: Cache .cabal - name: Cache .cabal
+2 -2
View File
@@ -20,7 +20,7 @@ jobs:
name: Lint & Style name: Lint & Style
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment - name: Setup Nix Environment
uses: ./.github/actions/setup-nix uses: ./.github/actions/setup-nix
with: with:
@@ -36,7 +36,7 @@ jobs:
name: Commit name: Commit
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with: with:
fetch-depth: 100 # fetch history (last 100 commits) instead of default shallow clone history, this is deemed enough for a PR history fetch-depth: 100 # fetch history (last 100 commits) instead of default shallow clone history, this is deemed enough for a PR history
- name: Setup Nix Environment - name: Setup Nix Environment
+3 -4
View File
@@ -41,16 +41,15 @@ jobs:
concurrency: concurrency:
# Never tag outdated commits on the main branch by skipping superseded commits # Never tag outdated commits on the main branch by skipping superseded commits
group: ci-tag-${{ (github.ref == 'refs/heads/main' && github.ref) || github.run_id }} group: ci-tag-${{ (github.ref == 'refs/heads/main' && github.ref) || github.run_id }}
# TODO: Enable this once https://github.com/orgs/community/discussions/13015 is solved cancel-in-progress: true
cancel-in-progress: false
if: vars.RELEASE_ENABLED if: vars.RELEASE_ENABLED
runs-on: ubuntu-24.04 runs-on: ubuntu-slim
needs: needs:
- docs - docs
- test - test
- build - build
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with: with:
ssh-key: ${{ secrets.POSTGREST_SSH_KEY }} ssh-key: ${{ secrets.POSTGREST_SSH_KEY }}
- name: Tag latest commit - name: Tag latest commit
+3 -2
View File
@@ -14,6 +14,7 @@ on:
- .github/actions/setup-nix/** - .github/actions/setup-nix/**
- default.nix - default.nix
- nix/** - nix/**
- flake.lock
- docs/** - docs/**
- '!**.md' - '!**.md'
@@ -27,7 +28,7 @@ jobs:
name: Build name: Build
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment - name: Setup Nix Environment
uses: ./.github/actions/setup-nix uses: ./.github/actions/setup-nix
with: with:
@@ -41,7 +42,7 @@ jobs:
name: Spellcheck name: Spellcheck
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment - name: Setup Nix Environment
uses: ./.github/actions/setup-nix uses: ./.github/actions/setup-nix
with: with:
+27 -2
View File
@@ -7,12 +7,37 @@ on:
jobs: jobs:
linkcheck: linkcheck:
name: Linkcheck
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment - name: Setup Nix Environment
uses: ./.github/actions/setup-nix uses: ./.github/actions/setup-nix
with: with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
tools: docs.linkcheck.bin tools: docs.linkcheck.bin
- run: postgrest-docs-linkcheck
- name: Run Linkcheck
id: linkcheck
run: postgrest-docs-linkcheck
# This actions creates the github token using the postgrest app secrets
- name: Create Github App Token (Runs only on linkcheck failure)
id: app-token
if: ${{ failure() && steps.linkcheck.outcome == 'failure' }} # only create the token on linkcheck failure
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with:
client-id: ${{ vars.POSTGREST_CI_APP_ID }}
private-key: ${{ secrets.POSTGREST_CI_PRIVATE_KEY }}
permission-issues: write # required for commenting on issues
- name: Notify on linkcheck failure by commenting on GH Issue 4106
if: ${{ failure() && steps.linkcheck.outcome == 'failure' }}
uses: peter-evans/create-or-update-comment@e8674b075228eee787fea43ef493e45ece1004c9 # v5.0.0
with:
token: ${{ steps.app-token.outputs.token }}
issue-number: 4106
body: |
**Linkcheck Job Failed!**
A broken link was detected in the docs. Please check the [failed run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details.
+94 -70
View File
@@ -9,8 +9,7 @@ on:
concurrency: concurrency:
# Terminate all previous runs of the same workflow for the same tag. # Terminate all previous runs of the same workflow for the same tag.
group: release-${{ github.ref }} group: release-${{ github.ref }}
# TODO: Enable this once https://github.com/orgs/community/discussions/13015 is solved cancel-in-progress: true
cancel-in-progress: false
jobs: jobs:
build: build:
@@ -20,13 +19,15 @@ jobs:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
prepare: github:
name: Prepare name: GitHub
runs-on: ubuntu-24.04 permissions:
contents: write
runs-on: ubuntu-slim
needs: needs:
- build - build
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Check the version to be released - name: Check the version to be released
run: | run: |
cabal_version="$(grep -oP '^version:\s*\K.*' postgrest.cabal)" cabal_version="$(grep -oP '^version:\s*\K.*' postgrest.cabal)"
@@ -48,25 +49,9 @@ jobs:
echo "Relevant extract from CHANGELOG.md:" echo "Relevant extract from CHANGELOG.md:"
cat CHANGES.md cat CHANGES.md
- name: Save CHANGES.md as artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: release-changes
path: CHANGES.md
if-no-files-found: error
github:
name: GitHub
permissions:
contents: write
runs-on: ubuntu-24.04
needs:
- prepare
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- name: Download all artifacts - name: Download all artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0 uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with: with:
path: artifacts path: artifacts
- name: Create release bundle with archives for all builds - name: Create release bundle with archives for all builds
@@ -75,23 +60,26 @@ jobs:
mkdir -p release-bundle mkdir -p release-bundle
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-linux-static-aarch64.tar.xz" \
-C artifacts/postgrest-linux-static-aarch64 postgrest
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-linux-static-x86-64.tar.xz" \ tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-linux-static-x86-64.tar.xz" \
-C artifacts/postgrest-linux-static-x86-64 postgrest -C artifacts/postgrest-linux-static-x86-64 postgrest
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-macos-aarch64.tar.xz" \ tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-macos-aarch64.tar.xz" \
-C artifacts/postgrest-macos-aarch64 postgrest -C artifacts/postgrest-macos-aarch64 postgrest
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-macos-x86-64.tar.xz" \
-C artifacts/postgrest-macos-x86-64 postgrest
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-freebsd-x86-64.tar.xz" \ tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-freebsd-x86-64.tar.xz" \
-C artifacts/postgrest-freebsd-x86-64 postgrest -C artifacts/postgrest-freebsd-x86-64 postgrest
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-ubuntu-aarch64.tar.xz" \
-C artifacts/postgrest-ubuntu-aarch64 postgrest
zip --junk-paths "release-bundle/postgrest-${GITHUB_REF_NAME}-windows-x86-64.zip" \ zip --junk-paths "release-bundle/postgrest-${GITHUB_REF_NAME}-windows-x86-64.zip" \
artifacts/postgrest-windows-x86-64/postgrest.exe artifacts/postgrest-windows-x86-64/postgrest.exe
- name: Save release bundle - name: Save release bundle
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with: with:
name: release-bundle name: release-bundle
path: release-bundle path: release-bundle
@@ -113,14 +101,14 @@ jobs:
gh release edit devel \ gh release edit devel \
-t devel \ -t devel \
--verify-tag \ --verify-tag \
-F artifacts/release-changes/CHANGES.md \ -F CHANGES.md \
--prerelease --prerelease
gh release upload --clobber devel release-bundle/* gh release upload --clobber devel release-bundle/*
else else
gh release create "${GITHUB_REF_NAME}" \ gh release create "${GITHUB_REF_NAME}" \
-t "${GITHUB_REF_NAME}" \ -t "${GITHUB_REF_NAME}" \
--verify-tag \ --verify-tag \
-F artifacts/release-changes/CHANGES.md \ -F CHANGES.md \
release-bundle/* release-bundle/*
fi fi
@@ -129,70 +117,55 @@ jobs:
name: Docker Hub name: Docker Hub
runs-on: ubuntu-24.04-arm runs-on: ubuntu-24.04-arm
needs: needs:
- prepare - github
if: | if: |
vars.DOCKER_REPO && vars.DOCKER_USER vars.DOCKER_REPO && vars.DOCKER_USER
env: env:
DOCKER_REPO: ${{ vars.DOCKER_REPO }} DOCKER_REPO: ${{ vars.DOCKER_REPO }}
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Download aarch64 Docker image
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: postgrest-docker-aarch64
- name: Download x86-64 Docker image - name: Download x86-64 Docker image
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0 uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with: with:
name: postgrest-docker-x86-64 name: postgrest-docker-x86-64
- name: Download aarch64 binary - uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0 - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
name: postgrest-ubuntu-aarch64
- uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with: with:
username: ${{ vars.DOCKER_USER }} username: ${{ vars.DOCKER_USER }}
password: ${{ secrets.DOCKER_PASS }} password: ${{ secrets.DOCKER_PASS }}
- name: Build aarch64 Docker image
run: |
# This only pushes the image via digest, not a tag. This will not appear
# in the image list on Docker Hub, yet. It will be later added to the main
# tag's manifest.
docker buildx build \
-t "$DOCKER_REPO/postgrest" \
--platform linux/arm64 \
--output push-by-digest=true,type=image,push=true \
--metadata-file metadata.json \
.
echo "SHA256_ARM=$(jq -r '."containerimage.digest"' metadata.json)" >> "$GITHUB_ENV"
- name: Publish images on Docker Hub - name: Publish images on Docker Hub
run: | run: |
docker load -i postgrest-docker.tar.gz docker load -i postgrest-docker-aarch64.tar.gz
docker tag postgrest:latest "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}-linux-arm64"
docker push "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}-linux-arm64"
docker tag postgrest:latest "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}" docker load -i postgrest-docker-x86-64.tar.gz
docker push "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}" docker tag postgrest:latest "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}-linux-amd64"
docker buildx imagetools create --append \ docker push "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}-linux-amd64"
-t "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}" \
"$DOCKER_REPO/postgrest@$SHA256_ARM" docker manifest create "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}" \
"$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}-linux-arm64" \
"$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}-linux-amd64"
docker manifest push "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}"
# Only tag 'latest' for full releases # Only tag 'latest' for full releases
if [ "${GITHUB_REF_NAME}" != "devel" ]; then if [ "${GITHUB_REF_NAME}" != "devel" ]; then
echo "Pushing to 'latest' tag for full release of ${GITHUB_REF_NAME} ..." echo "Pushing to 'latest' tag for full release of ${GITHUB_REF_NAME} ..."
docker tag postgrest:latest "$DOCKER_REPO"/postgrest:latest docker manifest create "$DOCKER_REPO/postgrest:latest" \
docker push "$DOCKER_REPO"/postgrest:latest "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}-linux-arm64" \
docker buildx imagetools create --append \ "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}-linux-amd64"
-t "$DOCKER_REPO/postgrest:latest" \ docker manifest push "$DOCKER_REPO/postgrest:latest"
"$DOCKER_REPO/postgrest@$SHA256_ARM"
else else
echo "Skipping push to 'latest' tag for pre-release..." echo "Skipping push to 'latest' tag for pre-release..."
fi fi
docker-description:
name: Docker Hub Description
runs-on: ubuntu-24.04
if: |
vars.DOCKER_REPO && vars.DOCKER_USER &&
github.ref == 'refs/tags/devel'
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0 - uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0
if: github.ref == 'refs/tags/devel'
name: Docker Hub Description
with: with:
username: ${{ vars.DOCKER_USER }} username: ${{ vars.DOCKER_USER }}
password: ${{ secrets.DOCKER_PASS }} password: ${{ secrets.DOCKER_PASS }}
@@ -200,3 +173,54 @@ jobs:
short-description: ${{ github.event.repository.description }} short-description: ${{ github.event.repository.description }}
readme-filepath: ./docker-hub-readme.md readme-filepath: ./docker-hub-readme.md
ghcr:
name: GitHub Container Registry
runs-on: ubuntu-24.04-arm
needs:
- github
permissions:
packages: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Download aarch64 Docker image
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: postgrest-docker-aarch64
- name: Download x86-64 Docker image
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: postgrest-docker-x86-64
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Publish images on Docker Hub
run: |
docker load -i postgrest-docker-aarch64.tar.gz
docker tag postgrest:latest "ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_REF_NAME}-linux-arm64"
docker push "ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_REF_NAME}-linux-arm64"
docker load -i postgrest-docker-x86-64.tar.gz
docker tag postgrest:latest "ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_REF_NAME}-linux-amd64"
docker push "ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_REF_NAME}-linux-amd64"
docker manifest create "ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_REF_NAME}" \
"ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_REF_NAME}-linux-arm64" \
"ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_REF_NAME}-linux-amd64"
docker manifest push "ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_REF_NAME}"
# Only tag 'latest' for full releases
if [ "${GITHUB_REF_NAME}" != "devel" ]; then
echo "Pushing to 'latest' tag for full release of ${GITHUB_REF_NAME} ..."
docker manifest create "ghcr.io/${GITHUB_REPOSITORY,,}:latest" \
"ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_REF_NAME}-linux-arm64" \
"ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_REF_NAME}-linux-amd64"
docker manifest push "ghcr.io/${GITHUB_REPOSITORY,,}:latest"
else
echo "Skipping push to 'latest' tag for pre-release..."
fi
+56 -17
View File
@@ -17,6 +17,7 @@ on:
- .github/actions/setup-nix/** - .github/actions/setup-nix/**
- default.nix - default.nix
- nix/** - nix/**
- flake.lock
- .stylish-haskell.yaml - .stylish-haskell.yaml
- cabal.project - cabal.project
- postgrest.cabal - postgrest.cabal
@@ -24,6 +25,10 @@ on:
- test/** - test/**
- '!**.md' - '!**.md'
defaults:
run:
shell: bash
concurrency: concurrency:
# Terminate all previous runs of the same workflow for pull requests # Terminate all previous runs of the same workflow for pull requests
group: test-${{ github.head_ref || github.run_id }} group: test-${{ github.head_ref || github.run_id }}
@@ -39,7 +44,7 @@ jobs:
# https://github.com/actions/runner/issues/241#issuecomment-842566950 # https://github.com/actions/runner/issues/241#issuecomment-842566950
shell: script -qec "bash --noprofile --norc -eo pipefail {0}" shell: script -qec "bash --noprofile --norc -eo pipefail {0}"
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment - name: Setup Nix Environment
uses: ./.github/actions/setup-nix uses: ./.github/actions/setup-nix
with: with:
@@ -48,17 +53,17 @@ jobs:
- run: postgrest-cabal-update - run: postgrest-cabal-update
- name: Run coverage (IO tests and Spec tests against PostgreSQL 15) - name: Run coverage (IO tests and Spec tests against latest supported PostgreSQL)
run: postgrest-coverage run: postgrest-coverage
- name: Upload coverage to codecov - name: Upload coverage to codecov
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2 uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with: with:
files: ./coverage/codecov.json files: ./coverage/codecov.json
token: ${{ secrets.CODECOV_TOKEN }} token: ${{ secrets.CODECOV_TOKEN }}
- name: Run doctests - name: Run doctests
if: always() if: always()
run: postgrest-test-doctests run: nix-shell --run postgrest-test-doctests
- name: Check the spec tests for idempotence - name: Check the spec tests for idempotence
if: always() if: always()
@@ -69,8 +74,9 @@ jobs:
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
pgVersion: [13, 14, 15, 16, 17] # Latest version is tested via `coverage` above.
name: PG ${{ matrix.pgVersion }} pgVersion: [pg-14, pg-15, pg-16, pg-17, oriole-18, pg-18]
name: ${{ matrix.pgVersion }}
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
defaults: defaults:
run: run:
@@ -78,33 +84,37 @@ jobs:
# https://github.com/actions/runner/issues/241#issuecomment-842566950 # https://github.com/actions/runner/issues/241#issuecomment-842566950
shell: script -qec "bash --noprofile --norc -eo pipefail {0}" shell: script -qec "bash --noprofile --norc -eo pipefail {0}"
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment - name: Setup Nix Environment
uses: ./.github/actions/setup-nix uses: ./.github/actions/setup-nix
with: with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
tools: tests.testSpec.bin tests.testIO.bin tests.testBigSchema.bin withTools.pg-${{ matrix.pgVersion }}.bin cabalTools.update.bin tools: tests.testSpec.bin tests.testObservability.bin tests.testIO.bin tests.testBigSchema.bin withTools.${{ matrix.pgVersion }}.bin cabalTools.update.bin
- run: postgrest-cabal-update - run: postgrest-cabal-update
- name: Run spec tests - name: Run spec tests
if: always() if: always()
run: postgrest-with-pg-${{ matrix.pgVersion }} postgrest-test-spec run: postgrest-with-${{ matrix.pgVersion }} postgrest-test-spec
- name: Run observability tests
if: always()
run: postgrest-with-${{ matrix.pgVersion }} postgrest-test-observability
- name: Run IO tests - name: Run IO tests
if: always() if: always()
run: postgrest-with-pg-${{ matrix.pgVersion }} postgrest-test-io -vv run: postgrest-with-${{ matrix.pgVersion }} postgrest-test-io -vv
- name: Run IO tests on a big schema - name: Run IO tests on a big schema
if: always() if: always()
run: postgrest-with-pg-${{ matrix.pgVersion }} postgrest-test-big-schema -vv run: postgrest-with-${{ matrix.pgVersion }} postgrest-test-big-schema -vv
memory: memory:
name: Memory name: Memory
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment - name: Setup Nix Environment
uses: ./.github/actions/setup-nix uses: ./.github/actions/setup-nix
with: with:
@@ -119,19 +129,20 @@ jobs:
loadtest: loadtest:
strategy: strategy:
fail-fast: false
matrix: matrix:
kind: ['mixed', 'jwt-hs', 'jwt-hs-cache', 'jwt-hs-cache-worst', 'jwt-rsa', 'jwt-rsa-cache', 'jwt-rsa-cache-worst'] kind: ['mixed', 'jwt-cache']
name: Loadtest name: Loadtest
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with: with:
fetch-depth: 0 fetch-depth: 0
- name: Setup Nix Environment - name: Setup Nix Environment
uses: ./.github/actions/setup-nix uses: ./.github/actions/setup-nix
with: with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
tools: loadtest.loadtestAgainst.bin loadtest.report.bin cabalTools.update.bin tools: loadtest.loadtestAgainst.bin loadtest.report.bin loadtest.report-load.bin cabalTools.update.bin
- run: postgrest-cabal-update - run: postgrest-cabal-update
@@ -145,7 +156,35 @@ jobs:
latest_tag=$(git tag --merged HEAD --sort=-creatordate "v*" | head -n1) latest_tag=$(git tag --merged HEAD --sort=-creatordate "v*" | head -n1)
fi fi
postgrest-loadtest-against -k ${{ matrix.kind }} "$TARGET_BRANCH" "$latest_tag" postgrest-loadtest-against -k ${{ matrix.kind }} "$TARGET_BRANCH" "$latest_tag"
postgrest-loadtest-report -g ${{ matrix.kind }} >> "$GITHUB_STEP_SUMMARY"
- name: Report P50
# This step checks whether any red cross indicators (:x:) are present in the step summary.
# The loadtest reporter writes them when any of individual steps fails the performance
# regression threshold.
run: |
! (postgrest-loadtest-report -g ${{ matrix.kind }} -p 50 \
| tee "$GITHUB_STEP_SUMMARY" \
| grep ':x:')
- name: Report P0
if: always()
run: |
postgrest-loadtest-report -g ${{ matrix.kind }} -p 0 >> "$GITHUB_STEP_SUMMARY"
- name: Report P90
if: always()
run: |
postgrest-loadtest-report -g ${{ matrix.kind }} -p 90 >> "$GITHUB_STEP_SUMMARY"
- name: Report P95
if: always()
run: |
postgrest-loadtest-report -g ${{ matrix.kind }} -p 95 >> "$GITHUB_STEP_SUMMARY"
- name: Report CPU/MEM
if: always()
run: |
postgrest-loadtest-report-load -g ${{ matrix.kind }} >> "$GITHUB_STEP_SUMMARY"
flake: flake:
strategy: strategy:
@@ -158,7 +197,7 @@ jobs:
name: Flake Check name: Flake Check
runs-on: ${{ matrix.runs-on }} runs-on: ${{ matrix.runs-on }}
steps: steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with: with:
fetch-depth: 0 fetch-depth: 0
- name: Setup Nix Environment - name: Setup Nix Environment
+6 -1
View File
@@ -25,4 +25,9 @@ loadtest
.history .history
.docs-build .docs-build
gen_targets.http gen_targets.http
gen_jwk.json gen_jwks.json
gen_private.json
.pytest_cache
.ruff_cache
postgrest-module-graph.png
.ghc.environment.*
+1 -1
View File
@@ -7,4 +7,4 @@ python:
build: build:
os: ubuntu-24.04 os: ubuntu-24.04
tools: tools:
python: "3.11" python: "3.12"
+1 -1
View File
@@ -200,7 +200,7 @@ steps:
# A common setting is the number of columns (parts of) code will be wrapped # A common setting is the number of columns (parts of) code will be wrapped
# to. Different steps take this into account. Default: 80. # to. Different steps take this into account. Default: 80.
columns: 70 columns: 80
# By default, line endings are converted according to the OS. You can override # By default, line endings are converted according to the OS. You can override
# preferred format here. # preferred format here.
+13 -8
View File
@@ -13,26 +13,26 @@ PostgREST ongoing development is only possible thanks to our Sponsors and Backer
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank">
<img width="296px" src="static/neon.jpg"> <img width="296px" src="static/supabase.svg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://tembo.io/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://www.euronodes.com/postgrest" target="_blank">
<img width="296px" src="static/tembo.png"> <img width="296px" src="static/euronodes.svg">
</a> </a>
</td> </td>
</tr> </tr>
<tr></tr> <tr></tr>
<tr> <tr>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://www.euronodes.com/postgrest" target="_blank"> <a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="static/euronodes.svg"> <img width="296px" src="static/neon.jpg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank"> <a href="https://www.bytebase.com/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="static/supabase.png"> <img width="296px" src="static/bytebase.svg">
</a> </a>
</td> </td>
</tr> </tr>
@@ -93,6 +93,11 @@ PostgREST ongoing development is only possible thanks to our Sponsors and Backer
<img width="222px" src="static/code-build.png"> <img width="222px" src="static/code-build.png">
</a> </a>
</td> </td>
<td align="center" valign="middle">
<a href="https://tembo.io/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="static/tembo.png">
</a>
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
+207 -2
View File
@@ -4,6 +4,211 @@ All notable changes to this project will be documented in this file. From versio
## Unreleased ## Unreleased
## [16.1] - 2026-08-10
### Fixed
- JWT validation uses wrong current time due to a bug in auto-update by @mkleczek in #5159
## [16.0] - 2026-08-07
### Changes
#### HTTP Server
- [Graceful shutdown](https://docs.postgrest.org/en/v16/references/http_server.html#graceful-shutdown) by @mkleczek, @Vlix in #4702
- [server-reuseport](https://docs.postgrest.org/en/v16/references/configuration.html#server-reuseport) allows starting multiple PostgREST instances using the same port on supported platforms by @mkleczek in #4703, #4694
#### Performance
- Optimize schema cache domain type resolution by using [pg_basetype](https://www.postgresql.org/docs/current/functions-info.html#FUNCTIONS-INFO-CATALOG) on PostgreSQL 17+ by @joelonsql in #4567
- [Prefer: count=exact](https://docs.postgrest.org/en/v16/references/api/pagination_count.html#exact-count) no longer does a double count on requests that do not use ranges or `db-max-rows` by @laurenceisla in #3957
- [Prefer: timezone](https://docs.postgrest.org/en/v16/references/api/preferences.html#prefer-timezone) no longer requires the schema cache by @steve-chavez in #5100
+ Previously this required caching [pg_timezone_names](https://www.postgresql.org/docs/current/view-pg-timezone-names.html) which was slow in some systems
#### Integrations
- PostgREST is now tested to work with [OrioleDB](https://github.com/orioledb/orioledb/) in #4845 by @wolfgangwalther
+ See [our guide for running OrioleDB on NixOS](https://docs.postgrest.org/en/v16/integrations/nixos.html)
#### JWT
- [JWT Role Extraction](https://docs.postgrest.org/en/v16/references/auth.html#jwt-role-extract) is now more flexible, supporting the standard JSON Path defined in RFC 9535 by @taimoorzaeem in #4984
#### API
- [Prefer: timezone](https://docs.postgrest.org/en/v16/references/api/preferences.html#timezone) now supports numeric offsets like `05:00` or `-4` by @steve-chavez in #5100
- Fix unexpected results when embedding and filtering the same table more than once by @laurenceisla in #4075
+ You need to set [url-use-legacy-target-names](https://docs.postgrest.org/en/v16/references/configuration.html#url-use-legacy-target-names) to `false`.
- Deprecate filters, orders and limits with the name of an embedded table when it has an alias by @steve-chavez, @laurenceisla in #4075
+ e.g. `?select=alias:table(*)&table.id=eq.1` will not be possible anymore, use `?select=alias:table(*)&alias.id=eq.1` instead.
+ You will see a warning in the logs and a `Warning` header on the client response when this happens.
+ You can disable this behavior now by setting `url-use-legacy-target-names = false`.
- Add `Vary` header to responses by @develop7 in #4609
- Fix automatic transaction retries on `40001 (serialization_failure)` errors to prevent replication lag by @laurenceisla in #3673
#### Observability
- [GHC runtime metrics](https://docs.postgrest.org/en/v16/references/observability.html#ghc-runtime-metrics) by @mkleczek in #4862
- [client-error-verbosity](https://docs.postgrest.org/en/v16/references/configuration.html#client-error-verbosity) to customize responses error verbosity by @taimoorzaeem in #4088, #3980, #3824
- [log-level](https://docs.postgrest.org/en/v16/references/configuration.html#log-level) config is now reloadable by @taimoorzaeem in #5113
- Log error when `db-schemas` config contains schema `pg_catalog` or `information_schema` by @taimoorzaeem in #4359
- Log schema cache queries timings on `log-level=debug` by @steve-chavez in #4805
#### Admin Server
- [admin-server-unix-socket](https://docs.postgrest.org/en/v16/references/configuration.html#admin-server-unix-socket)/[admin-server-unix-socket-mode](https://docs.postgrest.org/en/v16/references/configuration.html#admin-server-unix-socket-mode) to run the admin server on a unix socket by @wolfgangwalther in #5003
- Fix responding with `Something went wrong` on Admin server when under EMFILE by @mkleczek in #5077
#### Deployment
- Make executable for aarch64-linux static instead of Ubuntu-based by @wolfgangwalther in #4193
- Docker image for aarch64-linux is now built from scratch instead of being Ubuntu-based by @wolfgangwalther in #4193
- Besides Docker Hub, docker images are now published to Github Container Registry by @wolfgangwalther in #2836
#### Schema Cache
- Fix requests failing when the schema cache fails to reload, when this happens PostgREST will continue serving requests in "best effort" by @mkleczek in #4873 #4869
- Fix reporting 503s errors unnecessarily while the schema cache is loading at startup by @mkleczek in #4880
- Fix schema cache dump missing RPC transaction isolation level by @taimoorzaeem in #5079
#### Listener
- Fix config `db-channel-enabled` not reloading by @taimoorzaeem in #4894
### Migration to v16
- Drop support for PostgreSQL EOL version 13 by @wolfgangwalther in #4193
+ PostgreSQL 13 end of life was on 2025 ([ref](https://www.postgresql.org/support/versioning/))
+ Upgrade your PostgreSQL version to at least 14 to use this new PostgREST version.
- Fail at startup when `db-schemas` contains schema `pg_catalog` or `information_schema` by @taimoorzaeem in #4359
+ Previously it failed at runtime with `PGRST205` on requests related to these schemas.
+ Remove `pg_catalog` and `information_schema` from `db-schemas`.
- `Prefer: timezone` no longer complies with `handling=lenient` and instead always fails by @steve-chavez in #5128
+ Supporting this required caching `pg_timezone_names`, which was expensive.
+ Ensure your requests always have a valid timezone.
- `jwt-role-claim-key` no longer uses the JSPath DSL and instead uses JSON Path by @taimoorzaeem in #4984
+ Now all config values must start with `$` character.
Example: `.roles.read` -> `$.roles.read`
+ Keys with special characters, with the exception of `_` char must be quoted.
Example: `.roles.write-role` -> `$.roles["write-role"]`
+ String comparison operators (`^==`, `==^` and `*==`) are replaced with regular expression search.
Example: `.roles[?(@ ^== "postgrest_test_")]` -> `$.roles[?search(@, "^postgrest_test_")]`
+ Update the `jwt-role-claim-key` value accoring to the above rules. Also see the syntax reference: [RFC 9535](https://www.rfc-editor.org/rfc/rfc9535.html#name-jsonpath-syntax-and-semanti).
## [14.16] - 2026-07-27
### Fixed
- Fix admin server crashing without a way to recover by @taimoorzaeem in #5096
## [14.15] - 2026-07-13
### Fixed
- Fix admin server dying silently by @Vlix, @mkleczek, @steve-chavez in #5012
## [14.14] - 2026-06-29
### Fixed
- Fix admin server not logging cause of failure by @taimoorzaeem in #5012
## [14.13] - 2026-06-04
### Fixed
- Fix connection retrying message in `PGRST000` error by @netqo in #4980
+ Remove redundant "Retrying the connection." from message because it is logged separately
- Fix request failures when `work_mem` is set on a role by @laurenceisla in #4955
## [14.12] - 2026-05-20
### Fixed
- Fix race condition in pool_available metric causing negative values during network instability by @mkleczek in #4622
## [14.11] - 2026-05-04
### Fixed
- Fix login with uppercase and mixed case role names by @taimoorzaeem in #4678
- Restore Listener query shape so it can be found in `pg_stat_activity` by @mkleczek in #4857 #4859
- The LISTEN channel now automatically recovers when it stops working due to a PostgreSQL bug @laurenceisla in #3147
- Fix misleading "Functions" name on schema cache summary in startup logs by @taimoorzaeem in #4821
## [14.10] - 2026-04-16
### Added
- Log when the pool is released during schema cache reload on `log-level=debug` by @mkleczek in #4668
### Fixed
- Fix unnecessary connection pool flushes during schema cache reloading by @mkleczek in #4645
## [14.9] - 2026-04-10
### Added
- Log host, port and pg version of listener database connection by @mkleczek in #4617 #4618
### Fixed
- Remove red herring warp logs on default log-level, only emit them on `log-level=debug` by @steve-chavez in #4799
## [14.8] - 2026-04-03
### Added
- Log a `HINT` when the LISTEN channel stops working due to a PostgreSQL bug by @laurenceisla in #4581
### Fixed
- Fix invalid OpenAPI 2.0 format for integer types (`smallint`, `integer`, `bigint`) by @arturbent0 in #4641
## [14.7] - 2026-03-20
### Fixed
- Fix not logging SIGTERM and SIGINT by @steve-chavez in #4728
## [14.6] - 2026-03-06
### Fixed
- Fix leaking table and function names when calculating error hint by @taimoorzaeem in #4675
## [14.5] - 2026-02-12
### Fixed
- Don't hide async exceptions in logs by @stevechavez in #4646
## [14.4] - 2026-01-29
### Fixed
- Ensure Listener connections are released by @mkleczek in #4614
- Fix incorrectly filtering the returned representation for PATCH requests when using `or/and` filters by @laurenceisla in #3707
- Fix listener running with exception masked after first failure by @mkleczek in #4615
## [14.3] - 2026-01-03
### Fixed
- Fix performance and high memory usage of relation hint calculation by @mkleczek in #4462, #4463
## [14.2] - 2025-12-18 ## [14.2] - 2025-12-18
### Fixed ### Fixed
@@ -600,7 +805,7 @@ All notable changes to this project will be documented in this file. From versio
### Added ### Added
- #1933, #2109, Add a minimal health check endpoint - @steve-chavez - #1933, #2109, Add a minimal health check endpoint - @steve-chavez
+ For enabling this, the `admin-server-port` config must be set explictly + For enabling this, the `admin-server-port` config must be set explicitly
+ A `<host>:<admin_server_port>/live` endpoint is available for checking if postgrest is running on its port/socket. 200 OK = alive, 503 = dead. + A `<host>:<admin_server_port>/live` endpoint is available for checking if postgrest is running on its port/socket. 200 OK = alive, 503 = dead.
+ A `<host>:<admin_server_port>/ready` endpoint is available for checking a correct internal state(the database connection plus the schema cache). 200 OK = ready, 503 = not ready. + A `<host>:<admin_server_port>/ready` endpoint is available for checking a correct internal state(the database connection plus the schema cache). 200 OK = ready, 503 = not ready.
- #1988, Add the current user to the request log on stdout - @DavidLindbom, @wolfgangwalther - #1988, Add the current user to the request log on stdout - @DavidLindbom, @wolfgangwalther
@@ -1083,7 +1288,7 @@ All notable changes to this project will be documented in this file. From versio
- Customize content negotiation per route - @begriffs - Customize content negotiation per route - @begriffs
- Allow using nulls order without explicit order direction - @steve-chavez - Allow using nulls order without explicit order direction - @steve-chavez
- Fatal error on postgres unsupported version, format supported version in error message - @steve-chavez - Fatal error on postgres unsupported version, format supported version in error message - @steve-chavez
- Prevent database memory cosumption by prepared statements caches - @ruslantalpa - Prevent database memory consumption by prepared statements caches - @ruslantalpa
- Use specific columns in the RETURNING section - @ruslantalpa - Use specific columns in the RETURNING section - @ruslantalpa
- Fix columns alias for RETURNING - @steve-chavez - Fix columns alias for RETURNING - @steve-chavez
+23 -14
View File
@@ -1,17 +1,12 @@
# Contributing to PostgREST # Contributing to PostgREST
**First:** if you're unsure or afraid of _anything_, just ask or ## AI Policy
submit the issue or pull request anyways. You won't be yelled at
for giving your best effort. The worst that can happen is that
you'll be politely asked to change something. We appreciate any
sort of contributions, and don't want a wall of rules to get in the
way of that.
However, for those individuals who want a bit more guidance on the We adhere to [Gentoo's AI policy](https://wiki.gentoo.org/wiki/Project:Council/AI_policy):
best way to contribute to the project, read on. This document will
cover what we're looking for. By addressing all the points we're > It is expressly forbidden to contribute [...] any content that has been created with the assistance of Natural Language Processing artificial intelligence tools. This motion can be revisited, should a case been made over such a tool that does not pose copyright, ethical and quality concerns.
looking for, it raises the chances we can quickly merge or address
your contributions. You can find more about its rationale [here](https://wiki.gentoo.org/wiki/Project:Council/AI_policy#Rationale).
## Issues ## Issues
@@ -40,12 +35,14 @@ For questions on how to use PostgREST, please use
We have a fully nix-based development environment with many tools for a smooth development workflow available. We have a fully nix-based development environment with many tools for a smooth development workflow available.
Check the [development docs](https://github.com/PostgREST/postgrest/blob/main/nix/README.md) on how to set it up and use it. Check the [development docs](https://github.com/PostgREST/postgrest/blob/main/nix/README.md) on how to set it up and use it.
### Haskell Conventions
* All contributions must pass the tests before being merged. When * All contributions must pass the tests before being merged. When
you create a pull request your code will automatically be tested. you create a pull request your code will automatically be tested.
* All code must also pass [hlint](http://community.haskell.org/~ndm/hlint/) and [stylish-haskell](https://github.com/jaspervdj/stylish-haskell) * All fixes or features must have a test proving the improvement.
* All features must document the new behavior. Critical fixes that introduce new behavior must be documented too.
* All code must also pass a [linter](http://community.haskell.org/~ndm/hlint/) and [styler](https://github.com/jaspervdj/stylish-haskell)
with no warnings. This helps enforce a uniform style for all committers. Continuous integration will check this as well on every with no warnings. This helps enforce a uniform style for all committers. Continuous integration will check this as well on every
pull request. There are useful tools in the nix-shell that help with checking this locally. You can run `postgrest-check` to do this manually but pull request. There are useful tools in the nix-shell that help with checking this locally. You can run `postgrest-check` to do this manually but
we recommend adding it to `.git/hooks/pre-commit` as `nix-shell --run postgrest-check` to automatically check this before doing a commit. we recommend adding it to `.git/hooks/pre-commit` as `nix-shell --run postgrest-check` to automatically check this before doing a commit.
@@ -53,3 +50,15 @@ Check the [development docs](https://github.com/PostgREST/postgrest/blob/main/ni
### Running Tests ### Running Tests
For instructions on running tests, see the [development docs](https://github.com/PostgREST/postgrest/blob/main/nix/README.md#testing). For instructions on running tests, see the [development docs](https://github.com/PostgREST/postgrest/blob/main/nix/README.md#testing).
### Structuring commits in pull requests
To simplify reviews, make it easy to split pull requests if deemed necessary, and to maintain clean and meaningful history of changes, you will be asked to update your PR if it does not follow the below rules:
* It must be possible to merge the PR branch into target using `git merge --ff-only`, ie. the source branch must be rebased on top of target.
* No merge commits in the source branch.
* All commits in the source branch must be self contained, meaning: it should be possible to treat each commit as a separate PR.
* Commits in the source branch must contain only related changes (related means the changes target a single problem/goal). For example, any refactorings should be isolated from the actual change implementation into separate commits.
* Tests, documentation, and changelog updates should be contained in the same commits as the actual code changes they relate to. An exception to this rule is when test or documentation changes are made in separate PR.
* Commit messages must be prefixed with one of the prefixes defined in [the list used by commit verification scripts](https://github.com/PostgREST/postgrest/blob/main/nix/tools/gitTools.nix#L11).
* Commit messages should contain a longer description of the purpose of the changes contained in the commit and, for non-trivial changes, a description of the changes themselves.
-21
View File
@@ -1,21 +0,0 @@
# PostgREST Docker Hub image for aarch64.
# The x86-64 is a single-static-binary image built via Nix, see:
# nix/tools/docker/README.md
FROM ubuntu:noble@sha256:c35e29c9450151419d9448b0fd75374fec4fff364a27f176fb458d472dfc9e54 AS postgrest
RUN apt-get update -y \
&& apt install -y --no-install-recommends libpq-dev zlib1g-dev jq gcc libnuma-dev \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
COPY postgrest /usr/bin/postgrest
RUN chmod +x /usr/bin/postgrest
EXPOSE 3000
USER 1000
# Use the array form to avoid running the command using bash, which does not handle `SIGTERM` properly.
# See https://docs.docker.com/compose/faq/#why-do-my-services-take-10-seconds-to-recreate-or-stop
CMD ["postgrest"]
+1 -2
View File
@@ -1,5 +1,4 @@
Copyright (c) 2014 Joe Nelson Copyright (c) 2014-2026 The PostgREST contributors
Copyright (c) 2019 Steve Chavez
Permission is hereby granted, free of charge, to any person obtaining Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the a copy of this software and associated documentation files (the
+9 -9
View File
@@ -22,26 +22,26 @@ API than you are likely to write from scratch.
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank">
<img width="296px" src="static/neon.jpg"> <img width="296px" src="static/supabase.svg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://tembo.io/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://www.euronodes.com/postgrest" target="_blank">
<img width="296px" src="static/tembo.png"> <img width="296px" src="static/euronodes.svg">
</a> </a>
</td> </td>
</tr> </tr>
<tr></tr> <tr></tr>
<tr> <tr>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://www.euronodes.com/postgrest" target="_blank"> <a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="static/euronodes.svg"> <img width="296px" src="static/neon.jpg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank"> <a href="https://www.bytebase.com/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="static/supabase.png"> <img width="296px" src="static/bytebase.svg">
</a> </a>
</td> </td>
</tr> </tr>
@@ -125,7 +125,7 @@ and limited with - range headers. More about
## Data Integrity ## Data Integrity
Rather than relying on an Object Relational Mapper and custom Rather than relying on an Object Relational Mapper and custom
imperative coding, this system requires you put declarative constraints imperative coding, this system requires you to put declarative constraints
directly into your database. Hence no application can corrupt your directly into your database. Hence no application can corrupt your
data (including your API server). data (including your API server).
+5
View File
@@ -1,2 +1,7 @@
packages: postgrest.cabal packages: postgrest.cabal
tests: true tests: true
allow-newer:
hasql:postgresql-libpq
-- https://github.com/martijnbastiaan/doctest-parallel/blob/main/example/README.md#cabalproject
write-ghc-environment-files: always
+1 -1
View File
@@ -1 +1 @@
index-state: hackage.haskell.org 2025-10-29T04:02:18Z index-state: hackage.haskell.org 2026-08-10T16:58:32Z
+28 -4
View File
@@ -1,6 +1,6 @@
{ system ? builtins.currentSystem { system ? builtins.currentSystem
, compiler ? "ghc948" , compiler ? "ghc9123"
, # Commit of the Nixpkgs repository that we want to use. , # Commit of the Nixpkgs repository that we want to use.
# It defaults to reading the inputs from flake.lock, which serves # It defaults to reading the inputs from flake.lock, which serves
@@ -44,7 +44,6 @@ let
allOverlays.checked-shell-script allOverlays.checked-shell-script
allOverlays.gitignore allOverlays.gitignore
(allOverlays.haskell-packages { inherit compiler; }) (allOverlays.haskell-packages { inherit compiler; })
allOverlays.slocat
]; ];
# Evaluated expression of the Nixpkgs repository. # Evaluated expression of the Nixpkgs repository.
@@ -53,11 +52,20 @@ let
postgresqlVersions = postgresqlVersions =
[ [
{ name = "pg-19"; postgresql = pkgs.postgresql_19.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "pg-18"; postgresql = pkgs.postgresql_18.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "pg-17"; postgresql = pkgs.postgresql_17.withPackages (p: [ p.postgis p.pg_safeupdate ]); } { name = "pg-17"; postgresql = pkgs.postgresql_17.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "pg-16"; postgresql = pkgs.postgresql_16.withPackages (p: [ p.postgis p.pg_safeupdate ]); } { name = "pg-16"; postgresql = pkgs.postgresql_16.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "pg-15"; postgresql = pkgs.postgresql_15.withPackages (p: [ p.postgis p.pg_safeupdate ]); } { name = "pg-15"; postgresql = pkgs.postgresql_15.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "pg-14"; postgresql = pkgs.postgresql_14.withPackages (p: [ p.postgis p.pg_safeupdate ]); } { name = "pg-14"; postgresql = pkgs.postgresql_14.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "pg-13"; postgresql = pkgs.postgresql_13.withPackages (p: [ p.postgis p.pg_safeupdate ]); } {
name = "oriole-18";
postgresql = pkgs.orioledb.withPackages (p: [ p.postgis p.pg_safeupdate ]);
config = "
default_table_access_method = 'orioledb'
shared_preload_libraries = 'orioledb, pg_stat_statements'
";
}
]; ];
haskellPackages = pkgs.haskell.packages."${compiler}"; haskellPackages = pkgs.haskell.packages."${compiler}";
@@ -78,6 +86,16 @@ let
"-f dev --test-show-detail=direct"; "-f dev --test-show-detail=direct";
inherit (pkgs.haskell) lib; inherit (pkgs.haskell) lib;
nixos-lib = import (pkgs.path + "/nixos/lib") { };
runTest = postgrest: test: (nixos-lib.runTest {
hostPkgs = pkgs;
# Replace the top-level `pkgs.postgrest` attribute with our current version on this branch.
defaults.nixpkgs.overlays = [ (_: _: { inherit postgrest; }) ];
# Speeds up evaluation a little bit; documentation is really not required for tests.
defaults.documentation.enable = pkgs.lib.mkDefault false;
imports = [ test ];
}).config.result;
in in
rec { rec {
inherit nixpkgs pkgs; inherit nixpkgs pkgs;
@@ -108,6 +126,9 @@ rec {
inherit (pkgs.haskell.packages."${compiler}") ghcWithPackages; inherit (pkgs.haskell.packages."${compiler}") ghcWithPackages;
}; };
# Used by CI on MacOS
inherit (pkgs) nix-build-uncached;
### Tools ### Tools
cabalTools = cabalTools =
@@ -118,7 +139,7 @@ rec {
# Development tools. # Development tools.
devTools = devTools =
pkgs.callPackage nix/tools/devTools.nix { inherit tests style devCabalOptions hsie withTools; }; pkgs.callPackage nix/tools/devTools.nix { inherit tests style devCabalOptions hsie; };
# Documentation tools. # Documentation tools.
docs = docs =
@@ -160,4 +181,7 @@ rec {
# Docker images and loading script. # Docker images and loading script.
docker = docker =
pkgs.callPackage nix/tools/docker { postgrest = postgrestStatic; }; pkgs.callPackage nix/tools/docker { postgrest = postgrestStatic; };
# NixOS VM tests
nixpkgs-nixos-test = runTest postgrestStatic (pkgs.path + "/nixos/tests/postgrest.nix");
} }
+8 -10
View File
@@ -19,26 +19,26 @@ write from scratch.
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank">
<img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/neon.jpg"> <img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/supabase.svg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://tembo.io/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://www.euronodes.com/postgrest" target="_blank">
<img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/tembo.png"> <img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/euronodes.svg">
</a> </a>
</td> </td>
</tr> </tr>
<tr></tr> <tr></tr>
<tr> <tr>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://www.euronodes.com/postgrest" target="_blank"> <a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/euronodes.svg"> <img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/neon.jpg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank"> <a href="https://www.bytebase.com/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/supabase.png"> <img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/bytebase.svg">
</a> </a>
</td> </td>
</tr> </tr>
@@ -61,5 +61,3 @@ The image is built from scratch using
no commands are listed in the image history. See the [PostgREST no commands are listed in the image history. See the [PostgREST
repository](https://github.com/PostgREST/postgrest/tree/main/nix/tools/docker) for repository](https://github.com/PostgREST/postgrest/tree/main/nix/tools/docker) for
details on the build process and how to inspect the image. details on the build process and how to inspect the image.
This does not apply to the arm64 variant, which is based on Ubuntu.
+2 -2
View File
@@ -87,8 +87,8 @@ url of Authorization is [[../explanations/db_authz.html]]
url of CLI is [[../references/cli.html#cli]] url of CLI is [[../references/cli.html#cli]]
url of "Connection Pool" is [[../references/connection_pool.html]] url of "Connection Pool" is [[../references/connection_pool.html]]
url of Config is [[../references/configuration.html#configuration]] url of Config is [[../references/configuration.html#configuration]]
url of HTTPADMIN is [[../explanations/architecture.html#http]] url of HTTPADMIN is [[../references/http_server.html]]
url of HTTPAPI is [[../explanations/architecture.html#http]] url of HTTPAPI is [[../references/http_server.html]]
url of Listener is [[../references/listener.html#listener]] url of Listener is [[../references/listener.html#listener]]
url of Proxy is [[../explanations/nginx.html]] url of Proxy is [[../explanations/nginx.html]]
url of "Schema Cache" is [[../references/schema_cache.html#schema-cache]] url of "Schema Cache" is [[../references/schema_cache.html#schema-cache]]
+1 -1
View File
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 28 KiB

After

Width:  |  Height:  |  Size: 31 KiB

+1 -1
View File
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 29 KiB

After

Width:  |  Height:  |  Size: 32 KiB

+1 -1
View File
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 4.8 KiB

After

Width:  |  Height:  |  Size: 5.6 KiB

+1 -1
View File
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 5.1 KiB

After

Width:  |  Height:  |  Size: 5.8 KiB

+5 -3
View File
@@ -48,14 +48,14 @@ source_suffix = ".rst"
# The master toctree document. # The master toctree document.
master_doc = "index" master_doc = "index"
# This is overriden by readthedocs with the version tag anyway # This is overridden by readthedocs with the version tag anyway
version = "14" version = "devel"
# To avoid repetition in <title> we set this to an empty string. # To avoid repetition in <title> we set this to an empty string.
release = "" release = ""
# General information about the project. # General information about the project.
project = "PostgREST " + version project = "PostgREST " + version
author = "Joe Nelson, Steve Chavez" author = "The PostgREST contributors"
copyright = "2017, " + author copyright = "2017, " + author
# The language for content autogenerated by Sphinx. Refer to documentation # The language for content autogenerated by Sphinx. Refer to documentation
@@ -300,8 +300,10 @@ linkcheck_ignore = [
r"https://www.patreon.com/postgrest", r"https://www.patreon.com/postgrest",
r"https://blog.frankel.ch/poor-man-api", r"https://blog.frankel.ch/poor-man-api",
r"https://www.cybertec-postgresql.com/.*", r"https://www.cybertec-postgresql.com/.*",
r"https://stackoverflow.com/.*",
# Odd SSL error # Odd SSL error
r"https://www.dripdepot.com", r"https://www.dripdepot.com",
r"https://www.euronodes.com",
# New GitHub UI delays comment load, so anchor fails # New GitHub UI delays comment load, so anchor fails
r"https://github.com/.*#issuecomment", r"https://github.com/.*#issuecomment",
# Random 500 Internal Server Error # Random 500 Internal Server Error
+3 -1
View File
@@ -6,7 +6,7 @@ Community Tutorials
* `Building a Contacts List with PostgREST and Vue.js <https://www.youtube.com/watch?v=iHtsALtD5-U>`_ - * `Building a Contacts List with PostgREST and Vue.js <https://www.youtube.com/watch?v=iHtsALtD5-U>`_ -
In this video series, DigitalOcean shows how to build and deploy an Nginx + PostgREST(using a managed PostgreSQL database) + Vue.js webapp in an Ubuntu server droplet. In this video series, DigitalOcean shows how to build and deploy an Nginx + PostgREST(using a managed PostgreSQL database) + Vue.js webapp in an Ubuntu server droplet.
* `PostgREST + Auth0: Create REST API in mintutes, and add social login using Auth0 <https://samkhawase.com/blog/postgrest/>`_ - A step-by-step tutorial to show how to dockerize and integrate Auth0 to PostgREST service. * `PostgREST + Auth0: Create REST API in minutes, and add social login using Auth0 <https://samkhawase.com/blog/postgrest-1-introduction/>`_ - A step-by-step tutorial to show how to dockerize and integrate Auth0 to PostgREST service.
* `"CodeLess" backend using postgres, postgrest and oauth2 authentication with keycloak <https://www.mathieupassenaud.fr/codeless_backend/>`_ - * `"CodeLess" backend using postgres, postgrest and oauth2 authentication with keycloak <https://www.mathieupassenaud.fr/codeless_backend/>`_ -
A step-by-step tutorial for using PostgREST with KeyCloak(hosted on a managed service). A step-by-step tutorial for using PostgREST with KeyCloak(hosted on a managed service).
@@ -37,6 +37,7 @@ Example Apps
* `archtika <https://github.com/thiloho/archtika>`_ - self-hosted CMS * `archtika <https://github.com/thiloho/archtika>`_ - self-hosted CMS
* `delibrium-postgrest <https://gitlab.com/delibrium/delibrium-postgrest/>`_ - example school API and front-end in Vue.js * `delibrium-postgrest <https://gitlab.com/delibrium/delibrium-postgrest/>`_ - example school API and front-end in Vue.js
* `ETH-transactions-storage <https://github.com/Adamant-im/ETH-transactions-storage>`_ - indexer for Ethereum to get transaction list by ETH address * `ETH-transactions-storage <https://github.com/Adamant-im/ETH-transactions-storage>`_ - indexer for Ethereum to get transaction list by ETH address
* `fullstack template <https://github.com/jenstroeger/fullstack-webapp-template>`_ - a complete fullstack webapp template using PG as db and message queue, Python and Dramatiq to implement async jobs, db migrations, test runners, and more.
* `general <https://github.com/PierreRochard/general>`_ - example auth back-end * `general <https://github.com/PierreRochard/general>`_ - example auth back-end
* `guild-operators <https://github.com/cardano-community/koios-artifacts/tree/main/files/grest>`_ - example queries and functions that the Cardano Community uses for their Guild Operators' Repository * `guild-operators <https://github.com/cardano-community/koios-artifacts/tree/main/files/grest>`_ - example queries and functions that the Cardano Community uses for their Guild Operators' Repository
* `PostGUI <https://github.com/priyank-purohit/PostGUI>`_ - React Material UI admin panel * `PostGUI <https://github.com/priyank-purohit/PostGUI>`_ - React Material UI admin panel
@@ -82,6 +83,7 @@ Extensions
Client-Side Libraries Client-Side Libraries
--------------------- ---------------------
* `efcore-postgrest <https://github.com/pedro-gilmora/EF.PostgREST.Provider>`_ - C#
* `postgrest-csharp <https://github.com/supabase-community/postgrest-csharp>`_ - C# * `postgrest-csharp <https://github.com/supabase-community/postgrest-csharp>`_ - C#
* `postgrest-dart <https://github.com/supabase/postgrest-dart>`_ - Dart * `postgrest-dart <https://github.com/supabase/postgrest-dart>`_ - Dart
* `postgrest-ex <https://github.com/supabase-community/postgrest-ex>`_ - Elixir * `postgrest-ex <https://github.com/supabase-community/postgrest-ex>`_ - Elixir
+11 -16
View File
@@ -31,65 +31,60 @@ This section talks briefly about various important modules.
Main Main
---- ----
The starting point of the program is `Main.hs <https://github.com/PostgREST/postgrest/blob/main/main/Main.hs>`_. The starting point of the program is `Main.hs <https://github.com/PostgREST/postgrest/blob/main/src/executable/Main.hs>`_.
CLI CLI
--- ---
Main then calls `CLI.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/CLI.hs>`_, which is in charge of :ref:`cli`. Main then calls `CLI.hs <https://github.com/PostgREST/postgrest/blob/main/src/library/PostgREST/CLI.hs>`_, which is in charge of :ref:`cli`.
App App
--- ---
`App.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/App.hs>`_ is then in charge of composing the different modules. `App.hs <https://github.com/PostgREST/postgrest/blob/main/src/library/PostgREST/App.hs>`_ is then in charge of composing the different modules.
Auth Auth
---- ----
`Auth.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Auth.hs>`_ is in charge of :ref:`authn`. `Auth.hs <https://github.com/PostgREST/postgrest/blob/main/src/library/PostgREST/Auth.hs>`_ is in charge of :ref:`authn`.
Api Request Api Request
----------- -----------
`ApiRequest.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/ApiRequest.hs>`_ is in charge of parsing the URL query string (following PostgREST syntax), the request headers, and the request body. `ApiRequest.hs <https://github.com/PostgREST/postgrest/blob/main/src/library/PostgREST/ApiRequest.hs>`_ is in charge of parsing the URL query string (following PostgREST syntax), the request headers, and the request body.
A request might be rejected at this level if it's invalid. For example when providing an unknown media type to PostgREST or using an unknown HTTP method. A request might be rejected at this level if it's invalid. For example when providing an unknown media type to PostgREST or using an unknown HTTP method.
Plan Plan
---- ----
Using the Schema Cache, `Plan.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Plan.hs>`_ generates an internal AST, filling out-of-band SQL details (like an ``ON CONFLICT (pk)`` clause) required to complete the user request. Using the Schema Cache, `Plan.hs <https://github.com/PostgREST/postgrest/blob/main/src/library/PostgREST/Plan.hs>`_ generates an internal AST, filling out-of-band SQL details (like an ``ON CONFLICT (pk)`` clause) required to complete the user request.
A request might be rejected at this level if it's invalid. For example when doing resource embedding on a nonexistent resource. A request might be rejected at this level if it's invalid. For example when doing resource embedding on a nonexistent resource.
Query Query
----- -----
`Query.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Query.hs>`_ generates the SQL queries (parametrized and prepared) required to satisfy the user request. `Query.hs <https://github.com/PostgREST/postgrest/blob/main/src/library/PostgREST/Query.hs>`_ generates the SQL queries (parametrized and prepared) required to satisfy the user request.
Only at this stage a connection from the pool might be used. Only at this stage a connection from the pool might be used.
Schema Cache Schema Cache
------------ ------------
`SchemaCache.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/SchemaCache.hs>`_ is in charge of :ref:`schema_cache`. `SchemaCache.hs <https://github.com/PostgREST/postgrest/blob/main/src/library/PostgREST/SchemaCache.hs>`_ is in charge of :ref:`schema_cache`.
Config Config
------ ------
`Config.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Config.hs>`_ is in charge of :ref:`configuration`. `Config.hs <https://github.com/PostgREST/postgrest/blob/main/src/library/PostgREST/Config.hs>`_ is in charge of :ref:`configuration`.
Admin Admin
----- -----
`Admin.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Admin.hs>`_ is in charge of the :ref:`admin_server`. `Admin.hs <https://github.com/PostgREST/postgrest/blob/main/src/library/PostgREST/Admin.hs>`_ is in charge of the :ref:`admin_server`.
HTTP
----
The HTTP server is provided by `Warp <https://aosabook.org/en/posa/warp.html>`_.
Listener Listener
-------- --------
`Listener.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Listener.hs>`_ is in charge of the :ref:`listener`. `Reload.hs <https://github.com/PostgREST/postgrest/blob/main/src/library/PostgREST/AppState/Reload.hs>`_ is in charge of the :ref:`listener`.
+1 -1
View File
@@ -163,7 +163,7 @@ Another option is to define the function with the :code:`SECURITY DEFINER` optio
.. code-block:: postgres .. code-block:: postgres
-- login as a user wich has privileges on the private schemas -- login as a user which has privileges on the private schemas
-- create a sample function -- create a sample function
create or replace function login(email text, pass text, out token text) as $$ create or replace function login(email text, pass text, out token text) as $$
+18 -1
View File
@@ -16,7 +16,7 @@ Supported PostgreSQL versions
============================= =============================
=============== ================================= =============== =================================
**Supported** PostgreSQL >= 12 **Supported** PostgreSQL >= 14
=============== ================================= =============== =================================
PostgREST works with all PostgreSQL versions still `officially supported <https://www.postgresql.org/support/versioning/>`_. PostgREST works with all PostgreSQL versions still `officially supported <https://www.postgresql.org/support/versioning/>`_.
@@ -181,6 +181,23 @@ If you want to have a visual overview of your API in your browser you can add sw
With this you can see the swagger-ui in your browser on port 8080. With this you can see the swagger-ui in your browser on port 8080.
.. _docker_cpu_contraint:
Docker Resource Constraints
---------------------------
PostgREST does not support ``--cpus`` `constraint option <https://docs.docker.com/engine/containers/resource_constraints/#configure-the-default-cfs-scheduler>`_.
As a workaround, you may use the `GHC RTS <https://ghc.gitlab.haskell.org/ghc/doc/users_guide/runtime_control.html#runtime-system-rts-options>`_ ``-N`` option. For instance, to limit it to 2 CPU cores, do:
.. code::
# Set environment variable GHCRTS set to "-N2"
docker run --rm -p 3000:3000 \
-e PGRST_DB_URI="postgres://app_user:password@10.0.0.10/postgres" \
-e GHCRTS="-N2"
postgrest/postgrest
.. _build_source: .. _build_source:
Building from Source Building from Source
@@ -0,0 +1,62 @@
.. _debugging_performance_pg_stat_statements:
Debugging Performance with pg_stat_statements
=============================================
This how-to shows how to get a query identifier through PostgREST and then use it to inspect the same query in ``pg_stat_statements``.
.. important::
- :ref:`db-plan-enabled` must be enabled in PostgREST.
- PostgreSQL 14 or newer with ``pg_stat_statements`` available.
Get the Query Identifier from PostgREST
---------------------------------------
Request the plan in JSON format with the ``verbose`` option:
.. code-block:: bash
curl "http://localhost:3000/projects?select=id,name&order=id" \
-H "Accept: application/vnd.pgrst.plan+json; options=verbose"
The response will contain a top-level ``Query Identifier`` field:
.. code-block:: json
[
{
"Plan": {
"Node Type": "Aggregate"
},
"Query Identifier": -432192689578025496
}
]
Look up the query in pg_stat_statements
---------------------------------------
Use that identifier against ``pg_stat_statements``:
.. code-block:: postgres
select
calls,
total_exec_time,
mean_exec_time,
rows,
query
from pg_stat_statements
where queryid = -432192689578025496;
.. csv-table::
:header: "calls", "total_exec_time", "mean_exec_time", "rows", "query"
"13", "0.6355850000000001", "0.04889115384615385", "13", "WITH pgrst_source AS (...)"
This lets you correlate a PostgREST request with PostgreSQL runtime statistics such as:
- how often the query ran
- total and average execution time
- how many rows it produced
- the normalized SQL text recorded by PostgreSQL
@@ -318,144 +318,6 @@ You can insert a new product using a JSON object for the ``extra_info`` column:
To query and filter the data see :ref:`json_columns` for a complete reference. To query and filter the data see :ref:`json_columns` for a complete reference.
.. _ww_postgis:
PostGIS
-------
You can use the string representation for `PostGIS <https://postgis.net/>`_ data types such as ``geometry`` or ``geography`` (you need to `install PostGIS <https://postgis.net/documentation/getting_started/>`_ first).
.. code-block:: postgres
-- Activate the postgis module in the current database
create extension if not exists postgis;
create table coverage (
id int primary key,
name text unique,
area geometry
);
To add areas in polygon format, you can use string representation:
.. code-block:: bash
curl "http://localhost:3000/coverage" \
-X POST -H "Content-Type: application/json" \
-d @- << EOF
[
{ "id": 1, "name": "small", "area": "SRID=4326;POLYGON((0 0, 1 0, 1 1, 0 1, 0 0))" },
{ "id": 2, "name": "big", "area": "SRID=4326;POLYGON((0 0, 10 0, 10 10, 0 10, 0 0))" }
]
EOF
Now, when you request the information, PostgREST will automatically cast the ``area`` column into a ``Polygon`` geometry type. Although this is useful, you may need the whole output to be in `GeoJSON <https://geojson.org/>`_ format out of the box, which can be done by including the ``Accept: application/geo+json`` in the request. This will work for PostGIS versions 3.0.0 and up and will return the output as a `FeatureCollection Object <https://www.rfc-editor.org/rfc/rfc7946#section-3.3>`_:
.. code-block:: bash
curl "http://localhost:3000/coverage" \
-H "Accept: application/geo+json"
.. code-block:: json
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[1,0],[1,1],[0,1],[0,0]]
]
},
"properties": {
"id": 1,
"name": "small"
}
},
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[10,0],[10,10],[0,10],[0,0]]
]
},
"properties": {
"id": 2,
"name": "big"
}
}
]
}
If you need to add an extra property, like the area in square units by using ``st_area(area)``, you could add a generated column to the table and it will appear in the ``properties`` key of each ``Feature``.
.. code-block:: postgres
alter table coverage
add square_units double precision generated always as ( st_area(area) ) stored;
In the case that you are using older PostGIS versions, then creating a function is your best option:
.. code-block:: postgres
create or replace function coverage_geo_collection() returns json as $$
select
json_build_object(
'type', 'FeatureCollection',
'features', json_agg(
json_build_object(
'type', 'Feature',
'geometry', st_AsGeoJSON(c.area)::json,
'properties', json_build_object('id', c.id, 'name', c.name)
)
)
)
from coverage c;
$$ language sql;
Now this query will return the same results:
.. code-block:: bash
curl "http://localhost:3000/rpc/coverage_geo_collection"
.. code-block:: json
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[1,0],[1,1],[0,1],[0,0]]
]
},
"properties": {
"id": 1,
"name": "small"
}
},
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[10,0],[10,10],[0,10],[0,0]]
]
},
"properties": {
"id": 2,
"name": "big"
}
}
]
}
Ranges Ranges
------ ------
@@ -609,3 +471,20 @@ You can use other comparative filters and also all the `PostgreSQL special date/
"due_date": "2022-02-27T06:00:00-05:00" "due_date": "2022-02-27T06:00:00-05:00"
} }
] ]
.. raw:: html
<script type="text/javascript">
let hash = window.location.hash;
const redirects = {
// PostGIS
'#postgis': '../integrations/postgis.html#postgis',
};
let willRedirectTo = redirects[hash];
if (willRedirectTo) {
window.location.href = willRedirectTo;
}
</script>
+32 -23
View File
@@ -38,6 +38,28 @@ Sponsors
.. image:: ../static/cybertec.svg .. image:: ../static/cybertec.svg
:target: https://www.cybertec-postgresql.com/en/?utm_source=postgrest.org&utm_medium=referral&utm_campaign=postgrest :target: https://www.cybertec-postgresql.com/en/?utm_source=postgrest.org&utm_medium=referral&utm_campaign=postgrest
.. container:: img-dark
.. image:: ../static/supabase-dark.svg
:target: https://supabase.com/?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage
.. container:: img-light
.. image:: ../static/supabase.svg
:target: https://supabase.com/?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage
.. container:: img-dark
.. image:: ../static/euronodes.svg
:target: https://www.euronodes.com/postgrest
.. container:: img-light
.. image:: ../static/euronodes.svg
:target: https://www.euronodes.com/postgrest
|
.. container:: img-dark .. container:: img-dark
.. image:: ../static/neon-dark.jpg .. image:: ../static/neon-dark.jpg
@@ -48,30 +70,15 @@ Sponsors
.. image:: ../static/neon.jpg .. image:: ../static/neon.jpg
:target: https://neon.com/?utm_source=sponsor&utm_campaign=postgrest :target: https://neon.com/?utm_source=sponsor&utm_campaign=postgrest
.. image:: ../static/tembo.png
:target: https://www.tembo.io/?utm_source=sponsor&utm_campaign=postgrest
|
.. container:: img-dark .. container:: img-dark
.. image:: ../static/euronodes.svg .. image:: ../static/bytebase-dark.svg
:target: https://www.euronodes.com/postgrest :target: https://www.bytebase.com/?utm_source=sponsor&utm_campaign=postgrest
.. container:: img-light .. container:: img-light
.. image:: ../static/euronodes.svg .. image:: ../static/bytebase.svg
:target: https://www.euronodes.com/postgrest :target: https://www.bytebase.com/?utm_source=sponsor&utm_campaign=postgrest
.. container:: img-dark
.. image:: ../static/supabase-dark.png
:target: https://supabase.com/?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage
.. container:: img-light
.. image:: ../static/supabase.png
:target: https://supabase.com/?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage
.. The static/empty.png(created with `convert -size 320x95 xc:#fcfcfc empty.png`) is an ugly workaround .. The static/empty.png(created with `convert -size 320x95 xc:#fcfcfc empty.png`) is an ugly workaround
to create space and center the logos. It's not easy to layout with restructuredText. to create space and center the logos. It's not easy to layout with restructuredText.
@@ -112,11 +119,14 @@ Releases
PostgREST follows ``MAJOR.PATCH`` two-part versioning: PostgREST follows ``MAJOR.PATCH`` two-part versioning:
- ``MAJOR``: feature release, may deprecate or remove things. - ``MAJOR``: feature release, may deprecate or remove things.
- ``PATCH``: fix/security release only; no features, no behavior changes. - ``PATCH``: fix/security release only, no features and no behavior changes.
Starting from ``v14.0``, only even-numbered MAJOR versions will be released, reserving odd-numbered MAJOR versions for development. MAJOR releases are published twice a year, with their scope and target dates tracked through `GitHub milestones <https://github.com/PostgREST/postgrest/milestones>`_.
PATCH releases are published on an as-needed basis.
All the releases are published on `PostgREST's GitHub release page <https://github.com/PostgREST/postgrest/releases>`_. Starting from ``v14.0``, only even-numbered MAJOR versions are released, reserving odd-numbered MAJOR versions for development.
All releases are published on `PostgREST's GitHub release page <https://github.com/PostgREST/postgrest/releases>`_, along with the corresponding upgrade guides.
Tutorials Tutorials
--------- ---------
@@ -213,7 +223,6 @@ In Production
Here are some companies that use PostgREST in production. Here are some companies that use PostgREST in production.
* `Catarse <https://www.catarse.me>`_ * `Catarse <https://www.catarse.me>`_
* `Datrium <https://www.datrium.com>`_
* `Drip Depot <https://www.dripdepot.com>`_ * `Drip Depot <https://www.dripdepot.com>`_
* `Image-charts <https://www.image-charts.com>`_ * `Image-charts <https://www.image-charts.com>`_
* `Netwo <https://www.netwo.io>`_ * `Netwo <https://www.netwo.io>`_
+36
View File
@@ -0,0 +1,36 @@
NixOS
=====
Nixpkgs contains a `NixOS module to run PostgREST <https://search.nixos.org/options?channel=unstable&query=services.postgrest&type=options>`_, which can be enabled with ``services.postgrest.enable = true``.
A PostgreSQL server can be enabled on the same machine with ``services.postgresql.enable = true``. Connections will use the name of the system user as user and database names by default, in this case ``postgrest``.
A minimal example could look like this:
.. code-block:: nix
{
pkgs,
...
}:
{
services.postgresql = {
enable = true;
initialScript = pkgs.writeText "init.sql" ''
CREATE ROLE postgrest LOGIN NOINHERIT;
CREATE ROLE anon ROLE postgrest;
'';
};
services.postgrest = {
enable = true;
settings.db-anon-role = "anon";
settings.db-uri.dbname = "postgres";
};
}
This will expose the PostgREST server on localhost on the NixOS machine and allow anonymous access.
.. tip::
NixOS also allows to quickly spin up different PostgreSQL versions or even forks this way. For example, to test the current beta version of `OrioleDB <https://www.orioledb.com>`_, use ``services.postgresql.package = pkgs.orioledb``.
+154
View File
@@ -0,0 +1,154 @@
.. _ww_postgis:
PostGIS
=======
To work with `PostGIS <https://postgis.net/>`_ data types such as ``geometry`` or ``geography``, you'll need to `install PostGIS <https://postgis.net/documentation/getting_started/>`_ first.
.. code-block:: postgres
-- Activate the postgis module in the current database
create extension if not exists postgis;
create table coverage (
id int primary key,
name text unique,
area geometry
);
insert into coverage (id, name, area) values
(1, 'small', ST_GeomFromText('POLYGON((0 0, 1 0, 1 1, 0 1, 0 0))',4326)),
(2, 'big', ST_GeomFromText('POLYGON((0 0, 10 0, 10 10, 0 10, 0 0))', 4326);
.. _application/geo+json:
``application/geo+json``
------------------------
PostgREST supports the `standard <https://www.iana.org/assignments/media-types/application/geo+json>`_ ``application/geo+json`` media type which can be used to get the output in `GeoJSON <https://geojson.org/>`_ format. This will work for PostGIS versions 3.0.0 and up and will return the output as a `FeatureCollection Object <https://www.rfc-editor.org/rfc/rfc7946#section-3.3>`_:
.. code-block:: bash
curl "http://localhost:3000/coverage" \
-H "Accept: application/geo+json"
.. code-block:: json
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[1,0],[1,1],[0,1],[0,0]]
]
},
"properties": {
"id": 1,
"name": "small"
}
},
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[10,0],[10,10],[0,10],[0,0]]
]
},
"properties": {
"id": 2,
"name": "big"
}
}
]
}
Using generated columns
-----------------------
If you need to add an extra property, like the area in square units by using ``st_area(area)``, you could add a generated column to the table and it will appear in the ``properties`` key of each ``Feature``.
.. code-block:: postgres
alter table coverage
add square_units double precision generated always as ( st_area(area) ) stored;
In the case that you are using older PostGIS versions, then creating a function is your best option:
.. code-block:: postgres
create or replace function coverage_geo_collection() returns json as $$
select
json_build_object(
'type', 'FeatureCollection',
'features', json_agg(
json_build_object(
'type', 'Feature',
'geometry', st_AsGeoJSON(c.area)::json,
'properties', json_build_object('id', c.id, 'name', c.name)
)
)
)
from coverage c;
$$ language sql;
Now this query will return the same results:
.. code-block:: bash
curl "http://localhost:3000/rpc/coverage_geo_collection"
.. code-block:: json
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[1,0],[1,1],[0,1],[0,0]]
]
},
"properties": {
"id": 1,
"name": "small"
}
},
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[10,0],[10,10],[0,10],[0,0]]
]
},
"properties": {
"id": 2,
"name": "big"
}
}
]
}
Using string representation
---------------------------
To insert areas in polygon format, you can use string representation:
.. code-block:: bash
curl "http://localhost:3000/coverage" \
-X POST -H "Content-Type: application/json" \
-d @- << EOF
[
{ "id": 3, "name": "strip", "area": "SRID=4326;POLYGON((0 0, 50 0, 50 2, 0 2, 0 0))" },
{ "id": 4, "name": "diamond", "area": "SRID=4326;POLYGON((5 0, 10 5, 5 10, 0 5, 5 0))" }
]
EOF
PostgREST will automatically cast the ``area`` column into a ``Polygon`` geometry type.
+14 -2
View File
@@ -1,9 +1,11 @@
personal_ws-1.1 en 0 utf-8 personal_ws-1.1 en 0 utf-8
api api
autoscaling
API's API's
APIs APIs
APISIX APISIX
AST AST
async
aud aud
Auth Auth
auth auth
@@ -14,6 +16,7 @@ BOM
Bytea Bytea
Cardano Cardano
cd cd
CDNs
centric centric
CLI CLI
CMS CMS
@@ -28,10 +31,11 @@ CSV
durations durations
DDL DDL
DOM DOM
DSL
DevOps DevOps
Dramatiq
dockerize dockerize
enum enum
ECS
Enums Enums
Entra Entra
eq eq
@@ -41,7 +45,10 @@ EveryLayout
filename filename
FreeBSD FreeBSD
fts fts
fullstack
GC
GeoJSON GeoJSON
GHC
Github Github
Google Google
grantor grantor
@@ -70,7 +77,6 @@ isdistinct
JS JS
js js
JSON JSON
JSPath
JWK JWK
JWT JWT
jwt jwt
@@ -93,6 +99,7 @@ namespaced
Nanos Nanos
neq neq
nginx nginx
NixOS
nixpkgs nixpkgs
npm npm
nxl nxl
@@ -138,11 +145,13 @@ Redux
refactor refactor
reloadable reloadable
Reloadable Reloadable
reuseport
requester's requester's
RESTful RESTful
RLS RLS
RPC RPC
RSA RSA
RTS
safeupdate safeupdate
savepoint savepoint
schemas schemas
@@ -188,9 +197,12 @@ verifier
versioning versioning
Vondra Vondra
Vue Vue
webapp
webhooks webhooks
websearch websearch
Websockets Websockets
webuser webuser
wfts wfts
www www
debouncing
deduplicates
+3 -2
View File
@@ -3,7 +3,9 @@
Admin Server Admin Server
############ ############
PostgREST provides an admin server that can be enabled by setting :ref:`admin-server-port`. PostgREST provides an admin server that can be enabled by setting :ref:`admin-server-port` or :ref:`admin-server-unix-socket`.
Multiple PostgREST instances can share the same public API host and port when :ref:`server-reuseport` is enabled. Admin ports are not shared: give each instance a different :ref:`admin-server-port`, otherwise the new instance will fail to start.
.. _health_check: .. _health_check:
@@ -72,5 +74,4 @@ Provides the ``schema_cache`` endpoint that prints the runtime :ref:`schema_cach
"dbRepresentations": ["..."], "dbRepresentations": ["..."],
"dbRoutines": ["..."], "dbRoutines": ["..."],
"dbTables": ["..."], "dbTables": ["..."],
"dbTimezones": ["..."]
} }
+1
View File
@@ -21,6 +21,7 @@ PostgREST exposes three database objects of a schema as resources: tables, views
api/aggregate_functions.rst api/aggregate_functions.rst
api/openapi.rst api/openapi.rst
api/preferences.rst api/preferences.rst
api/vary_header.rst
api/* api/*
.. raw:: html .. raw:: html
+20
View File
@@ -69,6 +69,26 @@ If the function doesn't modify the database, it will also run under the GET meth
The function parameter names match the JSON object keys in the POST case, for the GET case they match the query parameters ``?a=1&b=2``. The function parameter names match the JSON object keys in the POST case, for the GET case they match the query parameters ``?a=1&b=2``.
If the function is defined to have default values for the parameters then arguments for these parameters can be omitted in the request. For instance:
.. code-block:: postgres
CREATE FUNCTION greet_user(username TEXT DEFAULT 'guest')
RETURNS TEXT AS $$
SELECT 'Hello ' || username || '!';
$$ LANGUAGE SQL IMMUTABLE;
.. code-block:: bash
curl -i "http://localhost:3000/rpc/greet_user"
.. code-block:: http
HTTP/1.1 200 OK
Context-Type: application/json; charset=utf-8
"Hello guest!"
.. _function_single_json: .. _function_single_json:
Functions with an array of JSON objects Functions with an array of JSON objects
+1 -1
View File
@@ -15,7 +15,7 @@ Using these domains, :ref:`functions <functions>` can become handlers and `user-
.. important:: .. important::
- PostgREST vendor media types (``application/vnd.pgrst.plan``, ``application/vnd.pgrst.object`` and ``application/vnd.pgrst.array``) cannot be overriden. - PostgREST vendor media types (``application/vnd.pgrst.plan``, ``application/vnd.pgrst.object`` and ``application/vnd.pgrst.array``) cannot be overridden.
- Long media types like ``application/vnd.openxmlformats-officedocument.wordprocessingml.document`` cannot be expressed as domains since they surpass `PostgreSQL identifier length <https://www.postgresql.org/docs/current/limits.html#LIMITS-TABLE>`_. - Long media types like ``application/vnd.openxmlformats-officedocument.wordprocessingml.document`` cannot be expressed as domains since they surpass `PostgreSQL identifier length <https://www.postgresql.org/docs/current/limits.html#LIMITS-TABLE>`_.
For these you can use the :ref:`any_handler`. For these you can use the :ref:`any_handler`.
+14 -9
View File
@@ -62,8 +62,12 @@ The server ignores unrecognized or unfulfillable preferences by default. You can
Timezone Timezone
======== ========
The ``timezone`` preference allows you to change the `PostgreSQL timezone <https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-TIMEZONE>`_. It accepts all time zones in `pg_timezone_names <https://www.postgresql.org/docs/current/view-pg-timezone-names.html>`_. .. important::
``handling=lenient`` is ignored for ``timezone``. Invalid time zones always return an error.
The ``timezone`` preference allows you to change the `PostgreSQL timezone <https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-TIMEZONE>`_.
It accepts all time zones in `pg_timezone_names <https://www.postgresql.org/docs/current/view-pg-timezone-names.html>`_ and numeric offsets.
.. code-block:: bash .. code-block:: bash
@@ -84,34 +88,35 @@ The ``timezone`` preference allows you to change the `PostgreSQL timezone <https
{"t":"2023-10-18T09:37:59.611-07:00"} {"t":"2023-10-18T09:37:59.611-07:00"}
] ]
For an invalid time zone, PostgREST returns values with the default time zone (configured on ``postgresql.conf`` or as a setting on the :ref:`authenticator <roles>`). Offsets are also accepted:
.. code-block:: bash .. code-block:: bash
curl -i "http://localhost:3000/timestamps" \ curl -i "http://localhost:3000/timestamps" \
-H "Prefer: timezone=Jupiter/Red_Spot" -H "Prefer: timezone=05:30"
.. code-block:: http .. code-block:: http
HTTP/1.1 200 OK HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8 Content-Type: application/json; charset=utf-8
Preference-Applied: timezone=05:30
.. code-block:: json .. code-block:: json
[ [
{"t":"2023-10-18T12:37:59.611+00:00"}, {"t":"2023-10-18T17:07:59.611+05:30"},
{"t":"2023-10-18T14:37:59.611+00:00"}, {"t":"2023-10-18T19:07:59.611+05:30"},
{"t":"2023-10-18T16:37:59.611+00:00"} {"t":"2023-10-18T21:07:59.611+05:30"}
] ]
Note that there's no ``Preference-Applied`` in the response. You can also use negative offsets like ``-03:00``.
However, with ``handling=strict``, an invalid time zone preference will throw an :ref:`error <pgrst122>`. For an invalid time zone, PostgREST returns a database error.
.. code-block:: bash .. code-block:: bash
curl -i "http://localhost:3000/timestamps" \ curl -i "http://localhost:3000/timestamps" \
-H "Prefer: handling=strict, timezone=Jupiter/Red_Spot" -H "Prefer: timezone=Jupiter/Red_Spot"
.. code-block:: http .. code-block:: http
+1 -1
View File
@@ -1244,7 +1244,7 @@ You can order the correlated arrays explicitly. For example, to order by the fil
.. warning:: .. warning::
Aliasing spreaded columns is recommended since JSON allows duplicate keys. Example: Aliasing spread columns is recommended since JSON allows duplicate keys. Example:
.. code-block:: bash .. code-block:: bash
@@ -51,7 +51,7 @@ Builtin handlers are offered for common standard media types.
* ``text/csv`` and ``application/json``, for all API endpoints. See :ref:`tables_views` and :ref:`functions`. * ``text/csv`` and ``application/json``, for all API endpoints. See :ref:`tables_views` and :ref:`functions`.
* ``application/openapi+json``, for the root endpoint. See :ref:`open-api`. * ``application/openapi+json``, for the root endpoint. See :ref:`open-api`.
* ``application/geo+json``, see :ref:`ww_postgis`. * ``application/geo+json``, see :ref:`application/geo+json`.
* ``*/*``, resolves to ``application/json`` for API endpoints and to ``application/openapi+json`` for the root endpoint. * ``*/*``, resolves to ``application/json`` for API endpoints and to ``application/openapi+json`` for the root endpoint.
The following vendor media types handlers are also supported. The following vendor media types handlers are also supported.
+4
View File
@@ -5,6 +5,10 @@ Schemas
PostgREST can expose a single or multiple schema's tables, views and functions. The :ref:`active database role <roles>` must have the usage privilege on the schemas to access them. PostgREST can expose a single or multiple schema's tables, views and functions. The :ref:`active database role <roles>` must have the usage privilege on the schemas to access them.
.. important::
``pg_catalog`` and ``information_schema`` are not allowed in :ref:`db-schemas`. This is done to prevent leaking sensitive information and hence they cannot be accessed directly. If you wish to expose objects of these schemas, expose another schema that contains wrapper views or functions over ``pg_catalog`` or ``information_schema`` objects.
Single schema Single schema
------------- -------------
+1 -1
View File
@@ -639,7 +639,7 @@ However, it can work with surrogate primary keys (e.g. ``id serial primary key``
.. code-block:: bash .. code-block:: bash
curl "http://localhost:3000/employees?colums=id,name,salary" \ curl "http://localhost:3000/employees?columns=id,name,salary" \
-X POST -H "Content-Type: application/json" \ -X POST -H "Content-Type: application/json" \
-H "Prefer: resolution=merge-duplicates, missing=default" \ -H "Prefer: resolution=merge-duplicates, missing=default" \
-d @- << EOF -d @- << EOF
+2 -2
View File
@@ -14,7 +14,7 @@ Custom Queries
The PostgREST URL grammar limits the kinds of queries clients can perform. It prevents arbitrary, potentially poorly constructed and slow client queries. It's good for quality of service, but means database administrators must create custom views and functions to provide richer endpoints. The most common causes for custom endpoints are The PostgREST URL grammar limits the kinds of queries clients can perform. It prevents arbitrary, potentially poorly constructed and slow client queries. It's good for quality of service, but means database administrators must create custom views and functions to provide richer endpoints. The most common causes for custom endpoints are
* Table unions * SET operators like `UNION, INTERSECT and EXCEPT <https://www.postgresql.org/docs/current/queries-union.html>`_.
* More complicated joins than those provided by :ref:`resource_embedding`. * More complicated joins than those provided by :ref:`resource_embedding`.
* Geo-spatial queries that require an argument, like "points near (lat,lon)" * Geo-spatial queries that require an argument, like "points near (lat,lon)"
@@ -51,7 +51,7 @@ You can request table/columns with spaces in them by percent encoding the spaces
Reserved characters Reserved characters
~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
If filters include PostgREST reserved characters(``,``, ``.``, ``:``, ``()``) you'll have to surround them in percent encoded double quotes ``%22`` for correct processing. If filters include PostgREST reserved characters(``,``, ``.``, ``:``, ``*``, ``(``, ``)``) you'll have to surround them in percent encoded double quotes ``%22`` for correct processing.
Here ``Hebdon,John`` and ``Williams,Mary`` are values. Here ``Hebdon,John`` and ``Williams,Mary`` are values.
+16
View File
@@ -0,0 +1,16 @@
.. _vary_header:
Vary Header
===========
In order to assist caching proxies and CDNs, PostgREST includes a ``Vary`` header of value
``Accept, Prefer, Range`` in its responses which should fit most of the bills. As any other
response header, it's available for override
by updating ``response.headers`` GUC variable accordingly, for example:
.. code-block:: postgres
-- Override the Vary header to include Accept, Prefer and X-Test-Vary headers
perform set_config('response.headers', '[{"Vary": "Accept, Prefer, X-Test-Vary"}]', true);
In this case PostgREST will use provided value verbatim.
+12 -21
View File
@@ -217,47 +217,38 @@ It's recommended to leave the JWT cache enabled as our load tests indicate ~20%
- If the ``jwt-secret`` is changed and the config is reloaded, the JWT cache will reset. - If the ``jwt-secret`` is changed and the config is reloaded, the JWT cache will reset.
- JWTs that pass :ref:`jwt_signature` are cached, regardless if they pass :ref:`jwt_claims_validation`. We do this to ensure responses stays fast under common failure cases (such as expired JWTs). - JWTs that pass :ref:`jwt_signature` are cached, regardless if they pass :ref:`jwt_claims_validation`. We do this to ensure responses stays fast under common failure cases (such as expired JWTs).
- You can use the :ref:`server-timing_header` to see the peformance benefit of JWT caching. - You can use the :ref:`server-timing_header` to see the performance benefit of JWT caching.
.. _jwt_role_extract: .. _jwt_role_extract:
JWT Role Extraction JWT Role Extraction
------------------- -------------------
A JSPath DSL that specifies the location of the :code:`role` key in the JWT claims. It's configured by :ref:`jwt-role-claim-key`. This can be used to consume a JWT provided by a third party service like Auth0, Okta, Microsoft Entra or Keycloak. A JSON Path (`RFC 9535 <https://www.rfc-editor.org/rfc/rfc9535.html>`_) can be specified for the location of the :code:`role` key in the JWT claims. It's configured by :ref:`jwt-role-claim-key`. This can be used to consume a JWT provided by a third party service like Auth0, Okta, Microsoft Entra or Keycloak.
The DSL follows the `JSONPath <https://goessner.net/articles/JsonPath/>`_ expression grammar with extended string comparison operators. Supported operators are: You can quickly try out JSON Path by visiting https://serdejsonpath.live.
- ``==`` selects the first array element that exactly matches the right operand
- ``!=`` selects the first array element that does not match the right operand
- ``^==`` selects the first array element that starts with the right operand
- ``==^`` selects the first array element that ends with the right operand
- ``*==`` selects the first array element that contains the right operand
Usage examples: Usage examples:
.. code:: bash .. code:: bash
# {"postgrest":{"roles": ["other", "author"]}} # {"postgrest":{"roles": ["other", "author"]}}
# the DSL accepts characters that are alphanumerical or one of "_$@" as keys jwt-role-claim-key = "$$.postgrest.roles[1]"
jwt-role-claim-key = ".postgrest.roles[1]"
# {"https://www.example.com/role": { "key": "author" }} # {"https://www.example.com/role": { "key": "author" }}
# non-alphanumerical characters can go inside quotes(escaped in the config value) # non-alphanumerical characters can go inside single quotes
jwt-role-claim-key = ".\"https://www.example.com/role\".key" jwt-role-claim-key = "$$['https://www.example.com/role'].key"
# {"postgrest":{"roles": ["other", "author"]}} # {"postgrest":{"roles": ["other", "author"]}}
# `@` represents the current element in the array # filter based on equality or regular expression
# all the these match the string "author" jwt-role-claim-key = "$$.postgrest.roles[?(@ == 'author')]"
jwt-role-claim-key = ".postgrest.roles[?(@ == \"author\")]" jwt-role-claim-key = "$$.postgrest.roles[?search(@, '^au')]"
jwt-role-claim-key = ".postgrest.roles[?(@ != \"other\")]"
jwt-role-claim-key = ".postgrest.roles[?(@ ^== \"aut\")]"
jwt-role-claim-key = ".postgrest.roles[?(@ ==^ \"hor\")]"
jwt-role-claim-key = ".postgrest.roles[?(@ *== \"utho\")]"
.. note:: .. note::
The string comparison operators are implemented as a custom extension to the JSPath and does not strictly follow the `RFC 9535 <https://www.rfc-editor.org/rfc/rfc9535.html>`_. - If JSON Path query returns multiple values, the first one gets selected.
- Only when using the :ref:`file_config`, all ``$`` characters in the value must be escaped with an additional ``$`` char. For :ref:`env_variables_config` and :ref:`in_db_config`, only use a single ``$`` char.
- In our implementation, only the `search()` function from `JSON Path Functions <https://www.rfc-editor.org/rfc/rfc9535.html#name-function-extensions>`_ is available for filtering.
JWT Security JWT Security
------------ ------------
+154 -5
View File
@@ -176,6 +176,46 @@ admin-server-port
Specifies the port for the :ref:`admin_server`. Cannot be equal to :ref:`server-port`. Specifies the port for the :ref:`admin_server`. Cannot be equal to :ref:`server-port`.
.. _admin-server-unix-socket:
admin-server-unix-socket
------------------------
=============== =================================
**Type** String
**Default** `n/a`
**Reloadable** N
**Environment** PGRST_ADMIN_SERVER_UNIX_SOCKET
**In-Database** `n/a`
=============== =================================
`Unix domain socket <https://en.wikipedia.org/wiki/Unix_domain_socket>`_ where to bind the :ref:`admin_server`.
If specified, this takes precedence over :ref:`admin-server-port`. Example:
.. code:: bash
admin-server-unix-socket = "/tmp/pgrst-admin.sock"
.. _admin-server-unix-socket-mode:
admin-server-unix-socket-mode
-----------------------------
=============== ===================================
**Type** String
**Default** 660
**Reloadable** N
**Environment** PGRST_ADMIN_SERVER_UNIX_SOCKET_MODE
**In-Database** `n/a`
=============== ===================================
`Unix file mode <https://en.wikipedia.org/wiki/File_system_permissions>`_ to be set for the socket specified in :ref:`admin-server-unix-socket`
Needs to be a valid octal between 600 and 777.
.. code:: bash
admin-server-unix-socket-mode = "660"
.. _app.settings.*: .. _app.settings.*:
app.settings.* app.settings.*
@@ -195,6 +235,33 @@ app.settings.*
The :code:`current_setting` function has `an optional boolean second <https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-SET>`_ argument to avoid it from raising an error if the value was not defined. Default values to :code:`app.settings` can then be given by combining this argument with :code:`coalesce` and :code:`nullif` : :code:`coalesce(nullif(current_setting('app.settings.my_custom_variable', true), ''), 'default value')`. The use of :code:`nullif` is necessary because if set in a transaction, the setting is sometimes not "rolled back" to :code:`null`. See also :ref:`this section <guc_req_headers_cookies_claims>` for more information on this behaviour. The :code:`current_setting` function has `an optional boolean second <https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-SET>`_ argument to avoid it from raising an error if the value was not defined. Default values to :code:`app.settings` can then be given by combining this argument with :code:`coalesce` and :code:`nullif` : :code:`coalesce(nullif(current_setting('app.settings.my_custom_variable', true), ''), 'default value')`. The use of :code:`nullif` is necessary because if set in a transaction, the setting is sometimes not "rolled back" to :code:`null`. See also :ref:`this section <guc_req_headers_cookies_claims>` for more information on this behaviour.
.. _client-error-verbosity:
client-error-verbosity
----------------------
=============== =======================
**Type** String
**Default** verbose
**Reloadable** Y
**Environment** PGRST_CLIENT_ERROR_VERBOSITY
**In-Database** pgrst.client_error_verbosity
=============== =======================
Specifies the verbosity of PostgREST errors. See :ref:`client_error_verbosity`.
.. code:: bash
# Return error "code", "message", "details" and "hint"
client-error-verbosity = "verbose"
# Return only "code" and "message"
client-error-verbosity = "minimal"
.. note::
This setting only affects client side error messages. Server side logs are not affected by this setting.
.. _db-aggregates-enabled: .. _db-aggregates-enabled:
db-aggregates-enabled db-aggregates-enabled
@@ -264,7 +331,7 @@ db-channel-enabled
When this is set to :code:`true`, the notification channel specified in :ref:`db-channel` is enabled. When this is set to :code:`true`, the notification channel specified in :ref:`db-channel` is enabled.
You should set this to ``false`` when using PostgresSQL behind an external connection pooler such as PgBouncer working in transaction pooling mode. See :ref:`this section <external_connection_poolers>` for more information. You should set this to ``false`` when using PostgreSQL behind an external connection pooler such as PgBouncer working in transaction pooling mode. See :ref:`this section <external_connection_poolers>` for more information.
.. _db-config: .. _db-config:
@@ -479,7 +546,7 @@ db-prepared-statements
When disabled, the generated queries will be parameterized (invulnerable to SQL injection) but they will not be prepared (cached in the database session). Not using prepared statements will noticeably decrease performance, so it's recommended to always have this setting enabled. When disabled, the generated queries will be parameterized (invulnerable to SQL injection) but they will not be prepared (cached in the database session). Not using prepared statements will noticeably decrease performance, so it's recommended to always have this setting enabled.
You should only set this to ``false`` when using PostgresSQL behind an external connection pooler such as PgBouncer working in transaction pooling mode. See :ref:`this section <external_connection_poolers>` for more information. You should only set this to ``false`` when using PostgreSQL behind an external connection pooler such as PgBouncer working in transaction pooling mode. See :ref:`this section <external_connection_poolers>` for more information.
.. _db-root-spec: .. _db-root-spec:
@@ -526,7 +593,7 @@ db-tx-end
**In-Database** pgrst.db_tx_end **In-Database** pgrst.db_tx_end
=============== ================================= =============== =================================
Specifies how to terminate the database transactions. Specifies how to terminate the database transactions. See :ref:`prefer_tx`.
.. code:: bash .. code:: bash
@@ -612,7 +679,7 @@ jwt-role-claim-key
=============== ================================= =============== =================================
**Type** String **Type** String
**Default** .role **Default** $.role
**Reloadable** Y **Reloadable** Y
**Environment** PGRST_JWT_ROLE_CLAIM_KEY **Environment** PGRST_JWT_ROLE_CLAIM_KEY
**In-Database** pgrst.jwt_role_claim_key **In-Database** pgrst.jwt_role_claim_key
@@ -622,6 +689,10 @@ jwt-role-claim-key
See :ref:`jwt_role_extract` on how to specify key paths and usage examples. See :ref:`jwt_role_extract` on how to specify key paths and usage examples.
.. warning::
Only when using :ref:`file_config`, the ``$`` char needs to be escaped, so use ``$$`` and PostgREST will interpret it as a single ``$`` character.
.. _jwt-secret: .. _jwt-secret:
jwt-secret jwt-secret
@@ -681,7 +752,7 @@ log-level
=============== ================================= =============== =================================
**Type** String **Type** String
**Default** error **Default** error
**Reloadable** N **Reloadable** Y
**Environment** PGRST_LOG_LEVEL **Environment** PGRST_LOG_LEVEL
**In-Database** `n/a` **In-Database** `n/a`
=============== ================================= =============== =================================
@@ -857,6 +928,50 @@ server-port
The TCP port to bind the web server. Use ``0`` to automatically assign a port. The TCP port to bind the web server. Use ``0`` to automatically assign a port.
.. _server-reuseport:
server-reuseport
----------------
=============== =================================
**Type** Bool
**Default** false
**Reloadable** N
**Environment** PGRST_SERVER_REUSEPORT
**In-Database** `n/a`
=============== =================================
Enables ``SO_REUSEPORT`` on the TCP server socket. This allows multiple
PostgREST processes to bind to the same :ref:`server-host` and
:ref:`server-port` when the operating system supports it.
For example, two PostgREST processes can use the same configuration:
.. code:: ini
server-host = "127.0.0.1"
server-port = 3000
server-reuseport = true
New connections are then distributed by the operating system between the
running PostgREST processes. This can be used to start a replacement process
before stopping the old one, or to run several PostgREST processes behind one
port.
If ``server-reuseport`` is disabled, starting another PostgREST process on
the same host and port will fail with the usual address-in-use error.
Enabling this setting on an operating system that does not support
``SO_REUSEPORT`` is a configuration error. PostgREST will fail to start
instead of falling back to a normal TCP socket.
When running multiple PostgREST instances on the same :ref:`server-port`, use
a different ``admin-server-port`` for each instance. Admin ports are not shared
between instances, so readiness checks always target one specific PostgREST
instance.
This setting does not apply when :ref:`server-unix-socket` is used.
.. _server-trace-header: .. _server-trace-header:
server-trace-header server-trace-header
@@ -927,3 +1042,37 @@ server-unix-socket-mode
.. code:: bash .. code:: bash
server-unix-socket-mode = "660" server-unix-socket-mode = "660"
.. _url-use-legacy-target-names:
url-use-legacy-target-names
---------------------------
=============== =================================
**Type** Boolean
**Default** True
**Reloadable** Y
**Environment** PGRST_URL_USE_LEGACY_TARGET_NAMES
**In-Database** pgrst.url_use_legacy_target_names
=============== =================================
When active, it allows using the the name of an embedded table in filters, orders or limits even if it has an alias:
.. code:: bash
curl "http://localhost:3000/table?select=alias:target(*)&target.order=id" -i
.. code:: text
Warning: 299 PostgRESTv16 "Embedded resource was referenced by relation name even though it has an alias. This is deprecated and will stop working in a future release. Update `target` to `alias` in query string filters, orders or limits."
[...]
Note that the response includes a deprecation message in the ``Warning`` header.
This will also show in the PostgREST logs:
.. code::
28/May/2026:20:33:22 -0500: WARNING: Embedded resource was referenced by relation name even though it has an alias. This is deprecated and will stop working in a future release.
28/May/2026:20:33:22 -0500: Update filters, orders or limits that use `target` to `alias` in `GET /table?select=alias:target(*)&target.order=id`
This feature will be removed in a future release, so you should start using the ``alias`` in these cases.
+2
View File
@@ -47,6 +47,8 @@ Under a busy system, the :ref:`db-pool-max-idletime` won't be reached and the co
To avoid this problem and save resources, a connection max lifetime (:ref:`db-pool-max-lifetime`) is enforced. To avoid this problem and save resources, a connection max lifetime (:ref:`db-pool-max-lifetime`) is enforced.
After the max lifetime is reached, connections from the pool will be released and new ones will be created. This doesn't affect running requests, only unused connections will be released. After the max lifetime is reached, connections from the pool will be released and new ones will be created. This doesn't affect running requests, only unused connections will be released.
.. _pool_timeout:
Acquisition Timeout Acquisition Timeout
------------------- -------------------
+36 -1
View File
@@ -199,7 +199,7 @@ Related to the HTTP request elements.
| | | :ref:`switching schemas <multiple-schemas>` is not present | | | | :ref:`switching schemas <multiple-schemas>` is not present |
| PGRST106 | | in the :ref:`db-schemas` configuration variable. | | PGRST106 | | in the :ref:`db-schemas` configuration variable. |
+---------------+-------------+-------------------------------------------------------------+ +---------------+-------------+-------------------------------------------------------------+
| .. _pgrst107: | 415 | The ``Content-Type`` sent in the request is invalid. | | .. _pgrst107: | 406 | The ``Accept`` media type sent in the request is invalid. |
| | | | | | | |
| PGRST107 | | | | PGRST107 | | |
+---------------+-------------+-------------------------------------------------------------+ +---------------+-------------+-------------------------------------------------------------+
@@ -473,3 +473,38 @@ For example, doing a request on a table with high count (say 30_000_000), we get
Proxy-Status: PostgREST; error=57014 Proxy-Status: PostgREST; error=57014
The PostgreSQL error code ``57014`` (`ref <https://www.postgresql.org/docs/current/errcodes-appendix.html>`_) reveals that the error is due to a short ``statement_timeout`` value. The PostgreSQL error code ``57014`` (`ref <https://www.postgresql.org/docs/current/errcodes-appendix.html>`_) reveals that the error is due to a short ``statement_timeout`` value.
.. _client_error_verbosity:
Client Error Verbosity
======================
For HTTP clients, the error verbosity can be set via :ref:`client-error-verbosity` config.
With ``verbose``, it returns ``code``, ``message``, ``details`` and ``hint``.
.. code:: bash
curl "localhost:3000/itemsxx"
.. code-block:: json
{
"code": "PGRST205",
"message": "Could not find the table 'public.itemsxx' in the schema cache",
"details": "Perhaps you meant the table 'public.items'",
"hint": null
}
With ``minimal``, just ``code`` and ``message`` is returned.
.. code:: bash
curl "localhost:3000/itemsxx"
.. code-block:: json
{
"code": "PGRST205",
"message": "Could not find the table 'public.itemsxx' in the schema cache"
}
+18
View File
@@ -0,0 +1,18 @@
.. _http_server:
HTTP Server
###########
The HTTP server is provided by `Warp <https://aosabook.org/en/posa/warp.html>`_.
Graceful shutdown
-----------------
PostgREST uses Warp's graceful shutdown, when a ``SIGTERM`` is received:
- It stops accepting new requests.
- Allows requests that are already in progress to finish.
- Closes idle ``Keep-Alive`` connections instead of waiting for them to expire.
- Responses sent during shutdown indicate that the connection should not be reused (e.g. for HTTP/1.x, it sends ``Connection: close``).
This allows PostgREST to shut down promptly without interrupting in-flight requests. Useful for zero-downtime upgrades and autoscaling/load-balancing under cloud environments (AWS ECS, Kubernetes).
+3 -1
View File
@@ -46,7 +46,9 @@ This will cause the :ref:`connection_pool` to connect to the read replica host a
.. note:: .. note::
Under the hood, PostgREST forces `target_session_attrs=read-write <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS>`_ for the ``LISTEN`` session. - Under the hood, PostgREST forces `target_session_attrs=read-write <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS>`_ for the ``LISTEN`` session.
So if you specify ``target_session_attrs=read-only`` as mentioned above, PostgREST will override it for the ``LISTEN``.
- ``read-only`` is only available on libpq >= 14, if you use a lower version you will get an error like ``invalid target_session_attrs value: \"read-only\"``.
.. _listener_automatic_recovery: .. _listener_automatic_recovery:
+43 -2
View File
@@ -39,7 +39,7 @@ For diagnostic information about the server itself, PostgREST logs to ``stderr``
06/May/2024:08:16:11 -0500: Listening for database notifications on the "pgrst" channel 06/May/2024:08:16:11 -0500: Listening for database notifications on the "pgrst" channel
06/May/2024:08:16:11 -0500: Config reloaded 06/May/2024:08:16:11 -0500: Config reloaded
06/May/2024:08:16:11 -0500: Schema cache queried in 3.8 milliseconds 06/May/2024:08:16:11 -0500: Schema cache queried in 3.8 milliseconds
06/May/2024:08:16:11 -0500: Schema cache loaded 15 Relations, 8 Relationships, 8 Functions, 0 Domain Representations, 4 Media Type Handlers 06/May/2024:08:16:11 -0500: Schema cache loaded 15 Relations, 8 Relationships, 8 RPCs, 0 Domain Representations, 4 Media Type Handlers
06/May/2024:14:11:27 -0500: Received a config reload message on the "pgrst" channel 06/May/2024:14:11:27 -0500: Received a config reload message on the "pgrst" channel
06/May/2024:14:11:27 -0500: Config reloaded 06/May/2024:14:11:27 -0500: Config reloaded
@@ -120,7 +120,7 @@ Restart the database and watch the log file in real-time to understand how HTTP
Metrics Metrics
======= =======
The ``metrics`` endpoint on the :ref:`admin_server` endpoint provides metrics in `Prometheus text format <https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format>`_. The ``metrics`` endpoint on the :ref:`admin_server` endpoint provides metrics in `Prometheus text format <https://prometheus.io/docs/instrumenting/exposition_formats/#prometheus-text-format>`_.
.. code-block:: bash .. code-block:: bash
@@ -238,6 +238,45 @@ pgrst_jwt_cache_evictions_total
The total number of JWT cache evictions. The total number of JWT cache evictions.
GHC Runtime Metrics
-------------------
PostgREST can also expose GHC runtime system metrics. These use the ``ghc_*``
prefix and include
`GHC RTS statistics <https://ghc.gitlab.haskell.org/ghc/doc/libraries/base-4.22.0.0-inplace/GHC-Stats.html#g:1>`_
for runtime allocation, garbage collection, memory, and CPU/elapsed time.
These are useful for monitoring PostgREST process health and diagnosing memory
pressure or GC behavior.
To expose these metrics, enable GHC RTS statistics when starting PostgREST:
.. code-block:: bash
postgrest +RTS -T -RTS
When enabled, the admin ``/metrics`` endpoint includes samples such as:
.. code-block:: text
# HELP ghc_gcs_total Total number of GCs
# TYPE ghc_gcs_total counter
ghc_gcs_total 1
# HELP ghc_allocated_bytes_total Total bytes allocated
# TYPE ghc_allocated_bytes_total counter
ghc_allocated_bytes_total 12345678
Other available GHC runtime metrics include:
- ``ghc_gcs_total``
- ``ghc_major_gcs_total``
- ``ghc_allocated_bytes_total``
- ``ghc_max_live_bytes``
- ``ghc_max_mem_in_use_bytes``
- ``ghc_mutator_cpu_seconds_total``
- ``ghc_gc_cpu_seconds_total``
- ``ghc_elapsed_seconds_total``
Traces Traces
====== ======
@@ -391,6 +430,8 @@ By default the plan is assumed to generate the JSON representation of a resource
The other available parameters are ``analyze``, ``verbose``, ``settings``, ``buffers`` and ``wal``, which correspond to the `EXPLAIN command options <https://www.postgresql.org/docs/current/sql-explain.html>`_. To use the ``analyze`` and ``wal`` parameters for example, you would add them like ``Accept: application/vnd.pgrst.plan; options=analyze|wal``. The other available parameters are ``analyze``, ``verbose``, ``settings``, ``buffers`` and ``wal``, which correspond to the `EXPLAIN command options <https://www.postgresql.org/docs/current/sql-explain.html>`_. To use the ``analyze`` and ``wal`` parameters for example, you would add them like ``Accept: application/vnd.pgrst.plan; options=analyze|wal``.
For a workflow that takes the ``Query Identifier`` from a verbose PostgREST plan and uses it to inspect the same query in ``pg_stat_statements``, see :ref:`debugging_performance_pg_stat_statements`.
Note that akin to the EXPLAIN command, the changes will be committed when using the ``analyze`` option. To avoid this, you can use the :ref:`db-tx-end` and the ``Prefer: tx=rollback`` header. Note that akin to the EXPLAIN command, the changes will be committed when using the ``analyze`` option. To avoid this, you can use the :ref:`db-tx-end` and the ``Prefer: tx=rollback`` header.
Securing the Execution Plan Securing the Execution Plan
+21 -2
View File
@@ -3,10 +3,16 @@
Schema Cache Schema Cache
============ ============
PostgREST requires metadata from the database schema to provide a REST API that abstracts SQL details. One example of this is the interface for :ref:`resource_embedding`. PostgREST requires metadata from the database to provide a REST API that abstracts SQL details. One example of this is the interface for :ref:`resource_embedding`.
Getting this metadata requires expensive queries. To avoid repeating this work, PostgREST uses a schema cache. Getting this metadata requires expensive queries. To avoid repeating this work, PostgREST uses a schema cache.
.. note::
- Schema cache queries have been optimized over time to stay fast, even on complex databases. You can see a summary of their execution time in :ref:`pgrst_logging` and :ref:`metrics`.
- If the schema cache queries are slow, the most likely cause is *system catalog bloat*, see `issue#3212 <https://github.com/PostgREST/postgrest/issues/3212>`_ for more details.
- You can turn the :ref:`log-level` to ``debug`` to see the time of each schema cache query.
.. _schema_reloading: .. _schema_reloading:
Schema Cache Reloading Schema Cache Reloading
@@ -18,7 +24,7 @@ You can do this with UNIX signals or with PostgreSQL notifications. It's also po
.. note:: .. note::
- Requests will wait until the schema cache reload is done. This to prevent client errors due to an stale schema cache. - If the schema cache fails to reload (e.g. due to a ``statement_timeout`` or :ref:`pool timeout <pool_timeout>`), PostgREST will continue serving requests in a "best effort" basis.
- If you are using the :ref:`in_db_config`, a schema cache reload will :ref:`reload the configuration<config_reloading>` as well. - If you are using the :ref:`in_db_config`, a schema cache reload will :ref:`reload the configuration<config_reloading>` as well.
.. _schema_reloading_signals: .. _schema_reloading_signals:
@@ -53,6 +59,19 @@ To reload the schema cache from within the database, you can use the ``NOTIFY``
NOTIFY pgrst, 'reload schema' NOTIFY pgrst, 'reload schema'
Debouncing
~~~~~~~~~~
PostgREST does not reload the schema cache for each notification when several ``NOTIFY pgrst`` events are generated quickly after one another.
There are two cases to consider: when notifications are sent within a single transaction and when they are sent across multiple transactions.
In the first case, PostgreSQL deduplicates identical ``NOTIFY`` events within the same transaction. This means that even if multiple ``NOTIFY pgrst`` statements are executed before a ``COMMIT``, only a single notification is delivered to PostgREST.
In the second case, when notifications are sent from separate transactions in a short time span, PostgREST applies a debouncing mechanism to avoid excessive schema cache reloads.
Instead of reloading the schema cache for each notification, events are grouped within a small time window of 100 milliseconds. The reload function is executed once immediately when the first notification is received and once more after the burst of events settles, resulting in at most two executions within that time window.
.. _auto_schema_reloading: .. _auto_schema_reloading:
Automatic Schema Cache Reloading Automatic Schema Cache Reloading
+1 -1
View File
@@ -221,7 +221,7 @@ Notice that the ``response.headers`` should be set to an *array* of single-key o
.. note:: .. note::
PostgREST provided headers such as ``Content-Type``, ``Location``, etc. can be overriden this way. Note that irrespective of overridden ``Content-Type`` response header, the content will still be converted to JSON, unless you use :ref:`custom_media`. PostgREST provided headers such as ``Content-Type``, ``Location``, etc. can be overridden this way. Note that irrespective of overridden ``Content-Type`` response header, the content will still be converted to JSON, unless you use :ref:`custom_media`.
.. _guc_resp_status: .. _guc_resp_status:
+4 -4
View File
@@ -1,7 +1,7 @@
# This file is auto-generated by postgrest-nixpkgs-upgrade # This file is auto-generated by postgrest-nixpkgs-upgrade
sphinx==8.2.3 sphinx==9.1.0
sphinx-copybutton==0.5.2 sphinx-copybutton==0.5.2
sphinx-rtd-dark-mode==1.3.0 sphinx-rtd-dark-mode==1.3.0
sphinx-rtd-theme==3.0.2 sphinx-rtd-theme==3.1.0
sphinx-tabs==3.4.7 sphinx-tabs==3.5.0
sphinxext-opengraph==0.9.1 sphinxext-opengraph==0.13.0
+1 -1
View File
@@ -22,7 +22,7 @@ Step 1. Install PostgreSQL
If you're already familiar with using PostgreSQL and have it installed on your system you can use the existing installation (see :ref:`pg-dependency` for minimum requirements). For this tutorial we'll describe how to use the database in Docker because database configuration is otherwise too complicated for a simple tutorial. If you're already familiar with using PostgreSQL and have it installed on your system you can use the existing installation (see :ref:`pg-dependency` for minimum requirements). For this tutorial we'll describe how to use the database in Docker because database configuration is otherwise too complicated for a simple tutorial.
If Docker is not installed, you can get it `here <https://www.docker.com/get-started>`_. Next, let's pull and start the database image: If Docker is not installed, you can get it `here <https://www.docker.com/get-started>`_. Make sure that Docker service is `started <https://docs.docker.com/engine/daemon/start/#start-the-daemon-using-operating-system-utilities>`_. Next, let's pull and start the database image:
.. code-block:: bash .. code-block:: bash
+1 -1
View File
@@ -172,7 +172,7 @@ Go back to :ref:`tut1_step3` and change the payload to
.. code-block:: bash .. code-block:: bash
payload=$(echo -n "{\"role\":\"todo_user\",\"exp\":\"123456789\"}" | _base64) payload=$(echo -n "{\"role\":\"todo_user\",\"exp\":123456789}" | _base64)
echo -n "$header.$payload.$signature" echo -n "$header.$payload.$signature"
Generated
+4 -4
View File
@@ -2,16 +2,16 @@
"nodes": { "nodes": {
"nixpkgs": { "nixpkgs": {
"locked": { "locked": {
"lastModified": 1752006229, "lastModified": 1784115452,
"narHash": "sha256-BeuAPwNM2RBc5bvUTb0j4GRs2yBkDeRCw/8Y3v9Xesc=", "narHash": "sha256-BoYPdqk6jlKXy+DyUzyGV/CtRGfAhk2MmIgBhsemTGI=",
"owner": "nixos", "owner": "nixos",
"repo": "nixpkgs", "repo": "nixpkgs",
"rev": "c80edd02003fe3d8af527215a3ac069be9cfd47f", "rev": "35d3407a3816f3b341d8cf1d60abaf2b7b8166ac",
"type": "github" "type": "github"
}, },
"original": { "original": {
"owner": "nixos", "owner": "nixos",
"ref": "nixpkgs-25.05-darwin", "ref": "nixpkgs-unstable",
"repo": "nixpkgs", "repo": "nixpkgs",
"type": "github" "type": "github"
} }
+5 -1
View File
@@ -2,7 +2,7 @@
description = "REST API for any Postgres database"; description = "REST API for any Postgres database";
inputs = { inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-25.05-darwin"; nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
}; };
nixConfig = { nixConfig = {
@@ -46,5 +46,9 @@
meta.description = "REST API for any Postgres database"; meta.description = "REST API for any Postgres database";
}; };
}); });
devShells = genSystems (postgrest: {
default = import ./shell.nix { inherit postgrest; };
});
}; };
} }
+8 -63
View File
@@ -70,55 +70,16 @@ The PostgREST utilities available in `nix-shell` all have names that begin with
`<tab>`) in `nix-shell` to see all that are available: `<tab>`) in `nix-shell` to see all that are available:
```bash ```bash
# Note: The utilities listed here might not be up to date.
[nix-shell]$ postgrest-<tab> [nix-shell]$ postgrest-<tab>
postgrest-build postgrest-parallel-curl postgrest-build
postgrest-check postgrest-profiled-run postgrest-cabal-update
postgrest-clean postgrest-push-cachix postgrest-check
postgrest-commitlint postgrest-release postgrest-clean
postgrest-coverage postgrest-repl postgrest-commitlint
postgrest-coverage-draft-overlay postgrest-run
postgrest-docs-build postgrest-style
postgrest-docs-check postgrest-style-check
postgrest-docs-dictcheck postgrest-test-big-schema
postgrest-docs-linkcheck postgrest-test-doctests
postgrest-docs-render postgrest-test-io
postgrest-docs-serve postgrest-test-memory
postgrest-docs-spellcheck postgrest-test-replica
postgrest-dump-minimal-imports postgrest-test-spec
postgrest-dump-schema postgrest-test-spec-idempotence
postgrest-gen-ctags postgrest-watch
postgrest-gen-jwt postgrest-with-all
postgrest-gen-secret postgrest-with-git
postgrest-git-hooks postgrest-with-pgrst
postgrest-hsie-graph-modules postgrest-with-pg-13
postgrest-hsie-graph-symbols postgrest-with-pg-14
postgrest-hsie-minimal-imports postgrest-with-pg-15
postgrest-lint postgrest-with-pg-16
postgrest-loadtest postgrest-with-pg-17
postgrest-loadtest-against postgrest-with-slow-pg
postgrest-loadtest-report postgrest-with-slow-postgrest
postgrest-nixpkgs-upgrade
... ...
[nix-shell]$
``` ```
The `docker` module has large dependencies to be build before the shell becomes Most of these commands provide a `--help` output, make sure to check it out.
available, which could take an especially long time if the cachix binary cache
is not used. You can activate it by passing a flag to `nix-shell` with
`nix-shell --arg docker true`. This will make the respective utilities available:
```bash
$ nix-shell --arg docker true
[nix-shell]$ postgrest-docker-<tab>
postgrest-docker-load
...
```
Note that `postgrest-docker-load` is now also available.
To run one-off commands, you can also use `nix-shell --run <command>`, which To run one-off commands, you can also use `nix-shell --run <command>`, which
will launch the Nix shell, run that one command and exit. Note that the tab will launch the Nix shell, run that one command and exit. Note that the tab
@@ -134,16 +95,6 @@ $ nix-shell --run "postgrest-foo --bar"
``` ```
A third option is to install utilities that you use very often locally:
```bash
$ nix-env -f default.nix -iA devTools
# `postgrest-style` can now be run directly:
$ postgrest-style
```
If you use `nix-shell` very often, you might like to use If you use `nix-shell` very often, you might like to use
https://github.com/xzfc/cached-nix-shell, which skips evaluating all our Nix https://github.com/xzfc/cached-nix-shell, which skips evaluating all our Nix
expressions if nothing changed, reducing startup time for the shell expressions if nothing changed, reducing startup time for the shell
@@ -174,7 +125,7 @@ $ nix-shell --run "postgrest-with-all postgrest-test-spec"
# Run the tests against a specific version of PostgreSQL (use tab-completion in # Run the tests against a specific version of PostgreSQL (use tab-completion in
# nix-shell to see all available versions): # nix-shell to see all available versions):
$ nix-shell --run "postgrest-with-pg-13 postgrest-test-spec" $ nix-shell --run "postgrest-with-pg-17 postgrest-test-spec"
``` ```
@@ -209,13 +160,7 @@ The loadtests ensure that performance doesn't drop on a change. Underlyingly the
[nix-shell]$ postgrest-loadtest [nix-shell]$ postgrest-loadtest
# You can loadtest comparing to a different branch # You can loadtest comparing to a different branch
[nix-shell]$ postgrest-loadtest-against master [nix-shell]$ postgrest-loadtest-against main
# You can simulate latency client/postgrest and postgrest/database
[nix-shell]$ PGRST_DELAY=5ms PGDELAY=5ms postgrest-loadtest
# You can build postgrest directly with cabal for faster iteration
[nix-shell]$ PGRST_BUILD_CABAL=1 postgrest-loadtest
# Produce a markdown report to be used on CI # Produce a markdown report to be used on CI
[nix-shell]$ postgrest-loadtest-report [nix-shell]$ postgrest-loadtest-report
+6 -24
View File
@@ -16,11 +16,8 @@ The following checklist guides you through the complete process in more detail.
## Upgrade the pinned version of `nixpkgs` ## Upgrade the pinned version of `nixpkgs`
The pinned version of [`nixpkgs`](https://github.com/NixOS/nixpkgs) is defined The pinned version of [`nixpkgs`](https://github.com/NixOS/nixpkgs) is defined
in [`nix/nixpkgs-version.nix`](nixpkgs-version.nix). The pin refers directly to in [`flake.nix`](../flake.nix). To upgrade it, you can use a small utility
a GitHub tarball for the given revision, which is more efficient than pulling script defined in [`nix/tools/nixpkgsTools.nix`](tools/nixpkgsTools.nix):
the complete Git repository. To upgrade it to the current `main` of
`nixpkgs`, you can use a small utility script defined in
[`nix/nixpkgs-update.nix`](nixpkgs-update.nix):
```bash ```bash
# From the root of the repository, enter nix-shell # From the root of the repository, enter nix-shell
@@ -30,21 +27,12 @@ nix-shell
postgrest-nixpkgs-upgrade postgrest-nixpkgs-upgrade
# Exit the nix-shell with Ctrl-d # Exit the nix-shell with Ctrl-d
``` ```
## Review overlays ## Review overlays
Check whether the individual [overlays](overlays) are still required. Check whether the individual [overlays](overlays) are still required.
## Check if patches are still required and update them as needed
We track a number of PostgREST-specific patches in [`nix/patches`](patches).
Check whether the pull-requests/issues linked in the
[`default.nix`](patches/default.nix) have progressed and remove/modify the
patches if they did. If conflicting changes occurred, you might have to rebase
the respective patches.
## Build everything ## Build everything
Using the PostgREST binary Nix cache is recommended. Install Using the PostgREST binary Nix cache is recommended. Install
@@ -58,25 +46,19 @@ errors, this is probably due to one of our patches. Try to fix them and re-run
## Update the PostgREST binary cache ## Update the PostgREST binary cache
If you have access to the PostgREST cachix signing key, you can push the If you have access to the PostgREST cachix project, you can push the
artifacts that you built locally to the binary cache. This will accelerate the artifacts that you built locally to the binary cache. This will accelerate the
CI builds and tests, sometimes dramatically. This might sometimes even be CI builds and tests, sometimes dramatically. This might sometimes even be
required to avoid build timeouts in CI. required to avoid build timeouts in CI.
You'll need to set the `CACHIX_SIGNING_KEY` before proceeding, e.g. by creating You'll need to login with your token with `cachix authtoken <token>`.
a file containing `export CACHIX_SIGNING_KEY=...` and sourcing that file, which
avoids having the secret in your shell history.
To push all new artifacts to Cachix, run: To push all new artifacts to Cachix, run:
``` ```
nix-store -qR --include-outputs $$(nix-instantiate) | cachix push postgrest
# Or, equivalently
nix-shell --run postgrest-push-cachix nix-shell --run postgrest-push-cachix
``` ```
The `nix-store` command will query the nix-store to list all dependencies and The `postgrest-push-cachix` command will query the nix-store to list all
build artifacts of PostgREST. The `cachix` command will efficiently push dependencies and build artifacts of PostgREST. It will then push
everything that is not yet cached to the binary cache. everything that is not yet cached to the binary cache.
+19 -7
View File
@@ -4,6 +4,7 @@
{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE RecordWildCards #-} {-# LANGUAGE RecordWildCards #-}
{-# LANGUAGE TupleSections #-} {-# LANGUAGE TupleSections #-}
{-# LANGUAGE TypeApplications #-}
{-# LANGUAGE TypeFamilies #-} {-# LANGUAGE TypeFamilies #-}
-- | Haskell Imports and Exports tool -- | Haskell Imports and Exports tool
@@ -33,13 +34,15 @@ import Data.Function ((&))
import Data.List (intercalate) import Data.List (intercalate)
import Data.Maybe (catMaybes, mapMaybe) import Data.Maybe (catMaybes, mapMaybe)
import Data.Text (Text) import Data.Text (Text)
import GHC.Driver.Errors.Types (GhcMessage)
import GHC.Generics (Generic) import GHC.Generics (Generic)
import GHC.Hs.Extension (GhcPs) import GHC.Hs.Extension (GhcPs)
import GHC.Types.Error (getMessages) import GHC.Types.Error (Messages, defaultDiagnosticOpts, getMessages)
import GHC.Types.Name.Occurrence (occNameString) import GHC.Types.Name.Occurrence (occNameString)
import GHC.Types.Name.Reader (rdrNameOcc) import GHC.Types.Name.Reader (rdrNameOcc)
import GHC.Unit.Module.Name (moduleNameString) import GHC.Unit.Module (moduleNameString)
import GHC.Utils.Error (pprMsgEnvelopeBagWithLoc) import GHC.Utils.Error (pprMsgEnvelopeBagWithLoc)
import GHC.Utils.Outputable (showSDocUnsafe)
import System.Directory.Recursive (getFilesRecursive) import System.Directory.Recursive (getFilesRecursive)
import System.Exit (exitFailure) import System.Exit (exitFailure)
@@ -198,7 +201,7 @@ sourceSymbols source = do
return $ concatMap (importSymbols source filepath . GHC.unLoc) hsmodImports return $ concatMap (importSymbols source filepath . GHC.unLoc) hsmodImports
-- | Parse a Haskell module -- | Parse a Haskell module
parseModule :: FilePath -> IO GHC.HsModule parseModule :: FilePath -> IO (GHC.HsModule GhcPs)
parseModule filepath = do parseModule filepath = do
result <- ExactPrint.parseModule GHC.Paths.libdir filepath result <- ExactPrint.parseModule GHC.Paths.libdir filepath
case result of case result of
@@ -206,7 +209,13 @@ parseModule filepath = do
return $ GHC.unLoc hsmod return $ GHC.unLoc hsmod
Left errs -> Left errs ->
fail $ "Errors with " <> show filepath <> ":\n " fail $ "Errors with " <> show filepath <> ":\n "
<> show (pprMsgEnvelopeBagWithLoc $ getMessages errs) <> formatParseErrors errs
formatParseErrors :: Messages GhcMessage -> String
formatParseErrors errs =
intercalate "\n "
. fmap showSDocUnsafe
$ pprMsgEnvelopeBagWithLoc (defaultDiagnosticOpts @GhcMessage) (getMessages errs)
-- | Symbols imported in an import declaration. -- | Symbols imported in an import declaration.
-- --
@@ -214,9 +223,12 @@ parseModule filepath = do
-- only one item is returned. -- only one item is returned.
importSymbols :: FilePath -> FilePath -> GHC.ImportDecl GhcPs -> [ImportedSymbol] importSymbols :: FilePath -> FilePath -> GHC.ImportDecl GhcPs -> [ImportedSymbol]
importSymbols source filepath GHC.ImportDecl{..} = importSymbols source filepath GHC.ImportDecl{..} =
case ideclHiding of case ideclImportList of
Just (hiding, syms) -> Just (importListInterpretation, syms) ->
symbol (if hiding then Hiding else Explicit) . Just . GHC.unLoc <$> GHC.unLoc syms symbol (if importListInterpretation == GHC.EverythingBut then Hiding else Explicit)
. Just
. GHC.unLoc
<$> GHC.unLoc syms
Nothing -> Nothing ->
[ symbol Wildcard Nothing ] [ symbol Wildcard Nothing ]
where where
+7 -7
View File
@@ -5,10 +5,10 @@ project. It's available in PostgREST's `nix-shell` by default.
## Dumping imports ## Dumping imports
Given source code in the directories `src` and `main`, for example, you can run: Given source code in the directories `src/library` and `src/executable`, for example, you can run:
``` ```
hsie dump-imports src main hsie dump-imports src/library src/executable
``` ```
This dumps all imports of the modules in the given directory to a CSV file, This dumps all imports of the modules in the given directory to a CSV file,
@@ -18,7 +18,7 @@ To dump to a JSON file (e.g., to further process with `jq`), add the `--json`
flag: flag:
``` ```
hsie dump-imports --json src main hsie dump-imports --json src/library src/executable
``` ```
## Graphing imports ## Graphing imports
@@ -27,7 +27,7 @@ The tool can generate `graphviz` graphs of module and symbol imports by printing
a file to `stdout` that can directly be rendered with `dot`: a file to `stdout` that can directly be rendered with `dot`:
``` ```
hsie graph-modules src main | dot -Tpng -o modules.png hsie graph-modules src/library src/executable | dot -Tpng -o modules.png
``` ```
The command `graph-modules` prints a graph of which modules insert which other The command `graph-modules` prints a graph of which modules insert which other
@@ -39,7 +39,7 @@ To check whether modules are imported under consistent aliases in your project,
run: run:
``` ```
hsie check-aliases main src hsie check-aliases src/library src/executable
``` ```
This will exit with a non-zero exit code if any inconsistent aliases are found. This will exit with a non-zero exit code if any inconsistent aliases are found.
@@ -48,13 +48,13 @@ The following command checks whether any modules are imported as wildcards, i.e.
not qualified and without specifying symbols. not qualified and without specifying symbols.
``` ```
hsie check-wildcards main src hsie check-wildcards src/library src/executable
``` ```
To whitelist certain modules to be imported as wildcards, use `--ok`: To whitelist certain modules to be imported as wildcards, use `--ok`:
``` ```
hsie check-wildcards main src --ok Protolude --ok Test.Module hsie check-wildcards src/library src/executable --ok Protolude --ok Test.Module
``` ```
## Current limitations ## Current limitations
@@ -104,7 +104,7 @@ let
'' ''
+ lib.optionalString withTmpDir '' + lib.optionalString withTmpDir ''
tmpdir="$(${coreutils}/bin/mktemp -d --tmpdir ${name}-XXX)" tmpdir="$(${coreutils}/bin/mktemp -d --tmpdir=/tmp ${name}-XXX)"
# we keep the tmpdir when an error occurs for debugging # we keep the tmpdir when an error occurs for debugging
trap 'echo Temporary directory kept at: $tmpdir' ERR trap 'echo Temporary directory kept at: $tmpdir' ERR
-1
View File
@@ -3,5 +3,4 @@
checked-shell-script = import ./checked-shell-script; checked-shell-script = import ./checked-shell-script;
gitignore = import ./gitignore.nix; gitignore = import ./gitignore.nix;
haskell-packages = import ./haskell-packages.nix; haskell-packages = import ./haskell-packages.nix;
slocat = import ./slocat.nix;
} }
+46 -19
View File
@@ -47,37 +47,64 @@ let
# - To modify and try packages locally, see "Working with locally modified Haskell packages" in the Nix README. # - To modify and try packages locally, see "Working with locally modified Haskell packages" in the Nix README.
# Before upgrading fuzzyset to 0.3, check: https://github.com/PostgREST/postgrest/issues/3329 # Before upgrading fuzzyset to 0.3, check: https://github.com/PostgREST/postgrest/issues/3329
# jailbreak, because hspec limit for tests
fuzzyset = prev.fuzzyset_0_2_4; fuzzyset = prev.fuzzyset_0_2_4;
# TODO: Remove once available in nixpkgs haskellPackages # TODO: Remove once available in nixpkgs
configurator-pg = auto-update =
prev.callHackageDirect prev.callHackageDirect
{ {
pkg = "configurator-pg"; pkg = "auto-update";
ver = "0.2.11"; ver = "0.2.7";
sha256 = "sha256-mtGtNawDJgz2ZIEVca+IYXVu4oNw9xsfJiYWAqAbbgc="; sha256 = "sha256-fHX/OqF/cB9rbpGpLUtA29bcEJS43HUWHcK55yUxKoo=";
} }
{ }; { };
# TODO: Remove once available in nixpkgs haskellPackages # TODO: Remove once available in nixpkgs
streaming-commons = aeson-jsonpath =
prev.callHackageDirect prev.callHackageDirect
{ {
pkg = "streaming-commons"; pkg = "aeson-jsonpath";
ver = "0.2.3.1"; ver = "0.4.2.0";
sha256 = "sha256-Gl2eaJcWe1sxmcE/octWlH9uSnERguf+5H66K4fV87s="; sha256 = "sha256-K+3brf1zjSSjojtSCXFrip5rrP7AO/S4zndAxAnvEfc=";
} }
{ }; { };
# Downgrade hasql and related packages while we are still on GHC 9.4 for the static build. http2 =
hasql = lib.dontCheck (lib.doJailbreak prev.hasql_1_6_4_4); prev.callHackageDirect
hasql-dynamic-statements = lib.dontCheck prev.hasql-dynamic-statements_0_3_1_5; {
hasql-implicits = lib.dontCheck prev.hasql-implicits_0_1_1_3; pkg = "http2";
hasql-notifications = lib.dontCheck prev.hasql-notifications_0_2_2_2; ver = "5.4.0";
hasql-pool = lib.dontCheck prev.hasql-pool_1_0_1; sha256 = "sha256-PeEWVd61bQ8G7LvfLeXklzXqNJFaAjE2ecRMWJZESPE=";
hasql-transaction = lib.dontCheck prev.hasql-transaction_1_1_0_1; }
postgresql-binary = lib.dontCheck (lib.doJailbreak prev.postgresql-binary_0_13_1_3); { };
http-semantics =
prev.callHackageDirect
{
pkg = "http-semantics";
ver = "0.4.0";
sha256 = "sha256-rh0z51EKvsu5rQd5n2z3fSRjjEObouNZSBPO9NFYOF0=";
}
{ };
network-run =
prev.callHackageDirect
{
pkg = "network-run";
ver = "0.5.0";
sha256 = "sha256-vbXh+CzxDsGApjqHxCYf/ijpZtUCApFbkcF5gyN0THU=";
}
{ };
warp =
lib.dontCheck
(prev.callHackageDirect
{
pkg = "warp";
ver = "3.4.14";
sha256 = "sha256-RnoOUlC6dOP0sK/tYAJCX1oLzVFG1GILUY+yVbmvW8Y=";
}
{ });
}; };
in in
{ {
-13
View File
@@ -1,13 +0,0 @@
_: prev:
{
slocat = prev.buildGoModule {
name = "slocat";
src = prev.fetchFromGitHub {
owner = "robx";
repo = "slocat";
rev = "52e7512c6029fd00483e41ccce260a3b4b9b3b64";
sha256 = "sha256-qn6luuh5wqREu3s8RfuMCP5PKdS2WdwPrujRYTpfzQ8=";
};
vendorHash = null;
};
}
+10 -2
View File
@@ -51,7 +51,7 @@ let
docs = "Run PostgREST after building it interactively with cabal-install"; docs = "Run PostgREST after building it interactively with cabal-install";
args = args =
[ [
"ARG_USE_ENV([PGRST_DB_ANON_ROLE], [postgrest_test_anonymous], [PostgREST anonymous role])" "ARG_USE_ENV([PGRST_DB_ANON_ROLE], [], [PostgREST anonymous role. (default: 'postgrest_test_anonymous')])"
"ARG_USE_ENV([PGRST_DB_POOL], [1], [PostgREST pool size])" "ARG_USE_ENV([PGRST_DB_POOL], [1], [PostgREST pool size])"
"ARG_USE_ENV([PGRST_DB_POOL_ACQUISITION_TIMEOUT], [1], [PostgREST pool timeout])" "ARG_USE_ENV([PGRST_DB_POOL_ACQUISITION_TIMEOUT], [1], [PostgREST pool timeout])"
"ARG_USE_ENV([PGRST_JWT_SECRET], [reallyreallyreallyreallyverysafe], [PostgREST JWT secret])" "ARG_USE_ENV([PGRST_JWT_SECRET], [reallyreallyreallyreallyverysafe], [PostgREST JWT secret])"
@@ -62,6 +62,10 @@ let
withEnv = postgrest.env; withEnv = postgrest.env;
} }
'' ''
# when there's a default, argbash conflates empty string with unset, so we do this workaround to be able to do `PGRST_DB_ANON_ROLE="" <command>` for manual testing
if [[ ! ''${PGRST_DB_ANON_ROLE+x} ]]; then
PGRST_DB_ANON_ROLE="postgrest_test_anonymous"
fi
export PGRST_DB_ANON_ROLE export PGRST_DB_ANON_ROLE
export PGRST_DB_POOL export PGRST_DB_POOL
export PGRST_DB_POOL_ACQUISITION_TIMEOUT export PGRST_DB_POOL_ACQUISITION_TIMEOUT
@@ -80,7 +84,7 @@ let
docs = "Run a profiled build of postgREST. This will generate a postgrest.prof file that can be used to do optimization."; docs = "Run a profiled build of postgREST. This will generate a postgrest.prof file that can be used to do optimization.";
args = args =
[ [
"ARG_USE_ENV([PGRST_DB_ANON_ROLE], [postgrest_test_anonymous], [PostgREST anonymous role])" "ARG_USE_ENV([PGRST_DB_ANON_ROLE], [], [PostgREST anonymous role. (default: 'postgrest_test_anonymous')])"
"ARG_USE_ENV([PGRST_DB_POOL], [1], [PostgREST pool size])" "ARG_USE_ENV([PGRST_DB_POOL], [1], [PostgREST pool size])"
"ARG_USE_ENV([PGRST_DB_POOL_ACQUISITION_TIMEOUT], [1], [PostgREST pool timeout])" "ARG_USE_ENV([PGRST_DB_POOL_ACQUISITION_TIMEOUT], [1], [PostgREST pool timeout])"
"ARG_USE_ENV([PGRST_JWT_SECRET], [reallyreallyreallyreallyverysafe], [PostgREST JWT secret])" "ARG_USE_ENV([PGRST_JWT_SECRET], [reallyreallyreallyreallyverysafe], [PostgREST JWT secret])"
@@ -90,6 +94,10 @@ let
withEnv = postgrest.env; withEnv = postgrest.env;
} }
'' ''
# when there's a default, argbash conflates empty string with unset, so we do this workaround to be able to do `PGRST_DB_ANON_ROLE="" <command>` for manual testing
if [[ ! ''${PGRST_DB_ANON_ROLE+x} ]]; then
PGRST_DB_ANON_ROLE="postgrest_test_anonymous"
fi
export PGRST_DB_ANON_ROLE export PGRST_DB_ANON_ROLE
export PGRST_DB_POOL export PGRST_DB_POOL
export PGRST_DB_POOL_ACQUISITION_TIMEOUT export PGRST_DB_POOL_ACQUISITION_TIMEOUT
+5 -157
View File
@@ -5,15 +5,13 @@
, curl , curl
, devCabalOptions , devCabalOptions
, entr , entr
, git , fd
, graphviz , graphviz
, hsie , hsie
, nix , nix
, silver-searcher
, stdenv , stdenv
, style , style
, tests , tests
, withTools
, haskellPackages , haskellPackages
, ctags , ctags
, openssl , openssl
@@ -41,7 +39,7 @@ let
} }
'' ''
while true; do while true; do
(! ${silver-searcher}/bin/ag -l . | ${entr}/bin/entr -dr "$_arg_command" "''${_arg_leftovers[@]}") (! ${fd}/bin/fd -H -E .git | ${entr}/bin/entr -dr "$_arg_command" "''${_arg_leftovers[@]}")
done done
''; '';
@@ -82,6 +80,7 @@ let
} }
'' ''
${tests}/bin/postgrest-test-spec ${tests}/bin/postgrest-test-spec
${tests}/bin/postgrest-test-observability
${tests}/bin/postgrest-test-doctests ${tests}/bin/postgrest-test-doctests
${tests}/bin/postgrest-test-io ${tests}/bin/postgrest-test-io
${tests}/bin/postgrest-test-big-schema ${tests}/bin/postgrest-test-big-schema
@@ -90,156 +89,6 @@ let
${style}/bin/postgrest-style-check ${style}/bin/postgrest-style-check
''; '';
gitHooks =
let
name = "postgrest-git-hooks";
in
checkedShellScript
{
inherit name;
docs =
''
Enable or disable git pre-commit and pre-push hooks.
Basic is faster and will only run:
- pre-commit: postgrest-style
- pre-push: postgrest-lint
Full takes a lot more time and will run:
- pre-commit: postgrest-style && postgrest-lint
- pre-push: postgrest-check
Changes made by postgrest-style will be staged automatically.
Example usage:
postgrest-git-hooks disable
postgrest-git-hooks enable basic
postgrest-git-hooks enable full
The "run" operation and "--hook" argument are only used internally.
'';
args =
[
"ARG_POSITIONAL_SINGLE([operation], [Operation])"
"ARG_TYPE_GROUP_SET([OPERATION], [OPERATION], [operation], [disable,enable,run])"
"ARG_POSITIONAL_SINGLE([mode], [Mode], [basic])"
"ARG_TYPE_GROUP_SET([MODE], [MODE], [mode], [basic,full])"
"ARG_OPTIONAL_SINGLE([hook], , [Hook], [pre-commit])"
"ARG_TYPE_GROUP_SET([HOOK], [HOOK], [hook], [pre-commit,pre-push])"
];
positionalCompletion =
''
if test "$prev" == "${name}"; then
COMPREPLY=( $(compgen -W "enable disable" -- "$cur") )
elif test "$prev" == "enable" || test "$prev" == "disable"; then
COMPREPLY=( $(compgen -W "basic full" -- "$cur") )
fi
'';
workingDir = "/";
}
''
if [ run != "$_arg_operation" ]; then
# Remove all hooks first and ignore failures because the file might be missing.
# This assumes that we're only adding lines that include "postgrest-git-hooks"
# to the hook file.
sed -i -e '/postgrest-git-hooks/d' .git/hooks/pre-{commit,push} 2> /dev/null || true
if [ disable != "$_arg_operation" ]; then
# The nix-shell && + nix-shell || pattern makes sure we can run the hook
# in a pure nix-shell, where nix-shell itself is not available, too.
# The $(nix-shell --run "command -v ...") pattern ensures we only need to enable
# the hooks once and still run the latest of our hook scripts, even when we
# update them in the repo.
echo 'command -v nix-shell > /dev/null || postgrest-git-hooks --hook=pre-commit run' "$_arg_mode" \
>> .git/hooks/pre-commit
# shellcheck disable=SC2016
echo 'command -v nix-shell > /dev/null && $(nix-shell --quiet -Q --run "command -v postgrest-git-hooks") --hook=pre-commit run' "$_arg_mode" \
>> .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo 'command -v nix-shell > /dev/null || postgrest-git-hooks --hook=pre-push run' "$_arg_mode" \
>> .git/hooks/pre-push
# shellcheck disable=SC2016
echo 'command -v nix-shell > /dev/null && $(nix-shell --quiet -Q --run "command -v postgrest-git-hooks") --hook=pre-push run' "$_arg_mode" \
>> .git/hooks/pre-push
chmod +x .git/hooks/pre-push
fi
else
# When run from a git hook, the GIT_ environment variables conflict with our withGit helper.
# The following unsets all GIT_ variables.
unset "''${!GIT_@}"
# shellcheck disable=SC2317
function restore () {
ref="$(git stash list --format=format:%gD --grep "$1" -n1)"
# this will avoid merge conflicts when applying the stash
${git}/bin/git restore --source="$ref" .
# restore untracked files, too. could fail with no files
if [ "$(git show --numstat --format=oneline "$ref^3" | wc -l)" -gt 1 ]; then
${git}/bin/git restore --overlay --source="$ref^3" .
fi
${git}/bin/git stash drop "$ref"
}
case "$_arg_mode" in
basic)
case "$_arg_hook" in
pre-commit)
# To be able to automatically add only changes from postgrest-style to the staging area,
# we need to run postgrest-style twice. Otherwise we'd risk merge conflicts when popping
# the stash afterwards.
${style}/bin/postgrest-style
stash="postgrest-git-hooks-$RANDOM"
${git}/bin/git stash push --include-untracked --keep-index -m "$stash"
if [ "$(git stash list --grep $stash)" ]; then
# Only create the stash pop trap, if we actually created a stash.
# Otherwise stash pop will cause havoc.
trap 'restore "$stash"' EXIT
fi
${style}/bin/postgrest-style
${git}/bin/git add .
;;
pre-push)
# Create a clean working tree without any uncomitted changes.
${withTools.withGit} HEAD ${style}/bin/postgrest-lint
;;
esac
;;
full)
case "$_arg_hook" in
pre-commit)
# To be able to automatically add only changes from postgrest-style to the staging area,
# we need to run postgrest-style twice. Otherwise we'd risk merge conflicts when popping
# the stash afterwards.
${style}/bin/postgrest-style
stash="postgrest-git-hooks-$RANDOM"
${git}/bin/git stash push --include-untracked --keep-index -m "$stash"
if [ "$(git stash list --grep $stash)" ]; then
# Only create the stash pop trap, if we actually created a stash.
# Otherwise stash pop will cause havoc.
trap 'restore "$stash"' EXIT
fi
${style}/bin/postgrest-style
${git}/bin/git add .
${style}/bin/postgrest-lint
;;
pre-push)
# Create a clean working tree without any uncomitted changes.
${withTools.withGit} HEAD ${check}
;;
esac
;;
esac
fi
'';
dumpMinimalImports = dumpMinimalImports =
checkedShellScript checkedShellScript
{ {
@@ -280,10 +129,10 @@ let
{ {
name = "postgrest-hsie-graph-modules"; name = "postgrest-hsie-graph-modules";
docs = "Create a PNG graph of modules imported within the codebase."; docs = "Create a PNG graph of modules imported within the codebase.";
args = [ "ARG_POSITIONAL_SINGLE([outfile], [Output filename])" ]; args = [ "ARG_OPTIONAL_SINGLE([outfile], [o], [Output filename], [postgrest-module-graph.png])" ];
} }
'' ''
${hsie} graph-modules main src | ${graphviz}/bin/dot -Tpng -o "$_arg_outfile" ${hsie} graph-modules src/library src/executable | ${graphviz}/bin/dot -Tpng -o "$_arg_outfile"
''; '';
hsieGraphSymbols = hsieGraphSymbols =
@@ -394,7 +243,6 @@ buildToolbox
inherit inherit
check check
dumpMinimalImports dumpMinimalImports
gitHooks
hsieGraphModules hsieGraphModules
hsieGraphSymbols hsieGraphSymbols
hsieMinimalImports hsieMinimalImports
+6 -2
View File
@@ -43,7 +43,7 @@ let
} }
if [ "$_arg_language" == "" ]; then if [ "$_arg_language" == "" ]; then
# clean previous build, otherwise some errors might be supressed # clean previous build, otherwise some errors might be suppressed
rm -rf "../.docs-build/html/default" rm -rf "../.docs-build/html/default"
if [ -d languages ]; then if [ -d languages ]; then
@@ -54,7 +54,7 @@ let
build html "../.docs-build/html/default" build html "../.docs-build/html/default"
else else
# clean previous build, otherwise some errors might be supressed # clean previous build, otherwise some errors might be suppressed
rm -rf "../.docs-build/html/$_arg_language" rm -rf "../.docs-build/html/$_arg_language"
# update and build specific locale, can be used to create new locale # update and build specific locale, can be used to create new locale
@@ -122,6 +122,8 @@ let
workingDir = "/docs"; workingDir = "/docs";
} }
'' ''
echo "Checking spelling mistakes..."
export LC_ALL=C export LC_ALL=C
FILES=$(find . -type f -iname '*.rst' | tr '\n' ' ') FILES=$(find . -type f -iname '*.rst' | tr '\n' ' ')
@@ -144,6 +146,8 @@ let
workingDir = "/docs"; workingDir = "/docs";
} }
'' ''
echo "Detecting obsolete dictionary entries..."
export LC_ALL=C export LC_ALL=C
FILES=$(find . -type f -iname '*.rst' | tr '\n' ' ') FILES=$(find . -type f -iname '*.rst' | tr '\n' ' ')
+44 -91
View File
@@ -12,42 +12,39 @@
# from an array # from an array
import time import time
import argparse import argparse
import sys
import random import random
import jwt import jwcrypto.jwt as jwt
import jwcrypto.jwk as jwk
from typing import Optional
from pathlib import Path from pathlib import Path
URL = "http://postgrest" URL = "http://postgrest"
secret_key = b"reallyreallyreallyreallyverysafe" secret_key = "reallyreallyreallyreallyverysafe"
key = jwk.JWK.generate(kty="RSA", size=4096)
private_key = jwt.algorithms.RSAAlgorithm.from_jwk(key.export_private())
public_key = key.export_public()
def generate_jwt(now: int, exp_inc: Optional[int], is_hs: bool) -> str: def generate_target(
"""Generate an HS256 or RS256 JWT""" now: int,
payload = { key: jwt.JWK,
) -> list[str]:
"""Generate a target using an HS256 or RS256 JWT"""
headers = {
"sub": f"user_{random.getrandbits(32)}", "sub": f"user_{random.getrandbits(32)}",
"iat": now, "iat": now,
}
claims = {
"role": "postgrest_test_author", "role": "postgrest_test_author",
} }
if exp_inc is not None: headers["alg"] = "RS256" if key.get("kty") == "RSA" else "HS256"
payload["exp"] = now + exp_inc
k = secret_key if is_hs else private_key token = jwt.JWT(headers, claims)
alg = "HS256" if is_hs else "RS256" token.make_signed_token(key)
return jwt.encode(payload, k, alg)
return [
def append_targets(lines: list[str], token: str): f"OPTIONS {URL}/authors_only?{headers["alg"]}",
lines.append(f"OPTIONS {URL}/authors_only") f"Authorization: Bearer {token.serialize()}",
lines.append(f"Authorization: Bearer {token}") "", # blank line to separate requests
lines.append("") # blank line to separate requests ]
def main(): def main():
@@ -55,91 +52,47 @@ def main():
description="Generate Vegeta targets with unique JWTs" description="Generate Vegeta targets with unique JWTs"
) )
parser.add_argument( parser.add_argument(
"output", "generated_path",
help="Path to write the generated targets file", metavar="GENERATED_PATH",
) help="Path to write the generated files",
parser.add_argument(
"--worst",
dest="worst",
action=argparse.BooleanOptionalAction,
default=False,
help="Generate worst case targets for a JWT cache",
)
parser.add_argument(
"--rsa",
dest="jwk_path",
metavar="JWK_PATH",
type=Path, type=Path,
default=None,
help="Path for generating a RSA JWK file to sign tokens with",
) )
args = parser.parse_args() args = parser.parse_args()
is_hs = args.jwk_path is None targets_path = args.generated_path / "gen_targets.http"
nsamples = 1000 hs = jwt.JWK.from_password(secret_key)
if is_hs: rsa = jwt.JWK.generate(kty="RSA", size=4096)
ntargets = 200000
else:
# The asymmetric targets take too long to compute so we reduce them
ntargets = 50000
if not is_hs: jwks = jwt.JWKSet()
try: jwks.add(hs)
with open(args.jwk_path, "w") as jwk: jwks.add(rsa)
jwk.write(public_key)
print(f"Created {args.jwk_path} file containing the RSA JWK") jwks_path = args.generated_path / "gen_jwks.json"
except IOError as e:
print(f"Error writing to {args.jwk_path}: {e}", file=sys.stderr) # Technically, this exports the private keys, because HS does not have the concept
sys.exit(1) # of a public key. This is not a problem for tests, though, PostgREST can verify
# tokens with the private key just as well.
jwks_path.write_text(jwks.export())
print(f"Created JWKSet on {jwks_path}")
ntargets = 1000
print(f"Generating {ntargets} targets...") print(f"Generating {ntargets} targets...")
start_time = time.time() now = int(time.time())
now = int(start_time)
lines = [] lines = []
# We want to ensure 401 Unauthorized responses don't happen during for i in range(ntargets):
# JWT validation, this can happen when the jwt `exp` is too short. target = generate_target(now, hs if i % 2 == 0 else rsa)
# At the same time, we want to ensure the `exp` is not too big, lines.extend(target)
# so expires will occur and postgREST needs to
# clean cached expired JWTs
if args.worst:
# estimated time takes to build and run postgrest itself
build_run_postgrest_time = 2
# estimated time it takes to generate the targets file
# the division numbers are tuned by hand
if is_hs: # hs generation is much faster
gen_time = ntargets // 66666
else: # asymmetric is slower so the time is higher
gen_time = ntargets // 220
# estimated exp time so some JWTs will expire with open(targets_path, "w") as f:
inc = build_run_postgrest_time + gen_time f.write("\n".join(lines))
for i in range(ntargets):
token = generate_jwt(now, inc + i // 1000, is_hs)
append_targets(lines, token)
else:
tokens = [generate_jwt(now, None, is_hs) for _ in range(nsamples)]
for i in range(ntargets):
token = random.choice(tokens)
append_targets(lines, token)
try:
with open(args.output, "w") as f:
f.write("\n".join(lines))
except IOError as e:
print(f"Error writing to {args.output}: {e}", file=sys.stderr)
sys.exit(1)
elapsed = time.time() - start_time
print(f"Created {ntargets} targets", end=" ") print(f"Created {ntargets} targets", end=" ")
print(f"in {args.output} ({elapsed:.2f}s)")
if __name__ == "__main__": if __name__ == "__main__":
+142 -98
View File
@@ -1,7 +1,11 @@
{ buildToolbox { buildToolbox
, checkedShellScript , checkedShellScript
, git
, jq , jq
, libfaketime
, python3
, python3Packages , python3Packages
, runCommand
, vegeta , vegeta
, withTools , withTools
, writers , writers
@@ -18,6 +22,8 @@ let
]; ];
} }
'' ''
echo "Starting vegeta loadtest..."
# ARG_USE_ENV only adds defaults or docs for environment variables # ARG_USE_ENV only adds defaults or docs for environment variables
# We manually implement a required check here # We manually implement a required check here
# See also: https://github.com/matejak/argbash/issues/80 # See also: https://github.com/matejak/argbash/issues/80
@@ -40,86 +46,75 @@ let
docs = "Run the vegeta loadtests with PostgREST."; docs = "Run the vegeta loadtests with PostgREST.";
args = [ args = [
"ARG_OPTIONAL_SINGLE([output], [o], [Filename to dump json output to], [./loadtest/result.bin])" "ARG_OPTIONAL_SINGLE([output], [o], [Filename to dump json output to], [./loadtest/result.bin])"
"ARG_OPTIONAL_SINGLE([testdir], [t], [Directory to load tests and fixtures from], [./test/load])"
"ARG_OPTIONAL_SINGLE([kind], [k], [Kind of loadtest], [mixed])" "ARG_OPTIONAL_SINGLE([kind], [k], [Kind of loadtest], [mixed])"
"ARG_TYPE_GROUP_SET([KIND], [KIND], [kind], [mixed,jwt-hs,jwt-hs-cache,jwt-hs-cache-worst,jwt-rsa,jwt-rsa-cache,jwt-rsa-cache-worst])" "ARG_TYPE_GROUP_SET([KIND], [KIND], [kind], [mixed,jwt-cache,jwt-cache-worst])"
"ARG_OPTIONAL_SINGLE([monitor], [m], [Monitoring file], [./loadtest/result.csv])" "ARG_OPTIONAL_SINGLE([monitor], [m], [Monitoring file], [./loadtest/result.csv])"
"ARG_LEFTOVERS([additional vegeta arguments])" "ARG_LEFTOVERS([additional vegeta arguments])"
]; ];
workingDir = "/"; workingDir = "/";
} }
'' ''
# previously required settings to make this work with older branches
export PGRST_DB_ANON_ROLE="postgrest_test_anonymous" export PGRST_DB_ANON_ROLE="postgrest_test_anonymous"
export PGRST_DB_URI="postgresql://"
export PGRST_DB_SCHEMAS="test"
export PGRST_DB_CONFIG="false" export PGRST_DB_CONFIG="false"
export PGRST_DB_POOL="1" export PGRST_DB_POOL="1"
export PGRST_DB_SCHEMAS="test"
export PGRST_DB_TX_END="rollback-allow-override" export PGRST_DB_TX_END="rollback-allow-override"
export PGRST_LOG_LEVEL="crit" export PGRST_LOG_LEVEL="crit"
export PGRST_JWT_SECRET="reallyreallyreallyreallyverysafe" export PGRST_JWT_SECRET="reallyreallyreallyreallyverysafe"
# set previous PGRST_JWT_CACHE_MAX_LIFETIME configuration so that
# load test works across branches
# TODO clean once PGRST_JWT_CACHE_MAX_ENTRIES merged and released
export PGRST_JWT_CACHE_MAX_LIFETIME="86400"
mkdir -p "$(dirname "$_arg_output")" mkdir -p "$(dirname "$_arg_output")"
abs_output="$(realpath "$_arg_output")" abs_output="$(realpath "$_arg_output")"
case "$_arg_kind" in case "$_arg_kind" in
jwt-hs) jwt-cache)
${genTargetsHS} "$_arg_testdir"/gen_targets.http export PGRST_JWT_SECRET="@${generatedTargets}/gen_jwks.json"
export PGRST_JWT_CACHE_MAX_ENTRIES="0"
export PGRST_JWT_CACHE_MAX_LIFETIME="0" # shellcheck disable=SC2145
${withTools.withPg} -f test/load/fixtures.sql \
${withTools.withPgrst} --faketime '2000-01-01 00:00:00' -m "$_arg_monitor" \
sh -c "cd test/load && \
${runner} -targets ${generatedTargets}/gen_targets.http -output \"$abs_output\" \"''${_arg_leftovers[@]}\""
;; ;;
jwt-hs-cache) # here we sleep purposefully to check how much memory does the schema cache consume in the final report
${genTargetsHS} "$_arg_testdir"/gen_targets.http mixed)
;; # shellcheck disable=SC2145
${withTools.withPg} -f test/load/fixtures.sql \
jwt-hs-cache-worst) ${withTools.withPgrst} --timeout 2 --sleep 5 -m "$_arg_monitor" \
${genTargetsHS} --worst "$_arg_testdir"/gen_targets.http sh -c "cd test/load && \
;; ${runner} -targets targets.http -output \"$abs_output\" \"''${_arg_leftovers[@]}\""
jwt-rsa)
${genTargetsHS} --rsa="$_arg_testdir"/gen_jwk.json "$_arg_testdir"/gen_targets.http
export PGRST_JWT_CACHE_MAX_ENTRIES="0"
export PGRST_JWT_CACHE_MAX_LIFETIME="0"
export PGRST_JWT_SECRET="@$_arg_testdir/gen_jwk.json"
;;
jwt-rsa-cache)
${genTargetsHS} --rsa="$_arg_testdir"/gen_jwk.json "$_arg_testdir"/gen_targets.http
export PGRST_JWT_SECRET="@$_arg_testdir/gen_jwk.json"
;;
jwt-rsa-cache-worst)
${genTargetsHS} --worst --rsa="$_arg_testdir"/gen_jwk.json "$_arg_testdir"/gen_targets.http
export PGRST_JWT_SECRET="@$_arg_testdir/gen_jwk.json"
;;
*)
;; ;;
esac esac
if [ "$_arg_kind" == "mixed" ]; then
# shellcheck disable=SC2145
${withTools.withPg} -f "$_arg_testdir"/fixtures.sql \
${withTools.withSlowPg} \
${withTools.withPgrst} -m "$_arg_monitor" \
${withTools.withSlowPgrst} \
sh -c "cd \"$_arg_testdir\" && \
${runner} -targets targets.http -output \"$abs_output\" \"''${_arg_leftovers[@]}\""
else
# shellcheck disable=SC2145
${withTools.withPg} -f "$_arg_testdir"/fixtures.sql \
${withTools.withPgrst} -m "$_arg_monitor" \
sh -c "cd \"$_arg_testdir\" && \
${runner} -lazy -targets gen_targets.http -output \"$abs_output\" \"''${_arg_leftovers[@]}\""
fi
${vegeta}/bin/vegeta report -type=text "$_arg_output" ${vegeta}/bin/vegeta report -type=text "$_arg_output"
if [ "$_arg_kind" != "mixed" ]; then
# fail in case 401 happened on jwt loadtests
unauthorized_count="$(${vegeta}/bin/vegeta report -type=json "$_arg_output" \
| ${jq}/bin/jq -r '.status_codes["401"] // 0')"
if [ "$unauthorized_count" -gt 0 ]; then
last_unauthorized_body="$(${vegeta}/bin/vegeta encode "$_arg_output" \
| ${jq}/bin/jq -rn '
reduce inputs as $item (null;
if $item.code == 401 then $item else . end
)
| if . == null then
empty
else
(.body | @base64d)
end
')"
echo "loadtest failed: found $unauthorized_count 401 Unauthorized responses" >&2
if [ -n "$last_unauthorized_body" ]; then
printf '%s\n' "Last 401 response body:" >&2
printf '%s\n' "$last_unauthorized_body" >&2
fi
exit 1
fi
fi
''; '';
loadtestAgainst = loadtestAgainst =
@@ -134,6 +129,9 @@ let
Run the vegeta loadtest against every target branch and HEAD: Run the vegeta loadtest against every target branch and HEAD:
- once on the every <target-#> branch - once on the every <target-#> branch
- once in the current worktree - once in the current worktree
Note that the Nix tooling is always taken from the HEAD branch, while the PostgREST binary is taken from the target branch.
For a discussion on why this is set up like this, see https://github.com/PostgREST/postgrest/pull/5013#discussion_r3431508441.
''; '';
args = [ args = [
"ARG_POSITIONAL_INF([target], [Commit-ish reference to compare with], 1)" "ARG_POSITIONAL_INF([target], [Commit-ish reference to compare with], 1)"
@@ -148,8 +146,40 @@ let
workingDir = "/"; workingDir = "/";
} }
'' ''
# run loadtest for every target # Build postgrest for every target and HEAD.
for tgt in "''${_arg_target[@]}"; do # Keeps a reference to the postgrest binary and faketime lib for every branch to run later.
declare -A pgrst faketime
for tgt in "''${_arg_target[@]}" HEAD; do
# not using withTmpDir here, because we don't want to keep the directory on error
tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT
${git}/bin/git worktree add -f "$tmpdir" "$tgt" > /dev/null
pushd "$tmpdir" > /dev/null
build_start=$SECONDS
echo -n "${name}: Building postgrest (nix) on $tgt... "
# Using lib.getBin to also make this work with older checkouts, where .bin was not a thing, yet.
nix-build --no-out-link -E 'with import ./. {}; pkgs.lib.getBin postgrestPackage' > build.log 2>&1 || {
echo "failed, output:"
cat build.log
exit 1
}
pgrst[$tgt]="$(nix-build --no-out-link -E 'with import ./. {}; pkgs.lib.getBin postgrestPackage')/bin/postgrest"
# To avoid glibc mismatches with back-branches, we need to take libfaketime from the target branch.
faketime[$tgt]="$(nix-build --no-out-link -A pkgs.libfaketime)/lib/libfaketime.so.1"
build_end=$((SECONDS - build_start))
printf "done in %ss.\n" "$build_end"
popd > /dev/null
${git}/bin/git worktree remove -f "$tmpdir" > /dev/null
rm -rf "$tmpdir"
done
# Run loadtest for every target and HEAD.
# Running the tests is separated from building them to reduce the chances of
# other processes skewing the results between two runs.
for tgt in "''${_arg_target[@]}" HEAD; do
cat << EOF cat << EOF
@@ -157,12 +187,7 @@ let
EOF EOF
# Runs the test files from the current working tree FAKETIME_LIB="''${faketime[$tgt]}" PGRST_CMD="''${pgrst[$tgt]}" ${loadtest} -k "$_arg_kind" -m "loadtest/$tgt.csv" --output "loadtest/$tgt.bin"
# to make sure both tests are run with the same files.
# Save the results in the current working tree, too,
# otherwise they'd be lost in the temporary working tree
# created by withTools.withGit.
${withTools.withGit} "$tgt" ${loadtest} -k "$_arg_kind" -m "$PWD/loadtest/$tgt.csv" --output "$PWD/loadtest/$tgt.bin" --testdir "$PWD/test/load"
cat << EOF cat << EOF
@@ -171,22 +196,6 @@ let
EOF EOF
done done
# run loadtest once on HEAD
cat << EOF
Running "$_arg_kind" loadtest on HEAD...
EOF
${loadtest} -k "$_arg_kind" -m "$PWD/loadtest/head.csv" --output "$PWD/loadtest/head.bin" --testdir "$PWD/test/load"
cat << EOF
Done running on HEAD.
EOF
''; '';
reporter = reporter =
@@ -196,12 +205,14 @@ let
docs = "Create a named json report for a single result file."; docs = "Create a named json report for a single result file.";
args = [ args = [
"ARG_POSITIONAL_SINGLE([file], [Filename of result to create report for])" "ARG_POSITIONAL_SINGLE([file], [Filename of result to create report for])"
"ARG_OPTIONAL_SINGLE([percentile], [p], [Percentile to report latency for], 50)"
"ARG_LEFTOVERS([additional vegeta arguments])" "ARG_LEFTOVERS([additional vegeta arguments])"
]; ];
workingDir = "/"; workingDir = "/";
} }
'' ''
${vegeta}/bin/vegeta report -type=json "$_arg_file" \ ${vegeta}/bin/vegeta encode "$_arg_file" \
| ${jq}/bin/jq --arg percentile "$_arg_percentile" --slurp 'map(select(.url != "")) | group_by("\(.code) \(.method) \(.url)") | map({("\(.[0].code) \(.[0].method) \(.[0].url)" | sub("http://postgrest";"")): map(.latency) | sort | .[(length-1) * ($percentile | tonumber) / 100 | floor] / 10e3 }) | .[]' \
| ${jq}/bin/jq --arg branch "$(basename "$_arg_file" .bin)" '. + {branch: $branch}' | ${jq}/bin/jq --arg branch "$(basename "$_arg_file" .bin)" '. + {branch: $branch}'
''; '';
@@ -214,12 +225,31 @@ let
import sys import sys
import pandas as pd import pandas as pd
def evaluate_change(df):
try:
return ((df['HEAD'] / df['main'] - 1) * 100) \
.map(lambda r: "{icon} {ratio:.1f} %".format(
ratio=r,
# Hardcoded failure threshold for CI is 5% here.
icon="" if r < 5 else ":x:"
))
except KeyError:
return None
pd.read_json(sys.stdin) \ pd.read_json(sys.stdin) \
.set_index('param') \ .rename(columns={'latency': sys.argv[1]}) \
.drop(['branch', 'earliest', 'end', 'latest']) \ .set_index(sys.argv[1]) \
.fillna("") \ .drop(['branch']) \
.convert_dtypes() \ .convert_dtypes() \
.to_markdown(sys.stdout, floatfmt='.0f') .assign(change=evaluate_change) \
.to_markdown(
sys.stdout,
floatfmt='.1f',
colglobalalign='right',
colalign=('left',)
)
''; '';
@@ -230,32 +260,46 @@ let
docs = "Create a report of all loadtest reports as markdown."; docs = "Create a report of all loadtest reports as markdown.";
args = [ args = [
"ARG_OPTIONAL_SINGLE([group], [g], [Marker to group results])" "ARG_OPTIONAL_SINGLE([group], [g], [Marker to group results])"
"ARG_OPTIONAL_SINGLE([percentile], [p], [Percentile to report latency for], 50)"
]; ];
workingDir = "/"; workingDir = "/";
} }
'' ''
marker=''${_arg_group:+"($_arg_group)"} echo -e "## Loadtest results $_arg_group (P$_arg_percentile)\n"
echo -e "## Loadtest results $marker\n" find loadtest -type f -iname '*.bin' -exec ${reporter} -p "$_arg_percentile" {} \; \
| ${jq}/bin/jq '[paths(scalars) as $path | {latency: $path | join("."), (.branch): getpath($path)}]' \
| ${jq}/bin/jq --slurp 'flatten | group_by(.latency) | map(add)' \
| ${toMarkdown} "P$_arg_percentile latency [μs]"
'';
find loadtest -type f -iname '*.bin' -exec ${reporter} {} \; \ report-load =
| ${jq}/bin/jq '[paths(scalars) as $path | {param: $path | join("."), (.branch): getpath($path)}]' \ checkedShellScript
| ${jq}/bin/jq --slurp 'flatten | group_by(.param) | map(add)' \ {
| ${toMarkdown} name = "postgrest-loadtest-report-load";
docs = "Create a report of all CPU/MEM usage as markdown.";
echo -e "\n\n## Loadtest elapsed seconds vs CPU/MEM usage $marker\n" args = [
"ARG_OPTIONAL_SINGLE([group], [g], [Marker to group results])"
];
workingDir = "/";
}
''
echo -e "\n\n## Loadtest elapsed seconds vs CPU/MEM usage $_arg_group\n"
find loadtest -type f -iname '*.csv' \ find loadtest -type f -iname '*.csv' \
| sort -m \ | sort -m \
| ${mergeMonitorResults} | ${mergeMonitorResults}
''; '';
genTargetsHS = generatedTargets =
writers.writePython3 "postgrest-gen-loadtest-targets-hs" runCommand "postgrest-loadtest-targets"
{ {
libraries = [ python3Packages.pyjwt python3Packages.jwcrypto ]; nativeBuildInputs = [ (python3.withPackages (pyps: [ pyps.jwcrypto ])) ];
} }
(builtins.readFile ./generate_targets.py); ''
mkdir -p "$out"
${libfaketime}/bin/faketime '2000-01-01 00:00:00' python3 ${./generate_targets.py} "$out"
'';
mergeMonitorResults = mergeMonitorResults =
writers.writePython3 "postgrest-merge-monitor-results" writers.writePython3 "postgrest-merge-monitor-results"
@@ -266,5 +310,5 @@ let
in in
buildToolbox { buildToolbox {
name = "postgrest-loadtest"; name = "postgrest-loadtest";
tools = { inherit loadtest loadtestAgainst report; }; tools = { inherit loadtest loadtestAgainst report report-load; };
} }
+1 -1
View File
@@ -17,7 +17,7 @@ let
nix flake update nix flake update
echo "# This file is auto-generated by postgrest-nixpkgs-upgrade" > docs/requirements.txt echo "# This file is auto-generated by postgrest-nixpkgs-upgrade" > docs/requirements.txt
cat "$(nix-build -A docs.requirements)" >> docs/requirements.txt cat "$(nix-build --no-out-link -A docs.requirements)" >> docs/requirements.txt
''; '';
in in
+7 -6
View File
@@ -7,7 +7,6 @@ let
{ {
name = "postgrest-release"; name = "postgrest-release";
docs = "Patch postgrest.cabal, CHANGELOG.md, commit and push all in one go."; docs = "Patch postgrest.cabal, CHANGELOG.md, commit and push all in one go.";
args = [ "ARG_OPTIONAL_BOOLEAN([major], [m], [Bump to new major version (only applies on main branch).])" ];
workingDir = "/"; workingDir = "/";
} }
'' ''
@@ -20,7 +19,6 @@ let
git diff --exit-code HEAD postgrest.cabal > /dev/null git diff --exit-code HEAD postgrest.cabal > /dev/null
trap "" ERR trap "" ERR
# TODO: Support C+D bumps when implementing hackage releases
bump () { bump () {
current_version="$(grep -oP '^version:\s*\K.*' postgrest.cabal)" current_version="$(grep -oP '^version:\s*\K.*' postgrest.cabal)"
# shellcheck disable=SC2034 # shellcheck disable=SC2034
@@ -47,7 +45,10 @@ let
echo "Updating docs/conf.py ..." echo "Updating docs/conf.py ..."
sed -i -E "s/^(version = ).*$/\1\"$new_docs_version\"/" docs/conf.py > /dev/null sed -i -E "s/^(version = ).*$/\1\"$new_docs_version\"/" docs/conf.py > /dev/null
git add postgrest.cabal docs/conf.py > /dev/null echo "Updating Haskell source file links ..."
sed -i -E "s#(github\.com/PostgREST/postgrest/blob)/main/#\1/$new_version/#g" docs/explanations/architecture.rst
git add postgrest.cabal docs/conf.py docs/explanations/architecture.rst > /dev/null
} }
today_date_for_changelog="$(date '+%Y-%m-%d')" today_date_for_changelog="$(date '+%Y-%m-%d')"
@@ -62,19 +63,19 @@ let
git add CHANGELOG.md > /dev/null git add CHANGELOG.md > /dev/null
echo "Committing ..." echo "Committing ..."
git commit -m "bump version to $new_version" > /dev/null git commit -m "chore: bump version to $new_version" > /dev/null
if [[ "$current_branch" == "main" ]]; then if [[ "$current_branch" == "main" ]]; then
bump devel bump devel
# The order of operations is important here: # The order of operations is important here:
# - bump devel is run and $A is upated to the new version # - bump devel is run and $A is updated to the new version
# - the branch is created with the new A, but the commit before the devel bump # - the branch is created with the new A, but the commit before the devel bump
# - the devel bump is committed # - the devel bump is committed
git branch "v$A" git branch "v$A"
echo "Committing (devel bump)..." echo "Committing (devel bump)..."
git commit -m "bump version to $new_version" > /dev/null git commit -m "chore: bump version to $new_version" > /dev/null
fi fi
trap "echo Remote not found. Please push manually ..." ERR trap "echo Remote not found. Please push manually ..." ERR
+27 -13
View File
@@ -3,15 +3,16 @@
, buildToolbox , buildToolbox
, checkedShellScript , checkedShellScript
, deadnix , deadnix
, fd
, git , git
, hlint , hlint
, hsie , hsie
, nixpkgs-fmt , nixpkgs-fmt
, python3Packages , python3Packages
, ruff , ruff
, silver-searcher
, statix , statix
, stylish-haskell , stylish-haskell
, writeText
}: }:
let let
style = style =
@@ -20,27 +21,27 @@ let
name = "postgrest-style"; name = "postgrest-style";
docs = "Automatically format Haskell, Nix and Python files."; docs = "Automatically format Haskell, Nix and Python files.";
workingDir = "/"; workingDir = "/";
withTmpDir = true;
} }
'' ''
# Format Nix files # Format Nix files
${statix}/bin/statix fix ${statix}/bin/statix fix
${nixpkgs-fmt}/bin/nixpkgs-fmt . > /dev/null 2> /dev/null ${nixpkgs-fmt}/bin/nixpkgs-fmt .
# Format Haskell files # Format Haskell files
# --vimgrep fixes a bug in ag: https://github.com/ggreer/the_silver_searcher/issues/753 ${fd}/bin/fd '\.l?hs$' \
${silver-searcher}/bin/ag -l --vimgrep -g '\.l?hs$' . \
| xargs ${stylish-haskell}/bin/stylish-haskell -i | xargs ${stylish-haskell}/bin/stylish-haskell -i
# Format Python files # Format Python files
${black}/bin/black . 2> /dev/null TMPDIR="$tmpdir" ${black}/bin/black .
''; '';
# Script to check whether any uncommited changes result from postgrest-style # Script to check whether any uncommitted changes result from postgrest-style
styleCheck = styleCheck =
checkedShellScript checkedShellScript
{ {
name = "postgrest-style-check"; name = "postgrest-style-check";
docs = "Check whether postgrest-style results in any uncommited changes."; docs = "Check whether postgrest-style results in any uncommitted changes.";
workingDir = "/"; workingDir = "/";
} }
'' ''
@@ -51,6 +52,20 @@ let
${git}/bin/git diff-index --exit-code HEAD -- '*.hs' '*.lhs' '*.nix' '*.py' ${git}/bin/git diff-index --exit-code HEAD -- '*.hs' '*.lhs' '*.nix' '*.py'
''; '';
hlintConfig = writeText "hlintConfig.yml" ''
# Arguments passed to hlint
- arguments: [-j, -XQuasiQuotes, -XNoPatternSynonyms]
# Warnings
- warn: { lhs: "a == a", rhs: "True", note: "This comparison always evaluates to True" }
- warn: { lhs: "a /= a", rhs: "False", note: "This comparison always evaluates to False" }
- warn: { lhs: "a < a", rhs: "False", note: "This comparison always evaluates to False" }
- warn: { lhs: "a > a", rhs: "False", note: "This comparison always evaluates to False" }
- warn: { lhs: "a <= a", rhs: "True", note: "This comparison always evaluates to True" }
- warn: { lhs: "a >= a", rhs: "True", note: "This comparison always evaluates to True" }
'';
lint = lint =
checkedShellScript checkedShellScript
{ {
@@ -67,19 +82,18 @@ let
# ruff has gaps in scanning for unused code, so we use vulture # ruff has gaps in scanning for unused code, so we use vulture
echo "Scanning python files for unused code..." echo "Scanning python files for unused code..."
${silver-searcher}/bin/ag -l --vimgrep -g '\.l?py$' . \ ${fd}/bin/fd '\.l?py$' \
| xargs ${python3Packages.vulture}/bin/vulture --exclude docs/conf.py | xargs ${python3Packages.vulture}/bin/vulture --exclude docs/conf.py --min-confidence 80
echo "Linting python files..." echo "Linting python files..."
${ruff}/bin/ruff check . ${ruff}/bin/ruff check .
echo "Checking consistency of import aliases in Haskell code..." echo "Checking consistency of import aliases in Haskell code..."
${hsie} check-aliases main src ${hsie} check-aliases src/library src/executable
echo "Linting Haskell files..." echo "Linting Haskell files..."
# --vimgrep fixes a bug in ag: https://github.com/ggreer/the_silver_searcher/issues/753 ${fd}/bin/fd '\.l?hs$' \
${silver-searcher}/bin/ag -l --vimgrep -g '\.l?hs$' . \ | xargs ${hlint}/bin/hlint --hint=${hlintConfig}
| xargs ${hlint}/bin/hlint -j -X QuasiQuotes -X NoPatternSynonyms
''; '';
in in
+26 -6
View File
@@ -7,9 +7,9 @@
, glibcLocales ? null , glibcLocales ? null
, gnugrep , gnugrep
, hpc-codecov , hpc-codecov
, hostPlatform
, jq , jq
, lib , lib
, nginx
, postgrest , postgrest
, python3 , python3
, runtimeShell , runtimeShell
@@ -32,6 +32,20 @@ let
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:spec -- "''${_arg_leftovers[@]}" ${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:spec -- "''${_arg_leftovers[@]}"
''; '';
testObservability =
checkedShellScript
{
name = "postgrest-test-observability";
docs = "Run the Haskell observability test suite.";
args = [ "ARG_LEFTOVERS([hspec arguments])" ];
workingDir = "/";
withEnv = postgrest.env;
}
''
${withTools.withPg} -f test/observability/fixtures/load.sql \
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:observability -- "''${_arg_leftovers[@]}"
'';
testDoctests = testDoctests =
checkedShellScript checkedShellScript
{ {
@@ -41,8 +55,6 @@ let
withEnv = postgrest.env; withEnv = postgrest.env;
} }
'' ''
# This makes nix-env -iA tests.doctests.bin work.
export NIX_GHC=${postgrest.env.NIX_GHC}
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:doctests ${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:doctests
''; '';
@@ -80,6 +92,7 @@ let
args = [ "ARG_LEFTOVERS([pytest arguments])" ]; args = [ "ARG_LEFTOVERS([pytest arguments])" ];
workingDir = "/"; workingDir = "/";
withEnv = postgrest.env; withEnv = postgrest.env;
withPath = [ nginx ];
} }
'' ''
${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest ${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest
@@ -142,10 +155,11 @@ let
redirectTixFiles = false; redirectTixFiles = false;
withEnv = postgrest.env; withEnv = postgrest.env;
withTmpDir = true; withTmpDir = true;
withPath = [ nginx ];
} }
( (
# required for `hpc markup` in CI; glibcLocales is not available e.g. on Darwin # required for `hpc markup` in CI; glibcLocales is not available e.g. on Darwin
lib.optionalString (stdenv.isLinux && hostPlatform.libc == "glibc") '' lib.optionalString (stdenv.isLinux && stdenv.hostPlatform.libc == "glibc") ''
export LOCALE_ARCHIVE="${glibcLocales}/lib/locale/locale-archive" export LOCALE_ARCHIVE="${glibcLocales}/lib/locale/locale-archive"
'' + '' +
@@ -155,7 +169,7 @@ let
rm -rf coverage/* rm -rf coverage/*
# build once before running all the tests # build once before running all the tests
${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest lib:postgrest test:spec ${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest lib:postgrest test:spec test:observability
( (
trap 'echo Found dead code: Check file list above.' ERR ; trap 'echo Found dead code: Check file list above.' ERR ;
@@ -179,11 +193,16 @@ let
${withTools.withPg} -f test/spec/fixtures/load.sql \ ${withTools.withPg} -f test/spec/fixtures/load.sql \
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:spec ${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:spec
HPCTIXFILE="$tmpdir"/observability.tix \
${withTools.withPg} -f test/observability/fixtures/load.sql \
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:observability
# Note: No coverage for doctests, as doctests leverage GHCi and GHCi does not support hpc # Note: No coverage for doctests, as doctests leverage GHCi and GHCi does not support hpc
# collect all the tix files # collect all the tix files
${ghc}/bin/hpc sum --union --exclude=Paths_postgrest --output="$tmpdir"/tests.tix \ ${ghc}/bin/hpc sum --union --exclude=Paths_postgrest --output="$tmpdir"/tests.tix \
"$tmpdir"/io*.tix "$tmpdir"/big_schema*.tix "$tmpdir"/replica*.tix "$tmpdir"/spec.tix "$tmpdir"/io*.tix "$tmpdir"/big_schema*.tix "$tmpdir"/replica*.tix "$tmpdir"/spec.tix \
"$tmpdir"/observability.tix
# prepare the overlay # prepare the overlay
${ghc}/bin/hpc overlay --output="$tmpdir"/overlay.tix test/coverage.overlay ${ghc}/bin/hpc overlay --output="$tmpdir"/overlay.tix test/coverage.overlay
@@ -250,6 +269,7 @@ buildToolbox
tools = { tools = {
inherit inherit
testSpec testSpec
testObservability
testDoctests testDoctests
testSpecIdempotence testSpecIdempotence
testIO testIO
+55 -162
View File
@@ -1,20 +1,25 @@
{ buildToolbox { buildToolbox
, checkedShellScript , checkedShellScript
, curl , curl
, git
, lib , lib
, libfaketime
, postgresqlVersions , postgresqlVersions
, postgrest , postgrest
, python3Packages , python3Packages
, slocat
, writeText , writeText
, writers , writers
}: }:
let let
withTmpDb = withTmpDb =
{ name, postgresql }: { name, postgresql, config ? "" }:
let let
commandName = "postgrest-with-${name}"; commandName = "postgrest-with-${name}";
postgresqlConf = writeText "postgresql.conf" ("
autovacuum = false
listen_addresses = ''
log_statement = all
shared_preload_libraries=pg_stat_statements
" + config);
in in
checkedShellScript checkedShellScript
{ {
@@ -25,7 +30,7 @@ let
"ARG_OPTIONAL_SINGLE([fixtures], [f], [SQL file to load fixtures from])" "ARG_OPTIONAL_SINGLE([fixtures], [f], [SQL file to load fixtures from])"
"ARG_POSITIONAL_SINGLE([command], [Command to run])" "ARG_POSITIONAL_SINGLE([command], [Command to run])"
"ARG_LEFTOVERS([command arguments])" "ARG_LEFTOVERS([command arguments])"
"ARG_USE_ENV([PGUSER], [postgrest_test_authenticator], [Authenticator PG role])" "ARG_USE_ENV([PGUSER], [Postgrest_Test_Authenticator], [Authenticator PG role])" # user is written in mixed case to implicitly test that it is being properly quoted in schema cache queries
"ARG_USE_ENV([PGDATABASE], [postgres], [PG database name])" "ARG_USE_ENV([PGDATABASE], [postgres], [PG database name])"
"ARG_USE_ENV([PGRST_DB_SCHEMAS], [test], [Schema to expose])" "ARG_USE_ENV([PGRST_DB_SCHEMAS], [test], [Schema to expose])"
"ARG_USE_ENV([PGTZ], [utc], [Timezone to use])" "ARG_USE_ENV([PGTZ], [utc], [Timezone to use])"
@@ -73,6 +78,10 @@ let
TZ=$PGTZ initdb --no-locale --encoding=UTF8 --nosync -U postgres --auth=trust \ TZ=$PGTZ initdb --no-locale --encoding=UTF8 --nosync -U postgres --auth=trust \
>> "$setuplog" >> "$setuplog"
# Append our own config to the one initdb created to avoid replacing
# default values created by the latter.
cat ${postgresqlConf} >> "$tmpdir/db/postgresql.conf"
log "Starting the database cluster..." log "Starting the database cluster..."
# Instead of listening on a local port, we will listen on a unix domain socket. # Instead of listening on a local port, we will listen on a unix domain socket.
@@ -81,7 +90,7 @@ let
# On MacOS, it's 104 chars # On MacOS, it's 104 chars
# See: https://serverfault.com/questions/641347/check-if-a-path-exceeds-maximum-for-unix-domain-socket # See: https://serverfault.com/questions/641347/check-if-a-path-exceeds-maximum-for-unix-domain-socket
pg_ctl -l "$tmpdir/db.log" -w start -o "-F -c listen_addresses=\"\" -c hba_file=$HBA_FILE -k $PGHOST -c log_statement=\"all\" " \ pg_ctl -l "$tmpdir/db.log" -w start -o "-F -c hba_file=$HBA_FILE -k $PGHOST " \
>> "$setuplog" >> "$setuplog"
log "Creating a minimally privileged $PGUSER connection role..." log "Creating a minimally privileged $PGUSER connection role..."
@@ -106,7 +115,8 @@ let
log "Starting replica on $replica_host" log "Starting replica on $replica_host"
pg_ctl -D "$replica_dir" -l "$replica_dblog" -w start -o "-F -c listen_addresses=\"\" -c hba_file=$HBA_FILE -k $replica_host -c log_statement=\"all\" " \ # We set a low max_standby_streaming_delay to make the replication conflict fail faster in tests (otherwise it waits for the default 30s)
pg_ctl -D "$replica_dir" -l "$replica_dblog" -w start -o "-F -c hba_file=$HBA_FILE -k $replica_host -c max_standby_streaming_delay=\"3s\" " \
>> "$setuplog" >> "$setuplog"
>&2 echo "${commandName}: Replica enabled. You can connect to it with: psql 'postgres:///$PGDATABASE?host=$replica_host' -U postgres" >&2 echo "${commandName}: Replica enabled. You can connect to it with: psql 'postgres:///$PGDATABASE?host=$replica_host' -U postgres"
@@ -117,7 +127,7 @@ let
export PGRST_DB_URI="postgres:///$PGDATABASE?host=$PGREPLICAHOST,$PGHOST" export PGRST_DB_URI="postgres:///$PGDATABASE?host=$PGREPLICAHOST,$PGHOST"
fi fi
# shellcheck disable=SC2317 # shellcheck disable=SC2329
stop () { stop () {
log "Stopping the database cluster..." log "Stopping the database cluster..."
pg_ctl stop --mode=immediate >> "$setuplog" pg_ctl stop --mode=immediate >> "$setuplog"
@@ -132,9 +142,12 @@ let
fi fi
if test "$_arg_fixtures"; then if test "$_arg_fixtures"; then
log "Loading fixtures under the postgres role..." load_start=$SECONDS
>&2 printf "${commandName}: Loading fixtures under the postgres role..."
psql -U postgres -v PGUSER="$PGUSER" -v ON_ERROR_STOP=1 -f "$_arg_fixtures" >> "$setuplog" psql -U postgres -v PGUSER="$PGUSER" -v ON_ERROR_STOP=1 -f "$_arg_fixtures" >> "$setuplog"
log "Done. Running command..." psql -U postgres -v ON_ERROR_STOP=1 -c "VACUUM ANALYZE;" >> "$setuplog"
load_end=$((SECONDS - load_start))
>&2 printf " done in %ss. Running command...\n" "$load_end"
fi fi
("$_arg_command" "''${_arg_leftovers[@]}") ("$_arg_command" "''${_arg_leftovers[@]}")
@@ -183,134 +196,6 @@ let
withPg = withTmpDb (builtins.head postgresqlVersions); withPg = withTmpDb (builtins.head postgresqlVersions);
withSlowPg =
checkedShellScript
{
name = "postgrest-with-slow-pg";
docs = "Run the given command with simulated high latency postgresql";
args =
[
"ARG_POSITIONAL_SINGLE([command], [Command to run])"
"ARG_LEFTOVERS([command arguments])"
"ARG_USE_ENV([PGHOST], [], [PG host (socket name)])"
"ARG_USE_ENV([PGDELAY], [0ms], [extra PG latency (duration)])"
];
positionalCompletion = "_command";
workingDir = "/";
redirectTixFiles = false;
withTmpDir = true;
}
''
delay="''${PGDELAY:-0ms}"
echo "delaying data to/from postgres by $delay"
REALPGHOST="$PGHOST"
export PGHOST="$tmpdir/socket"
mkdir -p "$PGHOST"
${slocat}/bin/slocat -delay "$delay" -src "$PGHOST/.s.PGSQL.5432" -dst "$REALPGHOST/.s.PGSQL.5432" &
SLOCAT_PID=$!
# shellcheck disable=SC2317
stop_slocat() {
kill "$SLOCAT_PID" || true
wait "$SLOCAT_PID" || true
}
trap stop_slocat EXIT
sleep 1 # should wait for socket file to appear instead
("$_arg_command" "''${_arg_leftovers[@]}")
'';
withSlowPgrst =
checkedShellScript
{
name = "postgrest-with-slow-postgrest";
docs = "Run the given command with simulated high latency postgrest";
args =
[
"ARG_POSITIONAL_SINGLE([command], [Command to run])"
"ARG_LEFTOVERS([command arguments])"
"ARG_USE_ENV([PGRST_SERVER_UNIX_SOCKET], [], [PostgREST host (socket name)])"
"ARG_USE_ENV([PGRST_DELAY], [0ms], [extra PostgREST latency (duration)])"
];
positionalCompletion = "_command";
workingDir = "/";
redirectTixFiles = false;
withTmpDir = true;
}
''
delay="''${PGRST_DELAY:-0ms}"
echo "delaying data to/from PostgREST by $delay"
REAL_PGRST_SERVER_UNIX_SOCKET="$PGRST_SERVER_UNIX_SOCKET"
export PGRST_SERVER_UNIX_SOCKET="$tmpdir/postgrest.socket"
${slocat}/bin/slocat -delay "$delay" -src "$PGRST_SERVER_UNIX_SOCKET" -dst "$REAL_PGRST_SERVER_UNIX_SOCKET" &
SLOCAT_PID=$!
# shellcheck disable=SC2317
stop_slocat() {
kill "$SLOCAT_PID" || true
wait "$SLOCAT_PID" || true
}
trap stop_slocat EXIT
sleep 1 # should wait for socket file to appear instead
("$_arg_command" "''${_arg_leftovers[@]}")
'';
withGit =
let
name = "postgrest-with-git";
in
checkedShellScript
{
inherit name;
docs =
''
Create a new worktree of the postgrest repo in a temporary directory and
check out <commit>, then run <command> with arguments inside the temporary folder.
'';
args =
[
"ARG_POSITIONAL_SINGLE([commit], [Commit-ish reference to run command with])"
"ARG_POSITIONAL_SINGLE([command], [Command to run])"
"ARG_LEFTOVERS([command arguments])"
];
positionalCompletion =
''
if test "$prev" == "${name}"; then
__gitcomp_nl "$(__git_refs)"
else
_command_offset 2
fi
'';
workingDir = "/";
}
''
# not using withTmpDir here, because we don't want to keep the directory on error
tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT
${git}/bin/git worktree add -f "$tmpdir" "$_arg_commit" > /dev/null
cd "$tmpdir"
("$_arg_command" "''${_arg_leftovers[@]}")
${git}/bin/git worktree remove -f "$tmpdir" > /dev/null
'';
legacyConfig =
writeText "legacy.conf"
''
# Using this config file to support older postgrest versions for `postgrest-loadtest-against`
db-uri="$(PGRST_DB_URI)"
db-schema="$(PGRST_DB_SCHEMAS)"
db-anon-role="$(PGRST_DB_ANON_ROLE)"
db-pool="$(PGRST_DB_POOL)"
server-unix-socket="$(PGRST_SERVER_UNIX_SOCKET)"
log-level="$(PGRST_LOG_LEVEL)"
'';
waitForPgrstReady = waitForPgrstReady =
checkedShellScript checkedShellScript
{ {
@@ -352,15 +237,23 @@ let
''; '';
withPgrst = withPgrst =
let
commandName = "postgrest-with-pgrst";
in
checkedShellScript checkedShellScript
{ {
name = "postgrest-with-pgrst"; name = commandName;
docs = "Build and run PostgREST and run <command> with PGRST_SERVER_UNIX_SOCKET set."; docs = "Build and run PostgREST and run <command> with PGRST_SERVER_UNIX_SOCKET set.";
args = args =
[ [
"ARG_POSITIONAL_SINGLE([command], [Command to run])" "ARG_POSITIONAL_SINGLE([command], [Command to run])"
"ARG_LEFTOVERS([command arguments])" "ARG_LEFTOVERS([command arguments])"
"ARG_OPTIONAL_SINGLE([faketime], [f], [Fake the system time when starting PostgREST. This is useful to test expiry of JWT, for example in loadtests])"
"ARG_OPTIONAL_SINGLE([monitor], [m], [Enable CPU and memory monitoring of the PostgREST process and output to the designated file as markdown])" "ARG_OPTIONAL_SINGLE([monitor], [m], [Enable CPU and memory monitoring of the PostgREST process and output to the designated file as markdown])"
"ARG_OPTIONAL_SINGLE([timeout], [t], [Maximum time to wait for PostgREST to be ready], [5])"
"ARG_OPTIONAL_SINGLE([sleep], [s], [Sleep time after PostgREST is ready, this is useful for monitoring])"
"ARG_USE_ENV([FAKETIME_LIB], [${libfaketime}/lib/libfaketime.so.1], [Faketime Library to preload])"
"ARG_USE_ENV([PGRST_CMD], [postgrest-run], [PostgREST executable to run])"
]; ];
positionalCompletion = "_command"; positionalCompletion = "_command";
workingDir = "/"; workingDir = "/";
@@ -370,30 +263,25 @@ let
'' ''
export PGRST_SERVER_UNIX_SOCKET="$tmpdir"/postgrest.socket export PGRST_SERVER_UNIX_SOCKET="$tmpdir"/postgrest.socket
rm -f result if [ "''${PGRST_CMD}" == "postgrest-run" ]; then
if [ -z "''${PGRST_BUILD_CABAL:-}" ]; then build_start=$SECONDS
echo -n "Building postgrest (nix)... " echo -n "${commandName}: Building postgrest (cabal)... "
# Using lib.getBin to also make this work with older checkouts, where .bin was not a thing, yet.
nix-build -E 'with import ./. {}; pkgs.lib.getBin postgrestPackage' > "$tmpdir"/build.log 2>&1 || {
echo "failed, output:"
cat "$tmpdir"/build.log
exit 1
}
PGRST_CMD=$(echo ./result*/bin/postgrest)
else
echo -n "Building postgrest (cabal)... "
postgrest-build postgrest-build
PGRST_CMD=postgrest-run build_end=$((SECONDS - build_start))
printf "done in %ss.\n" "$build_end"
fi fi
echo "done."
ver=$($PGRST_CMD ${legacyConfig} --version) ver=$($PGRST_CMD --version)
echo -n "Starting $ver... " echo -n "${commandName}: Starting $ver... "
$PGRST_CMD ${legacyConfig} > "$tmpdir"/run.log 2>&1 & if [[ -n "$_arg_faketime" ]]; then
LD_PRELOAD="$FAKETIME_LIB" FAKETIME="$_arg_faketime" "$PGRST_CMD" > "$tmpdir"/run.log 2>&1 &
else
$PGRST_CMD > "$tmpdir"/run.log 2>&1 &
fi
pid=$! pid=$!
# shellcheck disable=SC2317 # shellcheck disable=SC2329
cleanup() { cleanup() {
# Send INT to all postgrest processes. # Send INT to all postgrest processes.
# Workaround to trigger dumping postgrest.prof for postgrest-profiled-run # Workaround to trigger dumping postgrest.prof for postgrest-profiled-run
@@ -407,17 +295,25 @@ let
} }
trap cleanup EXIT trap cleanup EXIT
timeout -s TERM 5 ${waitForPgrstReady} || { wait_start=$SECONDS
timeout -s TERM "$_arg_timeout" ${waitForPgrstReady} || {
echo "timed out, output:" echo "timed out, output:"
cat "$tmpdir"/run.log cat "$tmpdir"/run.log
exit 1 exit 1
} }
echo "done." wait_duration=$((SECONDS - wait_start))
printf "done in %ss.\n" "$wait_duration"
echo "${commandName}: You can tail the server logs with: tail -f $tmpdir/run.log"
if [[ -n "$_arg_monitor" ]]; then if [[ -n "$_arg_monitor" ]]; then
${monitorPid} "$pid" > "$_arg_monitor" & ${monitorPid} "$pid" > "$_arg_monitor" &
fi fi
if [[ -n "$_arg_sleep" ]]; then
sleep "$_arg_sleep"
fi
("$_arg_command" "''${_arg_leftovers[@]}") ("$_arg_command" "''${_arg_leftovers[@]}")
''; '';
@@ -433,11 +329,8 @@ buildToolbox
name = "postgrest-with"; name = "postgrest-with";
tools = { tools = {
inherit inherit
withGit
withPgAll withPgAll
withPgrst withPgrst;
withSlowPg
withSlowPgrst;
} // builtins.listToAttrs ( } // builtins.listToAttrs (
# Create a `postgrest-with-pg-` for each PostgreSQL version # Create a `postgrest-with-pg-` for each PostgreSQL version
builtins.map (pg: { inherit (pg) name; value = withTmpDb pg; }) postgresqlVersions builtins.map (pg: { inherit (pg) name; value = withTmpDb pg; }) postgresqlVersions
+95 -44
View File
@@ -1,28 +1,27 @@
cabal-version: 3.0
name: postgrest name: postgrest
version: 14.2 version: 17
synopsis: REST API for any Postgres database synopsis: REST API for any Postgres database
description: Reads the schema of a PostgreSQL database and creates RESTful routes description: Reads the schema of a PostgreSQL database and creates RESTful routes
for tables, views, and functions, supporting all HTTP methods that security for tables, views, and functions, supporting all HTTP methods that security
permits. permits.
license: MIT license: MIT
license-file: LICENSE license-file: LICENSE
author: Joe Nelson, Adam Baker, Steve Chavez author: Joe Nelson, Adam Baker, Steve Chavez, Wolfgang Walther
maintainer: Steve Chavez <stevechavezast@gmail.com> maintainer: Steve Chavez <stevechavezast@gmail.com>
category: Executable, PostgreSQL, Network APIs category: Executable, PostgreSQL, Network APIs
homepage: https://postgrest.org homepage: https://postgrest.org
bug-reports: https://github.com/PostgREST/postgrest/issues bug-reports: https://github.com/PostgREST/postgrest/issues
build-type: Simple build-type: Simple
extra-source-files: CHANGELOG.md extra-source-files: CHANGELOG.md
cabal-version: >= 1.10
tested-with: tested-with:
-- nix
GHC == 9.4.8
-- cabal on Ubuntu -- cabal on Ubuntu
-- stack on FreeBSD, MacOS, Ubuntu, Windows -- stack on FreeBSD, MacOS, Ubuntu, Windows
, GHC == 9.6.7 , GHC == 9.10.3
-- cabal on Ubuntu -- cabal on Ubuntu
, GHC == 9.8.4 -- nix
, GHC == 9.12.3
source-repository head source-repository head
type: git type: git
@@ -39,13 +38,16 @@ flag hpc
description: Enable HPC (dev only) description: Enable HPC (dev only)
library library
default-language: Haskell2010 default-language: GHC2021
default-extensions: OverloadedStrings default-extensions: OverloadedStrings
NoImplicitPrelude NoImplicitPrelude
hs-source-dirs: src hs-source-dirs: src/library
exposed-modules: PostgREST.Admin exposed-modules: PostgREST.Admin
PostgREST.App PostgREST.App
PostgREST.AppState PostgREST.AppState
PostgREST.AppState.Pool
PostgREST.AppState.Reload
PostgREST.AppState.Types
PostgREST.Auth PostgREST.Auth
PostgREST.Auth.Jwt PostgREST.Auth.Jwt
PostgREST.Auth.JwtCache PostgREST.Auth.JwtCache
@@ -55,6 +57,7 @@ library
PostgREST.Client PostgREST.Client
PostgREST.Config PostgREST.Config
PostgREST.Config.Database PostgREST.Config.Database
PostgREST.Debounce
PostgREST.Config.JSPath PostgREST.Config.JSPath
PostgREST.Config.PgVersion PostgREST.Config.PgVersion
PostgREST.Config.Proxy PostgREST.Config.Proxy
@@ -66,9 +69,10 @@ library
PostgREST.SchemaCache.Representations PostgREST.SchemaCache.Representations
PostgREST.SchemaCache.Table PostgREST.SchemaCache.Table
PostgREST.Error PostgREST.Error
PostgREST.Listener PostgREST.Error.Types
PostgREST.Logger PostgREST.Logger
PostgREST.MainTx PostgREST.MainTx
PostgREST.Logger.Apache
PostgREST.MediaType PostgREST.MediaType
PostgREST.Metrics PostgREST.Metrics
PostgREST.Network PostgREST.Network
@@ -81,6 +85,7 @@ library
PostgREST.Plan PostgREST.Plan
PostgREST.Plan.CallPlan PostgREST.Plan.CallPlan
PostgREST.Plan.MutatePlan PostgREST.Plan.MutatePlan
PostgREST.Plan.Negotiate
PostgREST.Plan.ReadPlan PostgREST.Plan.ReadPlan
PostgREST.Plan.Types PostgREST.Plan.Types
PostgREST.RangeQuery PostgREST.RangeQuery
@@ -96,27 +101,32 @@ library
PostgREST.Response.Performance PostgREST.Response.Performance
PostgREST.TimeIt PostgREST.TimeIt
PostgREST.Version PostgREST.Version
build-depends: base >= 4.9 && < 4.20 build-depends: base >= 4.9 && < 4.22
, HTTP >= 4000.3.7 && < 4000.5 , HTTP >= 4000.3.7 && < 4000.5
, Ranged-sets >= 0.3 && < 0.5 , Ranged-sets >= 0.3 && < 0.6
, aeson >= 2.0.3 && < 2.3 , aeson >= 2.0.3 && < 2.3
, auto-update >= 0.1.4 && < 0.3 , aeson-jsonpath >= 0.4.2 && < 0.5
, auto-update >= 0.2.7 && < 0.3
, base64-bytestring >= 1 && < 1.3 , base64-bytestring >= 1 && < 1.3
, bytestring >= 0.10.8 && < 0.13 , bytestring >= 0.10.8 && < 0.13
, case-insensitive >= 1.2 && < 1.3 , case-insensitive >= 1.2 && < 1.3
, cassava >= 0.4.5 && < 0.6 , cassava >= 0.4.5 && < 0.6
, configurator-pg >= 0.2.11 && < 0.3 , configurator-pg >= 0.2.11 && < 0.3
, containers >= 0.5.7 && < 0.7 , containers >= 0.5.7 && < 0.8
, cookie >= 0.4.2 && < 0.6 , cookie >= 0.4.2 && < 0.6
-- crypton 1.1.0 moved from `memory` to `ram`, which jose-jwt fails to build with right now.
-- should be possible to remove this once jose-jwt had a new release.
, crypton < 1.1.0
, directory >= 1.2.6 && < 1.4 , directory >= 1.2.6 && < 1.4
, either >= 4.4.1 && < 5.1 , either >= 4.4.1 && < 5.1
, extra >= 1.7.0 && < 2.0 , extra >= 1.7.0 && < 2.0
, fast-logger >= 3.2.0 && < 3.3
, fuzzyset >= 0.2.4 && < 0.3 , fuzzyset >= 0.2.4 && < 0.3
, hasql >= 1.6.1.1 && < 1.7 , hasql >= 1.9 && <= 1.9.3.1
, hasql-dynamic-statements >= 0.3.1 && < 0.4 , hasql-dynamic-statements >= 0.3.1 && <= 0.3.1.8
, hasql-notifications >= 0.2.2.2 && < 0.2.3 , hasql-notifications >= 0.2.4.0 && < 0.3
, hasql-pool >= 1.0.1 && < 1.1 , hasql-pool >= 1.1 && <= 1.3.0.4
, hasql-transaction >= 1.0.1 && < 1.2 , hasql-transaction >= 1.0.1 && <= 1.2.1
, http-client >= 0.7.19 && < 0.8 , http-client >= 0.7.19 && < 0.8
, http-types >= 0.12.2 && < 0.13 , http-types >= 0.12.2 && < 0.13
, insert-ordered-containers >= 0.2.2 && < 0.3 , insert-ordered-containers >= 0.2.2 && < 0.3
@@ -129,9 +139,9 @@ library
, network-uri >= 2.6.1 && < 2.8 , network-uri >= 2.6.1 && < 2.8
, optparse-applicative >= 0.13 && < 0.19 , optparse-applicative >= 0.13 && < 0.19
, parsec >= 3.1.11 && < 3.2 , parsec >= 3.1.11 && < 3.2
-- Technically unused, can be removed after updating to hasql >= 1.7
, postgresql-libpq >= 0.10 , postgresql-libpq >= 0.10
, prometheus-client >= 1.1.1 && < 1.2.0 , prometheus-client >= 1.1.1 && < 1.2.0
, prometheus-metrics-ghc >= 1.0.1.2 && < 1.2
, protolude >= 0.3.1 && < 0.4 , protolude >= 0.3.1 && < 0.4
, regex-tdfa >= 1.2.2 && < 1.4 , regex-tdfa >= 1.2.2 && < 1.4
, retry >= 0.7.4 && < 0.10 , retry >= 0.7.4 && < 0.10
@@ -139,30 +149,30 @@ library
, streaming-commons >= 0.2.3.1 && < 0.3 , streaming-commons >= 0.2.3.1 && < 0.3
, swagger2 >= 2.4 && < 2.9 , swagger2 >= 2.4 && < 2.9
, text >= 1.2.2 && < 2.2 , text >= 1.2.2 && < 2.2
, time >= 1.6 && < 1.13 , time >= 1.6 && < 1.15
, unordered-containers >= 0.2.8 && < 0.3 , unordered-containers >= 0.2.8 && < 0.3
, unix-compat >= 0.5.4 && < 0.8 , unix-compat >= 0.5.4 && < 0.8
, vault >= 0.3.1.5 && < 0.4
, vector >= 0.11 && < 0.14 , vector >= 0.11 && < 0.14
, wai >= 3.2.1 && < 3.3 , wai >= 3.2.1 && < 3.3
, wai-cors >= 0.2.5 && < 0.3 , wai-cors >= 0.2.5 && < 0.3
, wai-extra >= 3.1.8 && < 3.2 , wai-extra >= 3.1.8 && < 3.2
-- We already depend on wai-logger >= 2.3.7 indirectly via wai-extra, -- We already depend on wai-logger >= 2.3.7 indirectly via wai-extra,
-- but we want to depend on 2.4.0 which fixes 'unknownSocket' log output -- but we want to depend on 2.4.0 which fixes 'unknownSocket' log output
-- for unix sockets; this is tested in test/io/test_io.py. See -- for unix sockets; this is tested in test/io/test_log.py. See
-- https://github.com/kazu-yamamoto/logger/commit/3a71ca70afdbb93d4ecf0083eeba1fbbbcab3fc3 -- https://github.com/kazu-yamamoto/logger/commit/3a71ca70afdbb93d4ecf0083eeba1fbbbcab3fc3
, wai-logger >= 2.4.0 , wai-logger >= 2.4.0
, warp >= 3.3.19 && < 3.5 , warp >= 3.4.14 && < 3.5
, stm >= 2.5 && < 3 , stm >= 2.5 && < 3
, stm-hamt >= 1.2 && < 2 , stm-hamt >= 1.2 && < 2
, focus >= 1.0 && < 2 , focus >= 1.0 && < 2
, some >= 1.0.4.1 && < 2 , some >= 1.0.4.1 && < 2
, uuid >= 1.3 && < 2
-- -fno-spec-constr may help keep compile time memory use in check, -- -fno-spec-constr may help keep compile time memory use in check,
-- see https://gitlab.haskell.org/ghc/ghc/issues/16017#note_219304 -- see https://gitlab.haskell.org/ghc/ghc/issues/16017#note_219304
-- -optP-Wno-nonportable-include-path -- -optP-Wno-nonportable-include-path
-- prevents build failures on case-insensitive filesystems (macos), -- prevents build failures on case-insensitive filesystems (macos),
-- see https://github.com/commercialhaskell/stack/issues/3918 -- see https://github.com/commercialhaskell/stack/issues/3918
ghc-options: -Werror -Wall -fwarn-identities ghc-options: -j -Werror -Wall -fwarn-identities
-fno-spec-constr -optP-Wno-nonportable-include-path -fno-spec-constr -optP-Wno-nonportable-include-path
if flag(dev) if flag(dev)
@@ -171,22 +181,27 @@ library
ghc-options: -fhpc -hpcdir .hpc ghc-options: -fhpc -hpcdir .hpc
else else
ghc-options: -O2 ghc-options: -O2
if impl(ghc >= 9.12)
-- Makes GHC consider cross-module specialization for polymorphic functions
-- without explicitly needing to add INLINE, INLINABLE or SPECIALIZE pragmas.
-- Slightly increases the binary size but improves performance considerably.
ghc-options: -fexpose-overloaded-unfoldings -fspecialise-aggressively
if !os(windows) if !os(windows)
build-depends: build-depends:
unix unix
executable postgrest executable postgrest
default-language: Haskell2010 default-language: GHC2021
default-extensions: OverloadedStrings default-extensions: OverloadedStrings
NoImplicitPrelude NoImplicitPrelude
hs-source-dirs: main hs-source-dirs: src/executable
main-is: Main.hs main-is: Main.hs
build-depends: base >= 4.9 && < 4.20 build-depends: base >= 4.9 && < 4.22
, containers >= 0.5.7 && < 0.7 , containers >= 0.5.7 && < 0.8
, postgrest , postgrest
, protolude >= 0.3.1 && < 0.4 , protolude >= 0.3.1 && < 0.4
ghc-options: -threaded -rtsopts "-with-rtsopts=-N -I0 -qg" ghc-options: -j -threaded -rtsopts "-with-rtsopts=-N -I0 -qg"
-O2 -Werror -Wall -fwarn-identities -O2 -Werror -Wall -fwarn-identities
-fno-spec-constr -optP-Wno-nonportable-include-path -fno-spec-constr -optP-Wno-nonportable-include-path
@@ -201,7 +216,7 @@ executable postgrest
test-suite spec test-suite spec
type: exitcode-stdio-1.0 type: exitcode-stdio-1.0
default-language: Haskell2010 default-language: GHC2021
default-extensions: OverloadedStrings default-extensions: OverloadedStrings
QuasiQuotes QuasiQuotes
NoImplicitPrelude NoImplicitPrelude
@@ -216,6 +231,7 @@ test-suite spec
Feature.Auth.NoJwtSecretSpec Feature.Auth.NoJwtSecretSpec
Feature.ConcurrentSpec Feature.ConcurrentSpec
Feature.CorsSpec Feature.CorsSpec
Feature.HttpHeaderSpec
Feature.ExtraSearchPathSpec Feature.ExtraSearchPathSpec
Feature.NoSuperuserSpec Feature.NoSuperuserSpec
Feature.ObservabilitySpec Feature.ObservabilitySpec
@@ -241,7 +257,10 @@ test-suite spec
Feature.Query.PgSafeUpdateSpec Feature.Query.PgSafeUpdateSpec
Feature.Query.PlanSpec Feature.Query.PlanSpec
Feature.Query.PostGISSpec Feature.Query.PostGISSpec
Feature.Query.PreferencesSpec Feature.Query.Preferences.HandlingSpec
Feature.Query.Preferences.MaxAffectedSpec
Feature.Query.Preferences.TimezoneSpec
Feature.Query.PreparedStatementsSpec
Feature.Query.QueryLimitedSpec Feature.Query.QueryLimitedSpec
Feature.Query.QuerySpec Feature.Query.QuerySpec
Feature.Query.RangeSpec Feature.Query.RangeSpec
@@ -257,16 +276,16 @@ test-suite spec
Feature.RollbackSpec Feature.RollbackSpec
Feature.RpcPreRequestGucsSpec Feature.RpcPreRequestGucsSpec
SpecHelper SpecHelper
build-depends: base >= 4.9 && < 4.20 build-depends: base >= 4.9 && < 4.22
, aeson >= 2.0.3 && < 2.3 , aeson >= 2.0.3 && < 2.3
, aeson-qq >= 0.8.1 && < 0.9 , aeson-qq >= 0.8.1 && < 0.9
, async >= 2.1.1 && < 2.3 , async >= 2.1.1 && < 2.3
, base64-bytestring >= 1 && < 1.3 , base64-bytestring >= 1 && < 1.3
, bytestring >= 0.10.8 && < 0.13 , bytestring >= 0.10.8 && < 0.13
, case-insensitive >= 1.2 && < 1.3 , case-insensitive >= 1.2 && < 1.3
, containers >= 0.5.7 && < 0.7 , containers >= 0.5.7 && < 0.8
, hasql-pool >= 1.0.1 && < 1.1 , hasql-pool >= 1.0.1 && <= 1.3.0.4
, hasql-transaction >= 1.0.1 && < 1.2 , hasql-transaction >= 1.0.1 && <= 1.2.1
, heredoc >= 0.2 && < 0.3 , heredoc >= 0.2 && < 0.3
, hspec >= 2.3 && < 2.12 , hspec >= 2.3 && < 2.12
, hspec-expectations >= 0.8.4 && < 0.9 , hspec-expectations >= 0.8.4 && < 0.9
@@ -284,27 +303,59 @@ test-suite spec
, regex-tdfa >= 1.2.2 && < 1.4 , regex-tdfa >= 1.2.2 && < 1.4
, scientific >= 0.3.4 && < 0.4 , scientific >= 0.3.4 && < 0.4
, text >= 1.2.2 && < 2.2 , text >= 1.2.2 && < 2.2
, time >= 1.6 && < 1.15
, transformers-base >= 0.4.4 && < 0.5 , transformers-base >= 0.4.4 && < 0.5
, wai >= 3.2.1 && < 3.3 , wai >= 3.2.1 && < 3.3
, wai-extra >= 3.0.19 && < 3.2 , wai-extra >= 3.0.19 && < 3.2
ghc-options: -threaded -O0 -Werror -Wall -fwarn-identities ghc-options: -j -threaded -O0 -Werror -Wall -fwarn-identities
-fno-spec-constr -optP-Wno-nonportable-include-path -fno-spec-constr -optP-Wno-nonportable-include-path
-fno-warn-missing-signatures -fno-warn-missing-signatures
-fwrite-ide-info -fwrite-ide-info
-- https://github.com/PostgREST/postgrest/issues/387 -- https://github.com/PostgREST/postgrest/issues/387
-with-rtsopts=-K33K -with-rtsopts=-K33K
test-suite observability
type: exitcode-stdio-1.0
default-language: GHC2021
default-extensions: OverloadedStrings
QuasiQuotes
NoImplicitPrelude
hs-source-dirs: test/observability
main-is: Main.hs
other-modules: ObsHelper
Observation.JwtCache
Observation.MetricsSpec
Observation.SchemaCacheSpec
build-depends: base >= 4.9 && < 4.22
, base64-bytestring >= 1 && < 1.3
, bytestring >= 0.10.8 && < 0.13
, hasql-pool >= 1.0.1 && <= 1.3.0.4
, hasql-transaction >= 1.0.1 && <= 1.2.1
, hspec >= 2.3 && < 2.12
, hspec-expectations >= 0.8.4 && < 0.9
, hspec-wai >= 0.10 && < 0.12
, hspec-wai-json >= 0.10 && < 0.12
, http-types >= 0.12.3 && < 0.13
, jose-jwt >= 0.9.6 && < 0.11
, postgrest
, prometheus-client >= 1.1.1 && < 1.2.0
, protolude >= 0.3.1 && < 0.4
, text >= 1.2.2 && < 2.2
, wai >= 3.2.1 && < 3.3
ghc-options: -j -threaded -O0 -Werror -Wall -fwarn-identities
-fno-spec-constr -optP-Wno-nonportable-include-path
-fwrite-ide-info
-- https://github.com/PostgREST/postgrest/issues/387
-with-rtsopts=-K33K
test-suite doctests test-suite doctests
type: exitcode-stdio-1.0 type: exitcode-stdio-1.0
default-language: Haskell2010 default-language: GHC2021
default-extensions: OverloadedStrings
NoImplicitPrelude
hs-source-dirs: test/doc hs-source-dirs: test/doc
main-is: Main.hs main-is: Main.hs
build-depends: base >= 4.9 && < 4.20 build-depends: base >= 4.9 && < 4.22
, doctest >= 0.8 , doctest-parallel >= 0.4
, postgrest , postgrest
, pretty-simple , pretty-simple
, protolude >= 0.3.1 && < 0.4 ghc-options: -j -threaded -O0 -Werror -Wall -fwarn-identities
ghc-options: -threaded -O0 -Werror -Wall -fwarn-identities
-fno-spec-constr -optP-Wno-nonportable-include-path -fno-spec-constr -optP-Wno-nonportable-include-path
+5 -6
View File
@@ -7,11 +7,9 @@
# We highly recommend that use the PostgREST binary cache by installing cachix # We highly recommend that use the PostgREST binary cache by installing cachix
# (https://app.cachix.org/) and running `cachix use postgrest`. # (https://app.cachix.org/) and running `cachix use postgrest`.
{ docker ? false { docker ? false
, postgrest ? import ./default.nix { }
}: }:
let let
postgrest =
import ./default.nix { };
inherit (postgrest) pkgs; inherit (postgrest) pkgs;
inherit (pkgs) lib; inherit (pkgs) lib;
@@ -37,10 +35,7 @@ lib.overrideDerivation postgrest.env (
buildInputs = buildInputs =
base.buildInputs ++ [ base.buildInputs ++ [
pkgs.cabal-install pkgs.cabal-install
pkgs.cabal2nix
pkgs.git
pkgs.postgresql pkgs.postgresql
pkgs.update-nix-fetchgit
postgrest.hsie.bin postgrest.hsie.bin
] ]
++ toolboxes; ++ toolboxes;
@@ -49,6 +44,10 @@ lib.overrideDerivation postgrest.env (
'' ''
export HISTFILE=.history export HISTFILE=.history
# Bypass proxy for all hosts, it prevents HTTP client failures used in test
# suites. See: https://github.com/PostgREST/postgrest/issues/4633 for more info
export NO_PROXY=*
source ${pkgs.bash-completion}/etc/profile.d/bash_completion.sh source ${pkgs.bash-completion}/etc/profile.d/bash_completion.sh
source ${pkgs.git}/share/git/contrib/completion/git-completion.bash source ${pkgs.git}/share/git/contrib/completion/git-completion.bash
source ${postgrest.hsie.bash-completion} source ${postgrest.hsie.bash-completion}
-74
View File
@@ -1,74 +0,0 @@
module PostgREST.Admin
( runAdmin
) where
import qualified Data.Aeson as JSON
import qualified Network.HTTP.Types.Status as HTTP
import qualified Network.Wai as Wai
import qualified Network.Wai.Handler.Warp as Warp
import Control.Monad.Extra (whenJust)
import Network.Socket hiding (addrFamily)
import Network.Socket.ByteString
import PostgREST.AppState (AppState)
import PostgREST.MediaType (MediaType (..), toContentType)
import PostgREST.Metrics (metricsToText)
import PostgREST.Network (resolveSocketToAddress)
import PostgREST.Observation (Observation (..))
import qualified PostgREST.AppState as AppState
import Protolude
runAdmin :: AppState -> Warp.Settings -> IO ()
runAdmin appState settings = do
whenJust (AppState.getSocketAdmin appState) $ \adminSocket -> do
address <- resolveSocketToAddress adminSocket
observer $ AdminStartObs address
void . forkIO $ Warp.runSettingsSocket settings adminSocket adminApp
where
adminApp = admin appState
observer = AppState.getObserver appState
-- | PostgREST admin application
admin :: AppState.AppState -> Wai.Application
admin appState req respond = do
isMainAppReachable <- isRight <$> reachMainApp (AppState.getSocketREST appState)
isLoaded <- AppState.isLoaded appState
isPending <- AppState.isPending appState
case Wai.pathInfo req of
["live"] ->
respond $ Wai.responseLBS (if isMainAppReachable then HTTP.status200 else HTTP.status500) [] mempty
["ready"] ->
let
status | not isMainAppReachable = HTTP.status500
| isPending = HTTP.status503
| isLoaded = HTTP.status200
| otherwise = HTTP.status500
in
respond $ Wai.responseLBS status [] mempty
["schema_cache"] -> do
sCache <- AppState.getSchemaCache appState
respond $ Wai.responseLBS HTTP.status200 [] (maybe mempty JSON.encode sCache)
["metrics"] -> do
mets <- metricsToText
respond $ Wai.responseLBS HTTP.status200 [toContentType MTTextPlain] mets -- Content-Type is required for prometheus compliance
_ ->
respond $ Wai.responseLBS HTTP.status404 [] mempty
-- Try to connect to the main app socket
-- Note that it doesn't even send a valid HTTP request, we just want to check that the main app is accepting connections
reachMainApp :: Socket -> IO (Either IOException ())
reachMainApp appSock = do
sockAddr <- getSocketName appSock
sock <- socket (addrFamily sockAddr) Stream defaultProtocol
try $ do
connect sock sockAddr
withSocketsDo $ bracket (pure sock) close sendEmpty
where
sendEmpty sock = void $ send sock mempty
addrFamily (SockAddrInet _ _) = AF_INET
addrFamily (SockAddrInet6 {}) = AF_INET6
addrFamily (SockAddrUnix _) = AF_UNIX
-208
View File
@@ -1,208 +0,0 @@
{-|
Module : PostgREST.App
Description : PostgREST main application
This module is in charge of mapping HTTP requests to PostgreSQL queries.
Some of its functionality includes:
- Mapping HTTP request methods to proper SQL statements. For example, a GET request is translated to executing a SELECT query in a read-only TRANSACTION.
- Producing HTTP Headers according to RFCs.
- Content Negotiation
-}
{-# LANGUAGE RecordWildCards #-}
module PostgREST.App
( postgrest
, run
) where
import Control.Monad.Except (liftEither)
import Data.Either.Combinators (mapLeft, whenLeft)
import Data.Maybe (fromJust)
import Data.String (IsString (..))
import Network.Wai.Handler.Warp (defaultSettings, setHost, setPort,
setServerName)
import qualified Data.Text.Encoding as T
import qualified Network.Wai as Wai
import qualified Network.Wai.Handler.Warp as Warp
import qualified PostgREST.Admin as Admin
import qualified PostgREST.ApiRequest as ApiRequest
import qualified PostgREST.AppState as AppState
import qualified PostgREST.Auth as Auth
import qualified PostgREST.Cors as Cors
import qualified PostgREST.Error as Error
import qualified PostgREST.Listener as Listener
import qualified PostgREST.Logger as Logger
import qualified PostgREST.MainTx as MainTx
import qualified PostgREST.Plan as Plan
import qualified PostgREST.Query as Query
import qualified PostgREST.Response as Response
import qualified PostgREST.Unix as Unix (installSignalHandlers)
import PostgREST.ApiRequest (ApiRequest (..))
import PostgREST.AppState (AppState)
import PostgREST.Auth.Types (AuthResult (..))
import PostgREST.Config (AppConfig (..), LogLevel (..))
import PostgREST.Error (Error)
import PostgREST.Network (resolveSocketToAddress)
import PostgREST.Observation (Observation (..))
import PostgREST.Response.Performance (ServerTiming (..),
serverTimingHeader)
import PostgREST.SchemaCache (SchemaCache (..))
import PostgREST.TimeIt (timeItT)
import PostgREST.Version (docsVersion, prettyVersion)
import qualified Data.ByteString.Char8 as BS
import qualified Data.List as L
import qualified Network.HTTP.Types as HTTP
import Protolude hiding (Handler)
type Handler = ExceptT Error
run :: AppState -> IO ()
run appState = do
let observer = AppState.getObserver appState
conf@AppConfig{..} <- AppState.getConfig appState
AppState.schemaCacheLoader appState -- Loads the initial SchemaCache
Unix.installSignalHandlers (AppState.getMainThreadId appState) (AppState.schemaCacheLoader appState) (AppState.readInDbConfig False appState)
Listener.runListener appState
Admin.runAdmin appState (serverSettings conf)
let app = postgrest configLogLevel appState (AppState.schemaCacheLoader appState)
do
address <- resolveSocketToAddress (AppState.getSocketREST appState)
observer $ AppServerAddressObs address
Warp.runSettingsSocket (serverSettings conf) (AppState.getSocketREST appState) app
serverSettings :: AppConfig -> Warp.Settings
serverSettings AppConfig{..} =
defaultSettings
& setHost (fromString $ toS configServerHost)
& setPort configServerPort
& setServerName ("postgrest/" <> prettyVersion)
-- | PostgREST application
postgrest :: LogLevel -> AppState.AppState -> IO () -> Wai.Application
postgrest logLevel appState connWorker =
traceHeaderMiddleware appState .
Cors.middleware appState .
Auth.middleware appState .
Logger.middleware logLevel Auth.getRole $
-- fromJust can be used, because the auth middleware will **always** add
-- some AuthResult to the vault.
\req respond -> case fromJust $ Auth.getResult req of
Left err -> respond $ Error.errorResponseFor err
Right authResult -> do
appConf <- AppState.getConfig appState -- the config must be read again because it can reload
maybeSchemaCache <- AppState.getSchemaCache appState
let
eitherResponse :: IO (Either Error Wai.Response)
eitherResponse =
runExceptT $ postgrestResponse appState appConf maybeSchemaCache authResult req
response <- either Error.errorResponseFor identity <$> eitherResponse
-- Launch the connWorker when the connection is down. The postgrest
-- function can respond successfully (with a stale schema cache) before
-- the connWorker is done. However, when there's an empty schema cache
-- postgrest responds with the error `PGRST002`; this means that the schema
-- cache is still loading, so we don't launch the connWorker here because
-- it would duplicate the loading process, e.g. https://github.com/PostgREST/postgrest/issues/3704
-- TODO: this process may be unnecessary when the Listener is enabled. Revisit once https://github.com/PostgREST/postgrest/issues/1766 is done
when (isServiceUnavailable response && isJust maybeSchemaCache) connWorker
resp <- do
delay <- AppState.getNextDelay appState
return $ addRetryHint delay response
respond resp
postgrestResponse
:: AppState.AppState
-> AppConfig
-> Maybe SchemaCache
-> AuthResult
-> Wai.Request
-> Handler IO Wai.Response
postgrestResponse appState conf@AppConfig{..} maybeSchemaCache authResult@AuthResult{..} req = do
let observer = AppState.getObserver appState
sCache <-
case maybeSchemaCache of
Just sCache ->
return sCache
Nothing -> do
lift $ observer SchemaCacheEmptyObs
throwError Error.NoSchemaCacheError
body <- lift $ Wai.strictRequestBody req
let jwtTime = if configServerTimingEnabled then Auth.getJwtDur req else Nothing
timezones = dbTimezones sCache
prefs = ApiRequest.userPreferences conf req timezones
(parseTime, apiReq@ApiRequest{..}) <- withTiming $ liftEither . mapLeft Error.ApiRequestError $ ApiRequest.userApiRequest conf prefs req body
(planTime, plan) <- withTiming $ liftEither $ Plan.actionPlan iAction conf apiReq sCache
let mainQ = Query.mainQuery plan conf apiReq authResult configDbPreRequest
tx = MainTx.mainTx mainQ conf authResult apiReq plan sCache
obsQuery s = when configLogQuery $ observer $ QueryObs mainQ s
(txTime, txResult) <- withTiming $ do
case tx of
MainTx.NoDbTx r -> pure r
MainTx.DbTx{..} -> do
dbRes <- lift $ AppState.usePool appState (dqTransaction dqIsoLevel dqTxMode $ runExceptT dqDbHandler)
let eitherResp = join $ mapLeft (Error.PgErr . Error.PgError (Just authRole /= configDbAnonRole)) dbRes
-- TODO: we use obsQuery twice, one here and one below because in case of an error with the usePool above, the request will finish here and return an error message.
-- This is because of a combination of ExceptT + our Error module which has Wai.responseLBS.
-- This needs refactoring so only the below obsQuery is used.
lift $ whenLeft eitherResp $ obsQuery . Error.status
liftEither eitherResp
(respTime, resp) <- withTiming $ do
let response = Response.actionResponse txResult apiReq (T.decodeUtf8 prettyVersion, docsVersion) conf sCache iSchema iNegotiatedByProfile
status' = either Error.status Response.pgrstStatus response
-- TODO: see above obsQuery, only this obsQuery should remain after refactoring (because the QueryObs depends on the status)
lift $ obsQuery status'
liftEither response
return $ toWaiResponse (ServerTiming jwtTime parseTime planTime txTime respTime) resp
where
toWaiResponse :: ServerTiming -> Response.PgrstResponse -> Wai.Response
toWaiResponse timing (Response.PgrstResponse st hdrs bod) = Wai.responseLBS st (hdrs ++ ([serverTimingHeader timing | configServerTimingEnabled])) bod
withTiming :: Handler IO a -> Handler IO (Maybe Double, a)
withTiming f = if configServerTimingEnabled
then do
(t, r) <- timeItT f
pure (Just t, r)
else do
r <- f
pure (Nothing, r)
traceHeaderMiddleware :: AppState -> Wai.Middleware
traceHeaderMiddleware appState app req respond = do
conf <- AppState.getConfig appState
case configServerTraceHeader conf of
Nothing -> app req respond
Just hdr ->
let hdrVal = L.lookup hdr $ Wai.requestHeaders req in
app req (respond . Wai.mapResponseHeaders ([(hdr, fromMaybe mempty hdrVal)] ++))
addRetryHint :: Int -> Wai.Response -> Wai.Response
addRetryHint delay response = do
let h = ("Retry-After", BS.pack $ show delay)
Wai.mapResponseHeaders (\hs -> if isServiceUnavailable response then h:hs else hs) response
isServiceUnavailable :: Wai.Response -> Bool
isServiceUnavailable response = Wai.responseStatus response == HTTP.status503
-480
View File
@@ -1,480 +0,0 @@
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE RecordWildCards #-}
module PostgREST.AppState
( AppState
, destroy
, getConfig
, getSchemaCache
, getMainThreadId
, getPgVersion
, getNextDelay
, getNextListenerDelay
, getTime
, getJwtCacheState
, getSocketREST
, getSocketAdmin
, init
, initSockets
, initWithPool
, putNextListenerDelay
, putSchemaCache
, putPgVersion
, putIsListenerOn
, usePool
, readInDbConfig
, schemaCacheLoader
, getObserver
, isLoaded
, isPending
) where
import qualified Data.ByteString.Char8 as BS
import Data.Either.Combinators (whenLeft)
import qualified Data.Text as T (unpack)
import qualified Hasql.Pool as SQL
import qualified Hasql.Pool.Config as SQL
import qualified Hasql.Session as SQL
import qualified Hasql.Transaction.Sessions as SQL
import qualified Network.HTTP.Types.Status as HTTP
import qualified Network.Socket as NS
import qualified PostgREST.Auth.JwtCache as JwtCache
import qualified PostgREST.Error as Error
import qualified PostgREST.Logger as Logger
import qualified PostgREST.Metrics as Metrics
import PostgREST.Observation
import PostgREST.TimeIt (timeItT)
import PostgREST.Version (prettyVersion)
import Control.AutoUpdate (defaultUpdateSettings, mkAutoUpdate,
updateAction)
import Control.Debounce
import Control.Retry (RetryPolicy, RetryStatus (..), capDelay,
exponentialBackoff, retrying,
rsPreviousDelay)
import Data.IORef (IORef, atomicWriteIORef, newIORef,
readIORef)
import Data.Time.Clock (UTCTime, getCurrentTime)
import PostgREST.Auth.JwtCache (JwtCacheState, update)
import PostgREST.Config (AppConfig (..),
addFallbackAppName,
readAppConfig)
import PostgREST.Config.Database (queryDbSettings,
queryPgVersion,
queryRoleSettings)
import PostgREST.Config.PgVersion (PgVersion (..),
minimumPgVersion)
import PostgREST.SchemaCache (SchemaCache (..),
querySchemaCache,
showSummary)
import PostgREST.SchemaCache.Identifiers (quoteQi)
import PostgREST.Unix (createAndBindDomainSocket)
import Data.Streaming.Network (bindPortTCP, bindRandomPortTCP)
import Data.String (IsString (..))
import Protolude
data AppState = AppState
-- | Database connection pool
{ statePool :: SQL.Pool
-- | Database server version
, statePgVersion :: IORef PgVersion
-- | Schema cache
, stateSchemaCache :: IORef (Maybe SchemaCache)
-- | The schema cache status
, stateSCacheStatus :: IORef SchemaCacheStatus
-- | State of the LISTEN channel
, stateIsListenerOn :: IORef Bool
-- | starts the connection worker with a debounce
, debouncedSCacheLoader :: IO ()
-- | Config that can change at runtime
, stateConf :: IORef AppConfig
-- | Time used for verifying JWT expiration
, stateGetTime :: IO UTCTime
-- | Used for killing the main thread in case a subthread fails
, stateMainThreadId :: ThreadId
-- | Keeps track of the next delay for db connection retry
, stateNextDelay :: IORef Int
-- | Keeps track of the next delay for the listener
, stateNextListenerDelay :: IORef Int
-- | Network socket for REST API
, stateSocketREST :: NS.Socket
-- | Network socket for the admin UI
, stateSocketAdmin :: Maybe NS.Socket
-- | Observation handler
, stateObserver :: ObservationHandler
-- | JWT Cache
, stateJwtCache :: JwtCache.JwtCacheState
, stateLogger :: Logger.LoggerState
, stateMetrics :: Metrics.MetricsState
}
-- | Schema cache status
data SchemaCacheStatus
= SCLoaded
| SCPending
deriving Eq
type AppSockets = (NS.Socket, Maybe NS.Socket)
init :: AppConfig -> IO AppState
init conf@AppConfig{configLogLevel, configDbPoolSize} = do
loggerState <- Logger.init
metricsState <- Metrics.init configDbPoolSize
let observer = liftA2 (>>) (Logger.observationLogger loggerState configLogLevel) (Metrics.observationMetrics metricsState)
observer $ AppStartObs prettyVersion
pool <- initPool conf observer
(sock, adminSock) <- initSockets conf
state' <- initWithPool (sock, adminSock) pool conf loggerState metricsState observer
pure state' { stateSocketREST = sock, stateSocketAdmin = adminSock}
initWithPool :: AppSockets -> SQL.Pool -> AppConfig -> Logger.LoggerState -> Metrics.MetricsState -> ObservationHandler -> IO AppState
initWithPool (sock, adminSock) pool conf loggerState metricsState observer = do
appState <- AppState pool
<$> newIORef minimumPgVersion -- assume we're in a supported version when starting, this will be corrected on a later step
<*> newIORef Nothing
<*> newIORef SCPending
<*> newIORef False
<*> pure (pure ())
<*> newIORef conf
<*> mkAutoUpdate defaultUpdateSettings { updateAction = getCurrentTime }
<*> myThreadId
<*> newIORef 0
<*> newIORef 1
<*> pure sock
<*> pure adminSock
<*> pure observer
<*> JwtCache.init conf observer
<*> pure loggerState
<*> pure metricsState
deb <-
let decisecond = 100000 in
mkDebounce defaultDebounceSettings
{ debounceAction = retryingSchemaCacheLoad appState
, debounceFreq = decisecond
, debounceEdge = leadingEdge -- runs the worker at the start and the end
}
return appState { debouncedSCacheLoader = deb}
destroy :: AppState -> IO ()
destroy = destroyPool
initSockets :: AppConfig -> IO AppSockets
initSockets AppConfig{..} = do
let
cfg'usp = configServerUnixSocket
cfg'uspm = configServerUnixSocketMode
cfg'host = configServerHost
cfg'port = configServerPort
cfg'adminHost = configAdminServerHost
cfg'adminPort = configAdminServerPort
sock <- case cfg'usp of
-- I'm not using `streaming-commons`' bindPath function here because it's not defined for Windows,
-- but we need to have runtime error if we try to use it in Windows, not compile time error
Just path -> createAndBindDomainSocket path cfg'uspm
Nothing -> do
(_, sock) <-
if cfg'port /= 0
then do
sock <- bindPortTCP cfg'port (fromString $ T.unpack cfg'host)
pure (cfg'port, sock)
else do
-- explicitly bind to a random port, returning bound port number
(num, sock) <- bindRandomPortTCP (fromString $ T.unpack cfg'host)
pure (num, sock)
pure sock
adminSock <- case cfg'adminPort of
Just adminPort -> do
adminSock <- bindPortTCP adminPort (fromString $ T.unpack cfg'adminHost)
pure $ Just adminSock
Nothing -> pure Nothing
pure (sock, adminSock)
initPool :: AppConfig -> ObservationHandler -> IO SQL.Pool
initPool AppConfig{..} observer = do
SQL.acquire $ SQL.settings
[ SQL.size configDbPoolSize
, SQL.acquisitionTimeout $ fromIntegral configDbPoolAcquisitionTimeout
, SQL.agingTimeout $ fromIntegral configDbPoolMaxLifetime
, SQL.idlenessTimeout $ fromIntegral configDbPoolMaxIdletime
, SQL.staticConnectionSettings (toUtf8 $ addFallbackAppName prettyVersion configDbUri)
, SQL.observationHandler $ observer . HasqlPoolObs
]
-- | Run an action with a database connection.
usePool :: AppState -> SQL.Session a -> IO (Either SQL.UsageError a)
usePool AppState{stateObserver=observer, stateMainThreadId=mainThreadId, ..} sess = do
observer PoolRequest
res <- SQL.use statePool sess
observer PoolRequestFullfilled
whenLeft res (\case
SQL.AcquisitionTimeoutUsageError ->
observer $ PoolAcqTimeoutObs SQL.AcquisitionTimeoutUsageError
err@(SQL.ConnectionUsageError e) ->
let failureMessage = BS.unpack $ fromMaybe mempty e in
when (("FATAL: password authentication failed" `isInfixOf` failureMessage) || ("no password supplied" `isInfixOf` failureMessage)) $ do
observer $ ExitDBFatalError ServerAuthError err
killThread mainThreadId
err@(SQL.SessionUsageError (SQL.QueryError tpl _ (SQL.ResultError resultErr))) -> do
case resultErr of
SQL.UnexpectedResult{} -> do
observer $ ExitDBFatalError ServerPgrstBug err
killThread mainThreadId
SQL.RowError{} -> do
observer $ ExitDBFatalError ServerPgrstBug err
killThread mainThreadId
SQL.UnexpectedAmountOfRows{} -> do
observer $ ExitDBFatalError ServerPgrstBug err
killThread mainThreadId
-- Check for a syntax error (42601 is the pg code) only for queries that don't have `WITH pgrst_source` as prefix.
-- This would mean the error is on our schema cache queries, so we treat it as fatal.
-- TODO have a better way to mark this as a schema cache query
SQL.ServerError "42601" _ _ _ _ ->
unless ("WITH pgrst_source" `BS.isPrefixOf` tpl) $ do
observer $ ExitDBFatalError ServerPgrstBug err
killThread mainThreadId
-- Check for a "prepared statement <name> already exists" error (Code 42P05: duplicate_prepared_statement).
-- This would mean that a connection pooler in transaction mode is being used
-- while prepared statements are enabled in the PostgREST configuration,
-- both of which are incompatible with each other.
SQL.ServerError "42P05" _ _ _ _ -> do
observer $ ExitDBFatalError ServerError42P05 err
killThread mainThreadId
-- Check for a "transaction blocks not allowed in statement pooling mode" error (Code 08P01: protocol_violation).
-- This would mean that a connection pooler in statement mode is being used which is not supported in PostgREST.
SQL.ServerError "08P01" "transaction blocks not allowed in statement pooling mode" _ _ _ -> do
observer $ ExitDBFatalError ServerError08P01 err
killThread mainThreadId
SQL.ServerError{} ->
when (Error.status (Error.PgError False err) >= HTTP.status500) $
observer $ QueryErrorCodeHighObs err
err@(SQL.SessionUsageError (SQL.QueryError _ _ (SQL.ClientError _))) ->
-- An error on the client-side, usually indicates problems wth connection
observer $ QueryErrorCodeHighObs err
)
return res
-- | Flush the connection pool so that any future use of the pool will
-- use connections freshly established after this call.
flushPool :: AppState -> IO ()
flushPool AppState{..} = SQL.release statePool
-- | Destroy the pool on shutdown.
destroyPool :: AppState -> IO ()
destroyPool AppState{..} = SQL.release statePool
getPgVersion :: AppState -> IO PgVersion
getPgVersion = readIORef . statePgVersion
putPgVersion :: AppState -> PgVersion -> IO ()
putPgVersion = atomicWriteIORef . statePgVersion
getSchemaCache :: AppState -> IO (Maybe SchemaCache)
getSchemaCache = readIORef . stateSchemaCache
putSchemaCache :: AppState -> Maybe SchemaCache -> IO ()
putSchemaCache appState = atomicWriteIORef (stateSchemaCache appState)
schemaCacheLoader :: AppState -> IO ()
schemaCacheLoader = debouncedSCacheLoader
getNextDelay :: AppState -> IO Int
getNextDelay = readIORef . stateNextDelay
getNextListenerDelay :: AppState -> IO Int
getNextListenerDelay = readIORef . stateNextListenerDelay
putNextListenerDelay :: AppState -> Int -> IO ()
putNextListenerDelay = atomicWriteIORef . stateNextListenerDelay
getConfig :: AppState -> IO AppConfig
getConfig = readIORef . stateConf
putConfig :: AppState -> AppConfig -> IO ()
putConfig = atomicWriteIORef . stateConf
getTime :: AppState -> IO UTCTime
getTime = stateGetTime
getJwtCacheState :: AppState -> JwtCacheState
getJwtCacheState = stateJwtCache
getSocketREST :: AppState -> NS.Socket
getSocketREST = stateSocketREST
getSocketAdmin :: AppState -> Maybe NS.Socket
getSocketAdmin = stateSocketAdmin
getMainThreadId :: AppState -> ThreadId
getMainThreadId = stateMainThreadId
isConnEstablished :: AppState -> IO Bool
isConnEstablished appState = do
AppConfig{..} <- getConfig appState
if configDbChannelEnabled then -- if the listener is enabled, we can be sure the connection is up
readIORef $ stateIsListenerOn appState
else -- otherwise the only way to check the connection is to make a query
isRight <$> usePool appState (SQL.sql "SELECT 1")
putIsListenerOn :: AppState -> Bool -> IO ()
putIsListenerOn = atomicWriteIORef . stateIsListenerOn
isLoaded :: AppState -> IO Bool
isLoaded x = do
scacheStatus <- readIORef $ stateSCacheStatus x
connEstablished <- isConnEstablished x
return $ scacheStatus == SCLoaded && connEstablished
isPending :: AppState -> IO Bool
isPending x = do
scacheStatus <- readIORef $ stateSCacheStatus x
connEstablished <- isConnEstablished x
return $ scacheStatus == SCPending || not connEstablished
putSCacheStatus :: AppState -> SchemaCacheStatus -> IO ()
putSCacheStatus = atomicWriteIORef . stateSCacheStatus
getObserver :: AppState -> ObservationHandler
getObserver = stateObserver
-- | Try to load the schema cache and retry if it fails.
--
-- This is done by repeatedly: 1) flushing the pool, 2) querying the version and validating that the postgres version is supported by us, and 3) loading the schema cache.
-- It's necessary to flush the pool:
--
-- + Because connections cache the pg catalog(see #2620)
-- + For rapid recovery. Otherwise, the pool idle or lifetime timeout would have to be reached for new healthy connections to be acquired.
retryingSchemaCacheLoad :: AppState -> IO ()
retryingSchemaCacheLoad appState@AppState{stateObserver=observer, stateMainThreadId=mainThreadId} =
void $ retrying retryPolicy shouldRetry (\RetryStatus{rsIterNumber, rsPreviousDelay} -> do
when (rsIterNumber > 0) $ do
let delay = fromMaybe 0 rsPreviousDelay `div` oneSecondInUs
observer $ ConnectionRetryObs delay
putNextListenerDelay appState delay
flushPool appState
(,) <$> qPgVersion <*> (qInDbConfig *> qSchemaCache)
)
where
qPgVersion :: IO (Maybe PgVersion)
qPgVersion = do
AppConfig{..} <- getConfig appState
pgVersion <- usePool appState (queryPgVersion False) -- No need to prepare the query here, as the connection might not be established
case pgVersion of
Left e -> do
observer $ QueryPgVersionError e
unless configDbPoolAutomaticRecovery $ do
observer ExitDBNoRecoveryObs
killThread mainThreadId
return Nothing
Right actualPgVersion ->
if actualPgVersion < minimumPgVersion then do
observer $ ExitUnsupportedPgVersion actualPgVersion minimumPgVersion
killThread mainThreadId
return Nothing
else do
observer $ DBConnectedObs $ pgvFullName actualPgVersion
observer $ PoolInit configDbPoolSize
putPgVersion appState actualPgVersion
return $ Just actualPgVersion
qInDbConfig :: IO ()
qInDbConfig = do
AppConfig{..} <- getConfig appState
when configDbConfig $ readInDbConfig False appState
qSchemaCache :: IO (Maybe SchemaCache)
qSchemaCache = do
conf@AppConfig{..} <- getConfig appState
(resultTime, result) <-
let transaction = if configDbPreparedStatements then SQL.transaction else SQL.unpreparedTransaction in
timeItT $ usePool appState (transaction SQL.ReadCommitted SQL.Read $ querySchemaCache conf)
case result of
Left e -> do
putSCacheStatus appState SCPending
putSchemaCache appState Nothing
observer $ SchemaCacheErrorObs configDbSchemas configDbExtraSearchPath e
return Nothing
Right sCache -> do
-- IMPORTANT: While the pending schema cache state starts from running the above querySchemaCache, only at this stage we block API requests due to the usage of an
-- IORef on putSchemaCache. This is why SCacheStatus is put at SCPending here to signal the Admin server (using isPending) that we're on a recovery state.
putSCacheStatus appState SCPending
putSchemaCache appState $ Just sCache
observer $ SchemaCacheQueriedObs resultTime
(t, _) <- timeItT $ observer $ SchemaCacheSummaryObs $ showSummary sCache
observer $ SchemaCacheLoadedObs t
putSCacheStatus appState SCLoaded
return $ Just sCache
shouldRetry :: RetryStatus -> (Maybe PgVersion, Maybe SchemaCache) -> IO Bool
shouldRetry _ (pgVer, sCache) = do
AppConfig{..} <- getConfig appState
let itShould = configDbPoolAutomaticRecovery && (isNothing pgVer || isNothing sCache)
return itShould
retryPolicy :: RetryPolicy
retryPolicy =
let delayMicroseconds = 32*oneSecondInUs {-32 seconds-} in
capDelay delayMicroseconds $ exponentialBackoff oneSecondInUs
oneSecondInUs = 1000000 -- one second in microseconds
-- | Reads the in-db config and reads the config file again
-- | We don't retry reading the in-db config after it fails immediately, because it could have user errors. We just report the error and continue.
readInDbConfig :: Bool -> AppState -> IO ()
readInDbConfig startingUp appState@AppState{stateObserver=observer} = do
conf <- getConfig appState
pgVer <- getPgVersion appState
dbSettings <-
if configDbConfig conf then do
qDbSettings <- usePool appState (queryDbSettings (quoteQi <$> configDbPreConfig conf) (configDbPreparedStatements conf))
case qDbSettings of
Left e -> do
observer $ ConfigReadErrorObs e
pure mempty
Right x -> pure x
else
pure mempty
(roleSettings, roleIsolationLvl) <-
if configDbConfig conf then do
rSettings <- usePool appState (queryRoleSettings pgVer (configDbPreparedStatements conf))
case rSettings of
Left e -> do
observer $ QueryRoleSettingsErrorObs e
pure (mempty, mempty)
Right x -> pure x
else
pure mempty
readAppConfig dbSettings (configFilePath conf) (Just $ configDbUri conf) roleSettings roleIsolationLvl >>= \case
Left err ->
if startingUp then
panic err -- die on invalid config if the program is starting up
else
observer $ ConfigInvalidObs err
Right newConf -> do
putConfig appState newConf
-- After the config has reloaded, jwt-secret might have changed, so
-- if it has changed, it is important to invalidate the jwt cache
-- entries, because they were cached using the old secret
update (getJwtCacheState appState) newConf
if startingUp then
pass
else
observer ConfigSucceededObs
-77
View File
@@ -1,77 +0,0 @@
{-# LANGUAGE RecordWildCards #-}
{-|
Module : PostgREST.Auth
Description : PostgREST authentication functions.
This module provides functions to deal with the JWT authentication (http://jwt.io).
It also can be used to define other authentication functions,
in the future Oauth, LDAP and similar integrations can be coded here.
Authentication should always be implemented in an external service.
In the test suite there is an example of simple login function that can be used for a
very simple authentication system inside the PostgreSQL database.
-}
module PostgREST.Auth
( getResult
, getJwtDur
, getRole
, middleware
) where
import qualified Data.ByteString as BS
import qualified Data.Vault.Lazy as Vault
import qualified Network.HTTP.Types.Header as HTTP
import qualified Network.Wai as Wai
import qualified Network.Wai.Middleware.HttpAuth as Wai
import Data.List (lookup)
import PostgREST.TimeIt (timeItT)
import System.IO.Unsafe (unsafePerformIO)
import PostgREST.AppState (AppState, getConfig, getJwtCacheState,
getTime)
import PostgREST.Auth.Jwt (parseClaims)
import PostgREST.Auth.JwtCache (lookupJwtCache)
import PostgREST.Auth.Types (AuthResult (..))
import PostgREST.Config (AppConfig (..))
import PostgREST.Error (Error (..))
import Protolude
-- | Validate authorization header
-- Parse and store JWT claims for future use in the request.
middleware :: AppState -> Wai.Middleware
middleware appState app req respond = do
conf@AppConfig{..} <- getConfig appState
time <- getTime appState
let token = Wai.extractBearerAuth =<< lookup HTTP.hAuthorization (Wai.requestHeaders req)
parseJwt = runExceptT $ lookupJwtCache jwtCacheState token >>= parseClaims conf time
jwtCacheState = getJwtCacheState appState
-- If ServerTimingEnabled -> calculate JWT validation time
req' <- if configServerTimingEnabled then do
(dur, authResult) <- timeItT parseJwt
pure $ req { Wai.vault = Wai.vault req & Vault.insert authResultKey authResult & Vault.insert jwtDurKey dur }
else do
authResult <- parseJwt
pure $ req { Wai.vault = Wai.vault req & Vault.insert authResultKey authResult }
app req' respond
authResultKey :: Vault.Key (Either Error AuthResult)
authResultKey = unsafePerformIO Vault.newKey
{-# NOINLINE authResultKey #-}
getResult :: Wai.Request -> Maybe (Either Error AuthResult)
getResult = Vault.lookup authResultKey . Wai.vault
jwtDurKey :: Vault.Key Double
jwtDurKey = unsafePerformIO Vault.newKey
{-# NOINLINE jwtDurKey #-}
getJwtDur :: Wai.Request -> Maybe Double
getJwtDur = Vault.lookup jwtDurKey . Wai.vault
getRole :: Wai.Request -> Maybe BS.ByteString
getRole req = authRole <$> (rightToMaybe =<< getResult req)
-100
View File
@@ -1,100 +0,0 @@
{-# OPTIONS_GHC -Wno-unused-do-bind #-}
module PostgREST.Config.JSPath
( JSPath
, JSPathExp(..)
, FilterExp(..)
, dumpJSPath
, pRoleClaimKey
) where
import qualified Text.ParserCombinators.Parsec as P
import Data.Either.Combinators (mapLeft)
import Text.ParserCombinators.Parsec ((<?>))
import Text.Read (read)
import Protolude
-- | full jspath, e.g. .property[0].attr.detail[?(@ == "role1")]
type JSPath = [JSPathExp]
-- NOTE: We only accept one JSPFilter expr (at the end of input)
-- | jspath expression
data JSPathExp
= JSPKey Text -- .property or ."property-dash"
| JSPIdx Int -- [0]
| JSPFilter FilterExp -- [?(@ == "match")]
data FilterExp
= EqualsCond Text
| NotEqualsCond Text
| StartsWithCond Text
| EndsWithCond Text
| ContainsCond Text
dumpJSPath :: JSPathExp -> Text
-- TODO: this needs to be quoted properly for special chars
dumpJSPath (JSPKey k) = "." <> show k
dumpJSPath (JSPIdx i) = "[" <> show i <> "]"
dumpJSPath (JSPFilter cond) = "[?(@" <> expr <> ")]"
where
expr =
case cond of
EqualsCond text -> " == " <> show text
NotEqualsCond text -> " != " <> show text
StartsWithCond text -> " ^== " <> show text
EndsWithCond text -> " ==^ " <> show text
ContainsCond text -> " *== " <> show text
-- Used for the config value "role-claim-key"
pRoleClaimKey :: Text -> Either Text JSPath
pRoleClaimKey selStr =
mapLeft show $ P.parse pJSPath ("failed to parse role-claim-key value (" <> toS selStr <> ")") (toS selStr)
pJSPath :: P.Parser JSPath
pJSPath = P.many1 pJSPathExp <* P.eof
pJSPathExp :: P.Parser JSPathExp
pJSPathExp = pJSPKey <|> pJSPFilter <|> pJSPIdx
pJSPKey :: P.Parser JSPathExp
pJSPKey = do
P.char '.'
val <- toS <$> P.many1 (P.alphaNum <|> P.oneOf "_$@") <|> pQuotedValue
return (JSPKey val) <?> "pJSPKey: JSPath attribute key"
pJSPIdx :: P.Parser JSPathExp
pJSPIdx = do
P.char '['
num <- read <$> P.many1 P.digit
P.char ']'
return (JSPIdx num) <?> "pJSPIdx: JSPath array index"
pJSPFilter :: P.Parser JSPathExp
pJSPFilter = do
P.try $ P.string "[?("
condition <- pFilterConditionParser
P.char ')'
P.char ']'
P.eof -- this should be the last jspath expression
return (JSPFilter condition) <?> "pJSPFilter: JSPath filter exp"
pFilterConditionParser :: P.Parser FilterExp
pFilterConditionParser = do
P.char '@'
P.spaces
filt <- matchOperator
P.spaces
filt <$> pQuotedValue
where
matchOperator =
P.try (P.string "==^" $> EndsWithCond)
<|> P.try (P.string "==" $> EqualsCond)
<|> P.try (P.string "!=" $> NotEqualsCond)
<|> P.try (P.string "^==" $> StartsWithCond)
<|> P.try (P.string "*==" $> ContainsCond)
pQuotedValue :: P.Parser Text
pQuotedValue = toS <$> (P.char '"' *> P.many (P.noneOf "\"") <* P.char '"')
-81
View File
@@ -1,81 +0,0 @@
{-# LANGUAGE MultiWayIf #-}
{-# LANGUAGE RecordWildCards #-}
module PostgREST.Listener (runListener) where
import qualified Data.ByteString.Char8 as BS
import qualified Hasql.Connection as SQL
import qualified Hasql.Notifications as SQL
import PostgREST.AppState (AppState, getConfig)
import PostgREST.Config (AppConfig (..))
import PostgREST.Observation (Observation (..))
import PostgREST.Version (prettyVersion)
import qualified PostgREST.AppState as AppState
import qualified PostgREST.Config as Config
import Protolude
-- | Starts the Listener in a thread
runListener :: AppState -> IO ()
runListener appState = do
AppConfig{..} <- getConfig appState
when configDbChannelEnabled $
void . forkIO $ retryingListen appState
-- | Starts a LISTEN connection and handles notifications. It recovers with exponential backoff with a cap of 32 seconds, if the LISTEN connection is lost.
retryingListen :: AppState -> IO ()
retryingListen appState = do
AppConfig{..} <- AppState.getConfig appState
let
dbChannel = toS configDbChannel
handleFinally err = do
AppState.putIsListenerOn appState False
observer $ DBListenFail dbChannel (Right err)
unless configDbPoolAutomaticRecovery $
killThread mainThreadId
-- retry the listener
delay <- AppState.getNextListenerDelay appState
observer $ DBListenRetry delay
threadDelay (delay * oneSecondInMicro)
unless (delay == maxDelay) $
AppState.putNextListenerDelay appState (delay * 2)
retryingListen appState
-- forkFinally allows to detect if the thread dies
void . flip forkFinally handleFinally $ do
dbOrError <- SQL.acquire $ toUtf8 (Config.addTargetSessionAttrs $ Config.addFallbackAppName prettyVersion configDbUri)
case dbOrError of
Right db -> do
SQL.listen db $ SQL.toPgIdentifier dbChannel
AppState.putIsListenerOn appState True
delay <- AppState.getNextListenerDelay appState
when (delay > 1) $ do -- if we did a retry
-- assume we lost notifications, refresh the schema cache
AppState.schemaCacheLoader appState
-- reset the delay
AppState.putNextListenerDelay appState 1
observer $ DBListenStart dbChannel
SQL.waitForNotifications handleNotification db
Left err -> do
observer $ DBListenFail dbChannel (Left err)
exitFailure
where
observer = AppState.getObserver appState
mainThreadId = AppState.getMainThreadId appState
oneSecondInMicro = 1000000
maxDelay = 32
handleNotification channel msg =
if | BS.null msg -> observer (DBListenerGotSCacheMsg channel) >> cacheReloader
| msg == "reload schema" -> observer (DBListenerGotSCacheMsg channel) >> cacheReloader
| msg == "reload config" -> observer (DBListenerGotConfigMsg channel) >> AppState.readInDbConfig False appState
| otherwise -> pure () -- Do nothing if anything else than an empty message is sent
cacheReloader =
AppState.schemaCacheLoader appState
-142
View File
@@ -1,142 +0,0 @@
{-# LANGUAGE RecordWildCards #-}
{-|
Module : PostgREST.Logger
Description : Logging based on the Observation.hs module. Access logs get sent to stdout and server diagnostic get sent to stderr.
-}
-- TODO log with buffering enabled to not lose throughput on logging levels higher than LogError
module PostgREST.Logger
( middleware
, observationLogger
, init
, LoggerState
) where
import Control.AutoUpdate (defaultUpdateSettings,
mkAutoUpdate,
updateAction)
import Control.Debounce
import qualified Data.ByteString.Char8 as BS
import qualified Data.Text.Encoding as T
import qualified Hasql.Decoders as HD
import qualified Hasql.DynamicStatements.Snippet as SQL hiding (sql)
import qualified Hasql.DynamicStatements.Statement as SQL
import qualified Hasql.Statement as SQL
import Data.Time (ZonedTime, defaultTimeLocale, formatTime,
getZonedTime)
import qualified Network.Wai as Wai
import qualified Network.Wai.Middleware.RequestLogger as Wai
import Network.HTTP.Types.Status (Status, status400, status500)
import System.IO.Unsafe (unsafePerformIO)
import PostgREST.Config (LogLevel (..))
import PostgREST.Observation
import PostgREST.Query (MainQuery (..))
import Protolude
data LoggerState = LoggerState
{ stateGetZTime :: IO ZonedTime -- ^ Time with time zone used for logs
, stateLogDebouncePoolTimeout :: MVar (IO ()) -- ^ Logs with a debounce
}
init :: IO LoggerState
init = do
zTime <- mkAutoUpdate defaultUpdateSettings { updateAction = getZonedTime }
LoggerState zTime <$> newEmptyMVar
logWithDebounce :: LoggerState -> IO () -> IO ()
logWithDebounce loggerState action = do
debouncer <- tryReadMVar $ stateLogDebouncePoolTimeout loggerState
case debouncer of
Just d -> d
Nothing -> do
newDebouncer <-
let oneSecond = 1000000 in
mkDebounce defaultDebounceSettings
{ debounceAction = action
, debounceFreq = 5*oneSecond
, debounceEdge = leadingEdge -- logs at the start and the end
}
putMVar (stateLogDebouncePoolTimeout loggerState) newDebouncer
newDebouncer
-- TODO stop using this middleware to reuse the same "observer" pattern for all our logs
middleware :: LogLevel -> (Wai.Request -> Maybe BS.ByteString) -> Wai.Middleware
middleware logLevel getAuthRole =
unsafePerformIO $
Wai.mkRequestLogger Wai.defaultRequestLoggerSettings
{ Wai.outputFormat =
Wai.ApacheWithSettings $
Wai.defaultApacheSettings &
Wai.setApacheRequestFilter (\_ res -> shouldLogResponse logLevel $ Wai.responseStatus res) &
Wai.setApacheUserGetter getAuthRole
, Wai.autoFlush = True
, Wai.destination = Wai.Handle stdout
}
shouldLogResponse :: LogLevel -> Status -> Bool
shouldLogResponse logLevel = case logLevel of
LogCrit -> const False
LogError -> (>= status500)
LogWarn -> (>= status400)
LogInfo -> const True
LogDebug -> const True
-- All observations are logged except some that depend on the log-level
observationLogger :: LoggerState -> LogLevel -> ObservationHandler
observationLogger loggerState logLevel obs = case obs of
o@(PoolAcqTimeoutObs _) -> do
when (logLevel >= LogError) $ do
logWithDebounce loggerState $
logWithZTime loggerState $ observationMessage o
o@(QueryErrorCodeHighObs _) -> do
when (logLevel >= LogError) $ do
logWithZTime loggerState $ observationMessage o
o@SchemaCacheEmptyObs ->
when (logLevel >= LogError) $ do
logWithZTime loggerState $ observationMessage o
o@(HasqlPoolObs _) -> do
when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o
QueryObs gq status -> do
when (shouldLogResponse logLevel status) $
logMainQ loggerState gq
o@PoolRequest ->
when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o
o@PoolRequestFullfilled ->
when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o
o@JwtCacheEviction ->
when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o
o@(JwtCacheLookup _) ->
when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o
o ->
logWithZTime loggerState $ observationMessage o
logWithZTime :: LoggerState -> Text -> IO ()
logWithZTime loggerState txt = do
zTime <- stateGetZTime loggerState
hPutStrLn stderr $ toS (formatTime defaultTimeLocale "%d/%b/%Y:%T %z: " zTime) <> txt
logMainQ :: LoggerState -> MainQuery -> IO ()
logMainQ loggerState MainQuery{mqOpenAPI=(x, y, z),..} =
let snipts = renderSnippet <$> [mqTxVars, fromMaybe mempty mqPreReq, mqMain, x, y, z, fromMaybe mempty mqExplain]
-- Does not log SQL when it's empty (happens on OPTIONS requests and when the openapi queries are not generated)
logQ q = when (q /= mempty) $ logWithZTime loggerState $ showOnSingleLine '\n' $ T.decodeUtf8 q in
mapM_ logQ snipts
-- TODO: maybe patch upstream hasql-dynamic-statements so we have a less hackish way to convert
-- the SQL.Snippet or maybe don't use hasql-dynamic-statements and resort to plain strings for the queries and use regular hasql
renderSnippet :: SQL.Snippet -> ByteString
renderSnippet snippet =
let SQL.Statement sql _ _ _ = SQL.dynamicallyParameterized snippet decoder prepared
decoder = HD.noResult -- unused
prepared = False -- unused
in
sql
-79
View File
@@ -1,79 +0,0 @@
{-# LANGUAGE RecordWildCards #-}
{-|
Module : PostgREST.Logger
Description : Metrics based on the Observation module. See Observation.hs.
-}
module PostgREST.Metrics
( init
, MetricsState (..)
, observationMetrics
, metricsToText
) where
import qualified Data.ByteString.Lazy as LBS
import qualified Hasql.Pool.Observation as SQL
import Prometheus
import PostgREST.Observation
import Protolude
data MetricsState =
MetricsState {
poolTimeouts :: Counter,
poolAvailable :: Gauge,
poolWaiting :: Gauge,
poolMaxSize :: Gauge,
schemaCacheLoads :: Vector Label1 Counter,
schemaCacheQueryTime :: Gauge,
jwtCacheRequests :: Counter,
jwtCacheHits :: Counter,
jwtCacheEvictions :: Counter
}
init :: Int -> IO MetricsState
init configDbPoolSize = do
metricState <- MetricsState <$>
register (counter (Info "pgrst_db_pool_timeouts_total" "The total number of pool connection timeouts")) <*>
register (gauge (Info "pgrst_db_pool_available" "Available connections in the pool")) <*>
register (gauge (Info "pgrst_db_pool_waiting" "Requests waiting to acquire a pool connection")) <*>
register (gauge (Info "pgrst_db_pool_max" "Max pool connections")) <*>
register (vector "status" $ counter (Info "pgrst_schema_cache_loads_total" "The total number of times the schema cache was loaded")) <*>
register (gauge (Info "pgrst_schema_cache_query_time_seconds" "The query time in seconds of the last schema cache load")) <*>
register (counter (Info "pgrst_jwt_cache_requests_total" "The total number of JWT cache lookups")) <*>
register (counter (Info "pgrst_jwt_cache_hits_total" "The total number of JWT cache hits")) <*>
register (counter (Info "pgrst_jwt_cache_evictions_total" "The total number of JWT cache evictions"))
setGauge (poolMaxSize metricState) (fromIntegral configDbPoolSize)
pure metricState
-- Only some observations are used as metrics
observationMetrics :: MetricsState -> ObservationHandler
observationMetrics MetricsState{..} obs = case obs of
(PoolAcqTimeoutObs _) -> do
incCounter poolTimeouts
(HasqlPoolObs (SQL.ConnectionObservation _ status)) -> case status of
SQL.ReadyForUseConnectionStatus -> do
incGauge poolAvailable
SQL.InUseConnectionStatus -> do
decGauge poolAvailable
SQL.TerminatedConnectionStatus _ -> do
decGauge poolAvailable
SQL.ConnectingConnectionStatus -> pure ()
PoolRequest ->
incGauge poolWaiting
PoolRequestFullfilled ->
decGauge poolWaiting
SchemaCacheLoadedObs resTime -> do
withLabel schemaCacheLoads "SUCCESS" incCounter
setGauge schemaCacheQueryTime resTime
SchemaCacheErrorObs{} -> do
withLabel schemaCacheLoads "FAIL" incCounter
JwtCacheLookup True -> incCounter jwtCacheRequests *> incCounter jwtCacheHits
JwtCacheLookup False -> incCounter jwtCacheRequests
JwtCacheEviction -> incCounter jwtCacheEvictions
_ ->
pure ()
metricsToText :: IO LBS.ByteString
metricsToText = exportMetricsAsText
-173
View File
@@ -1,173 +0,0 @@
{-# LANGUAGE LambdaCase #-}
{-|
Module : PostgREST.Observation
Description : This module holds an Observation type which is the core of Observability for PostgREST.
The Observation and ObservationHandler (the observer) are abstractions that allow centralizing logging and metrics concerns,
only observer calls with an Observation constructor are applied at different parts in the codebase.
The Logger and Metrics modules then decide which observations to expose. Not all observations need to be logged nor all correspond to a metric.
-}
module PostgREST.Observation
( Observation(..)
, ObsFatalError(..)
, observationMessage
, ObservationHandler
, showOnSingleLine
) where
import qualified Data.ByteString.Lazy as LBS
import Data.List.NonEmpty (toList)
import qualified Data.Text as T
import qualified Data.Text.Encoding as T
import qualified Hasql.Connection as SQL
import qualified Hasql.Pool as SQL
import qualified Hasql.Pool.Observation as SQL
import Network.HTTP.Types.Status (Status)
import Numeric (showFFloat)
import PostgREST.Config.PgVersion
import qualified PostgREST.Error as Error
import PostgREST.Query (MainQuery)
import Protolude hiding (toList)
data Observation
= AdminStartObs Text
| AppStartObs ByteString
| AppServerAddressObs Text
| ExitUnsupportedPgVersion PgVersion PgVersion
| ExitDBNoRecoveryObs
| ExitDBFatalError ObsFatalError SQL.UsageError
| DBConnectedObs Text
| SchemaCacheEmptyObs
| SchemaCacheErrorObs (NonEmpty Text) [Text] SQL.UsageError
| SchemaCacheQueriedObs Double
| SchemaCacheSummaryObs Text
| SchemaCacheLoadedObs Double
| ConnectionRetryObs Int
| DBListenStart Text
| DBListenFail Text (Either SQL.ConnectionError (Either SomeException ()))
| DBListenRetry Int
| DBListenerGotSCacheMsg ByteString
| DBListenerGotConfigMsg ByteString
| QueryObs MainQuery Status
| ConfigReadErrorObs SQL.UsageError
| ConfigInvalidObs Text
| ConfigSucceededObs
| QueryRoleSettingsErrorObs SQL.UsageError
| QueryErrorCodeHighObs SQL.UsageError
| QueryPgVersionError SQL.UsageError
| PoolInit Int
| PoolAcqTimeoutObs SQL.UsageError
| HasqlPoolObs SQL.Observation
| PoolRequest
| PoolRequestFullfilled
| JwtCacheLookup Bool
| JwtCacheEviction
data ObsFatalError = ServerAuthError | ServerPgrstBug | ServerError42P05 | ServerError08P01
type ObservationHandler = Observation -> IO ()
observationMessage :: Observation -> Text
observationMessage = \case
AdminStartObs address ->
"Admin server listening on " <> address
AppStartObs ver ->
"Starting PostgREST " <> T.decodeUtf8 ver <> "..."
AppServerAddressObs address ->
"API server listening on " <> address
DBConnectedObs ver ->
"Successfully connected to " <> ver
ExitUnsupportedPgVersion pgVer minPgVer ->
"Cannot run in this PostgreSQL version (" <> pgvName pgVer <> "), PostgREST needs at least " <> pgvName minPgVer
ExitDBNoRecoveryObs ->
"Automatic recovery disabled, exiting."
ExitDBFatalError ServerAuthError usageErr ->
"Failed to establish a connection. " <> jsonMessage usageErr
ExitDBFatalError ServerPgrstBug usageErr ->
"This is probably a bug in PostgREST, please report it at https://github.com/PostgREST/postgrest/issues. " <> jsonMessage usageErr
ExitDBFatalError ServerError42P05 usageErr ->
"If you are using connection poolers in transaction mode, try setting db-prepared-statements to false. " <> jsonMessage usageErr
ExitDBFatalError ServerError08P01 usageErr ->
"Connection poolers in statement mode are not supported." <> jsonMessage usageErr
SchemaCacheEmptyObs ->
T.decodeUtf8 . LBS.toStrict . Error.errorPayload $ Error.NoSchemaCacheError
SchemaCacheErrorObs dbSchemas extraPaths usageErr ->
"Failed to load the schema cache using "
<> "db-schemas=" <> T.intercalate "," (toList dbSchemas)
<> " and "
<> "db-extra-search-path=" <> T.intercalate "," extraPaths
<> ". " <> jsonMessage usageErr
SchemaCacheQueriedObs resultTime ->
"Schema cache queried in " <> showMillis resultTime <> " milliseconds"
SchemaCacheSummaryObs summary ->
"Schema cache loaded " <> summary
SchemaCacheLoadedObs resultTime ->
"Schema cache loaded in " <> showMillis resultTime <> " milliseconds"
ConnectionRetryObs delay ->
"Attempting to reconnect to the database in " <> (show delay::Text) <> " seconds..."
QueryPgVersionError usageErr ->
"Failed to query the PostgreSQL version. " <> jsonMessage usageErr
DBListenStart channel -> do
"Listening for database notifications on the " <> show channel <> " channel"
DBListenFail channel listenErr ->
"Failed listening for database notifications on the " <> show channel <> " channel. " <>
either showListenerConnError showListenerException listenErr
DBListenRetry delay ->
"Retrying listening for database notifications in " <> (show delay::Text) <> " seconds..."
DBListenerGotSCacheMsg channel ->
"Received a schema cache reload message on the " <> show channel <> " channel"
DBListenerGotConfigMsg channel ->
"Received a config reload message on the " <> show channel <> " channel"
QueryObs{} ->
mempty -- TODO pending refactor: The logic for printing the query cannot be done here. Join the observationMessage function into observationLogger to avoid this mempty.
ConfigReadErrorObs usageErr ->
"Failed to query database settings for the config parameters." <> jsonMessage usageErr
QueryRoleSettingsErrorObs usageErr ->
"Failed to query the role settings. " <> jsonMessage usageErr
QueryErrorCodeHighObs usageErr ->
jsonMessage usageErr
ConfigInvalidObs err ->
"Failed reloading config: " <> err
ConfigSucceededObs ->
"Config reloaded"
PoolInit poolSize ->
"Connection Pool initialized with a maximum size of " <> show poolSize <> " connections"
PoolAcqTimeoutObs usageErr ->
jsonMessage usageErr
HasqlPoolObs (SQL.ConnectionObservation uuid status) ->
"Connection " <> show uuid <> (
case status of
SQL.ConnectingConnectionStatus -> " is being established"
SQL.ReadyForUseConnectionStatus -> " is available"
SQL.InUseConnectionStatus -> " is used"
SQL.TerminatedConnectionStatus reason -> " is terminated due to " <> case reason of
SQL.AgingConnectionTerminationReason -> "max lifetime"
SQL.IdlenessConnectionTerminationReason -> "max idletime"
SQL.ReleaseConnectionTerminationReason -> "release"
SQL.NetworkErrorConnectionTerminationReason _ -> "network error" -- usage error is already logged, no need to repeat the same message.
)
PoolRequest ->
"Trying to borrow a connection from pool"
PoolRequestFullfilled ->
"Borrowed a connection from the pool"
JwtCacheLookup _ ->
"Looked up a JWT in JWT cache"
JwtCacheEviction ->
"Evicted entry from JWT cache"
where
showMillis :: Double -> Text
showMillis x = toS $ showFFloat (Just 1) x ""
jsonMessage err = T.decodeUtf8 . LBS.toStrict . Error.errorPayload $ Error.PgError False err
showListenerConnError :: SQL.ConnectionError -> Text
showListenerConnError = maybe "Connection error" (showOnSingleLine '\t' . T.decodeUtf8)
showListenerException :: Either SomeException () -> Text
showListenerException (Right _) = "Failed getting notifications" -- should not happen as the listener will never finish (hasql-notifications uses `forever` internally) with a Right result
showListenerException (Left e) = showOnSingleLine '\t' $ show e
showOnSingleLine :: Char -> Text -> Text
showOnSingleLine split txt = T.intercalate " " $ T.filter (/= split) <$> T.lines txt -- the errors from hasql-notifications come intercalated with "\t\n"
-53
View File
@@ -1,53 +0,0 @@
module PostgREST.Plan.ReadPlan
( ReadPlanTree
, ReadPlan(..)
, JoinCondition(..)
, SpreadType(..)
) where
import Data.Tree (Tree (..))
import PostgREST.ApiRequest.Types (Alias, Depth, Hint,
JoinType, NodeName)
import PostgREST.Plan.Types (CoercibleLogicTree,
CoercibleOrderTerm,
CoercibleSelectField (..),
RelSelectField (..),
SpreadType (..))
import PostgREST.RangeQuery (NonnegRange)
import PostgREST.SchemaCache.Identifiers (FieldName,
QualifiedIdentifier)
import PostgREST.SchemaCache.Relationship (Relationship)
import Protolude
type ReadPlanTree = Tree ReadPlan
data JoinCondition =
JoinCondition
(QualifiedIdentifier, FieldName)
(QualifiedIdentifier, FieldName)
deriving (Eq, Show)
-- TODO: Enforce uniqueness of columns by changing to a Set instead of a List where applicable
data ReadPlan = ReadPlan
{ select :: [CoercibleSelectField]
, from :: QualifiedIdentifier
, fromAlias :: Maybe Alias
, where_ :: [CoercibleLogicTree]
, order :: [CoercibleOrderTerm]
, range_ :: NonnegRange
, relName :: NodeName
, relToParent :: Maybe Relationship
, relJoinConds :: [JoinCondition]
, relAlias :: Maybe Alias
, relAggAlias :: Alias
, relHint :: Maybe Hint
, relJoinType :: Maybe JoinType
, relSpread :: Maybe SpreadType
, relSelect :: [RelSelectField]
, depth :: Depth
-- ^ used for aliasing
}
deriving (Eq, Show)

Some files were not shown because too many files have changed in this diff Show More