Compare commits

..
120 Commits
Author SHA1 Message Date
renovate[bot]andWolfgang Walther feb83a8519 chore(deps): update actions/checkout action to v7.0.1 2026-07-23 20:11:44 +00:00
ncikandWolfgang Walther 835afbd383 docs: fix example jwt expiration epoch
Issue: the example epoch date is wrapped in parentheses. This causes the server to return PGRST303: "The JWT 'exp' claim must be a number"

Fix: remove parentheses
(cherry picked from commit 69d21a82c9)
2026-07-02 10:57:21 +02:00
renovate[bot]andWolfgang Walther bcae4a366f chore(deps): update all dependencies 2026-06-19 19:07:40 +00:00
renovate[bot]andWolfgang Walther cf53fd5e1b chore(deps): update all dependencies to v6.0.3 2026-06-03 07:40:07 +00:00
renovate[bot]andWolfgang Walther 74db7b7185 chore(deps): update cachix/cachix-action digest to 5f2d7c5 2026-05-12 13:10:49 +00:00
Wolfgang Walther 2a7a62ecc0 docs: fix broken link 2026-04-15 09:30:17 +02:00
renovate[bot]andWolfgang Walther 422c2934a2 chore(deps): update cachix/cachix-action action to v17 2026-03-18 21:24:19 +00:00
Wolfgang Walther 565b644951 ci: remove unused Dockerfile
This was used by CI, which is disabled on this branch.

Removing to prevent useless auto-updates by renovate.
2026-03-17 11:19:09 +01:00
renovate[bot]andWolfgang Walther a4b1957981 chore(deps): update all dependencies 2026-03-13 14:06:25 +00:00
Wolfgang Walther bbce632fc2 ci: remove all CI except docs
We don't need to run full CI on the outdated and finished v13 branch
anymore. We did the same thing to other branches in the past.
2026-03-13 15:01:51 +01:00
Wolfgang Walther f6e44aa7e2 docs: fix prometheus text format link
Reported by linkcheck.
2026-03-11 09:45:36 +01:00
dshukertjrandWolfgang Walther 8e728596b9 docs: Update the Supabase logo to a correct one 2026-02-19 10:53:22 +01:00
Wolfgang Walther 54490c4d22 docs: fix links
datrium.com doesn't exist anymore, while euronodes.com seems to only
fail SSL in CI.
2026-01-28 09:57:34 +01:00
renovate[bot]andWolfgang Walther c1a1d3ae66 chore(deps): update haskell-actions/setup action to v2.10.2 2026-01-11 17:35:40 +00:00
renovate[bot]andWolfgang Walther e10b027b10 chore(deps): update haskell-actions/setup action to v2.10.1 2026-01-05 19:01:54 +00:00
renovate[bot]andWolfgang Walther fe64548aba chore(deps): update docker/setup-buildx-action action to v3.12.0 2025-12-20 20:24:42 +00:00
renovate[bot]andWolfgang Walther 577fa9446f chore(deps): update haskell-actions/setup action to v2.9.1 2025-12-17 18:01:20 +00:00
renovate[bot]andWolfgang Walther dd2eafe9cc chore(deps): update haskell-actions/setup action to v2.9.0 2025-12-15 21:01:30 +00:00
renovate[bot]andWolfgang Walther d1abfa0c89 chore(deps): update actions/cache action to v5.0.1 2025-12-13 17:12:46 +00:00
renovate[bot]andWolfgang Walther 627df4dfb4 chore(deps): update all dependencies 2025-12-13 17:12:15 +00:00
renovate[bot]andWolfgang Walther eb54461e73 chore(deps): update actions/cache action to v5 2025-12-12 08:29:08 +00:00
renovate[bot]andWolfgang Walther e84aa247fc chore(deps): update codecov/codecov-action action to v5.5.2 2025-12-10 12:24:30 +00:00
renovate[bot]andWolfgang Walther e232a7bd78 chore(deps): update actions/checkout action to v6 2025-12-03 13:52:39 +00:00
Wolfgang Walther aaf4bd8510 ci/test: remove macos x86 flake check
See previous commit.
2025-11-18 11:05:49 +01:00
renovate[bot]andWolfgang Walther 5997949ac4 chore(deps): update actions/checkout action to v5.0.1 2025-11-18 10:04:37 +00:00
Wolfgang Walther 5c527b50c2 ci: remove macos x86 builds
The x86 GitHub runner will not be available anymore, soon.

We might be able to re-introduce this, once we can build a static
executable via Nix on darwin, too.
2025-11-18 11:01:53 +01:00
renovate[bot]andWolfgang Walther 8bdef234c6 chore(deps): update ubuntu:noble docker digest to c35e29c 2025-11-15 13:52:14 +00:00
renovate[bot]andWolfgang Walther 07e23ff3c8 chore(deps): update all dependencies 2025-10-25 08:12:59 +00:00
steve-chavez f9e1af2fa5 bump version to 13.0.8 2025-10-24 13:43:50 -05:00
Taimoor ZaeemandWolfgang Walther c5af0cc3f9 fix: loading utf-8 config files with ascii locale set
Updates configurator-pg to version 0.2.11. This resolves #4386.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-10-24 17:58:09 +00:00
renovate[bot]andWolfgang Walther aa34503ce0 chore(deps): update ubuntu:noble docker digest to 66460d5 2025-10-13 10:35:46 +00:00
renovate[bot]andWolfgang Walther f1584c1e7e chore(deps): update ubuntu:noble docker digest to 59a458b 2025-10-10 07:33:40 +00:00
renovate[bot]andWolfgang Walther e02461d457 chore(deps): update ubuntu:noble docker digest to 728785b 2025-10-03 13:34:36 +00:00
renovate[bot]andWolfgang Walther 2d73644e71 chore(deps): update ubuntu:noble docker digest to fdb6c9c 2025-10-02 10:04:17 +00:00
renovate[bot]andWolfgang Walther a77d9d5b2d chore(deps): update all dependencies 2025-10-02 09:24:49 +00:00
renovate[bot]andWolfgang Walther e6a2551813 chore(deps): update all dependencies 2025-10-02 09:23:44 +00:00
renovate[bot]andWolfgang Walther 8ceffa0efc chore(deps): update nixbuild/nix-quick-install-action action to v34 2025-09-25 08:51:14 +00:00
renovate[bot]andWolfgang Walther 2f62bee183 chore(deps): update actions/cache action to v4.3.0 2025-09-25 08:22:00 +00:00
renovate[bot]andWolfgang Walther 308c2a789a chore(deps): update ubuntu:noble docker digest to 353675e 2025-09-17 10:32:14 +00:00
renovate[bot]andWolfgang Walther 151002a7e0 chore(deps): update haskell-actions/setup action to v2.8.2 2025-09-16 19:13:05 +00:00
renovate[bot]andWolfgang Walther 81ef374723 chore(deps): update ubuntu:noble docker digest to 590e57a 2025-09-16 09:36:58 +00:00
steve-chavez e30bc63f49 bump version to 13.0.7 2025-09-14 14:08:53 -05:00
Taimoor ZaeemandSteve Chavez a8f40c4908 feat: improve error details of PGRST301 error 2025-09-14 13:37:17 -05:00
Taimoor ZaeemandSteve Chavez 75972e9ffe refactor: move jwt error messages to Error.hs module 2025-09-14 13:37:17 -05:00
Laurence IslaandSteve Chavez 4ba6b1b30c feat: improve error response when the requested schema is invalid
- It now shows the invalid schema in the "message"
- The exposed schemas are now listed in the "hint" instead of the "message"
2025-09-14 13:37:17 -05:00
renovate[bot]andWolfgang Walther a4927141ee chore(deps): update codecov/codecov-action action to v5.5.1 2025-09-04 19:25:55 +00:00
renovate[bot]andWolfgang Walther 1b353590ff chore(deps): update ubuntu:noble docker digest to 9cbed75 2025-09-03 12:50:39 +00:00
renovate[bot]andWolfgang Walther 394aa026d3 chore(deps): update ubuntu:noble docker digest to f3b7f1b 2025-09-02 08:18:24 +00:00
steve-chavez 272e2e7535 bump version to 13.0.6 2025-08-30 13:55:34 -05:00
Laurence IslaandWolfgang Walther ea153523d1 fix: empty enum in 'preferParams' openAPI parameter 2025-08-30 08:31:21 +02:00
Taimoor ZaeemandWolfgang Walther bf0a1173b5 fix: format of IPv6 address logged at PostgREST startup
The IPv6 address logged at the startup like `::1:80` was
wrong because the port isn't clearly separated. This commit
corrects it, now logging as `[::1]:80`.

This is done in accordance to RFC 3986. In short, we did this
have a clear separation between the port and host because
the components of an IPv6 are separated with the ':' character.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-08-29 22:31:01 +02:00
renovate[bot]andWolfgang Walther 3a28968f3c chore(deps): update nixbuild/nix-quick-install-action action to v33 2025-08-25 15:08:25 +00:00
Laurence IslaandWolfgang Walther 86aac1ead5 fix: logging the Haskell type instead of the listener error message directly
Previously: Just "connection error..."
Now: connection error...
2025-08-25 10:25:48 +00:00
steve-chavez 1f1f40a3b9 bump version to 13.0.5 2025-08-24 12:29:22 -05:00
Taimoor ZaeemandWolfgang Walther 86b0f82b6c fix(admin): metrics endpoint not responding with Content-Type header
The prometheus metrics text format requires `Content-Type` header
for correct scraping which fails otherwise. Closes #4271.

Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
2025-08-21 13:47:19 +00:00
Taimoor ZaeemandWolfgang Walther d863065a51 fix: empty spread embeddings return unexpected SQL error
Fixes the SQL error from postgres when an empty spread embeddings
like `...table()` is requested.
2025-08-21 13:47:19 +00:00
renovate[bot]andWolfgang Walther 6ecacbc4cc chore(deps): update codecov/codecov-action action to v5.5.0 2025-08-20 18:10:51 +00:00
renovate[bot]andWolfgang Walther beb7d4f0e0 chore(deps): update ubuntu:noble docker digest to 7c06e91 2025-08-13 07:19:03 +00:00
renovate[bot]andWolfgang Walther b01d08d42f chore(deps): update actions/cache action to v4.2.4 2025-08-11 15:32:45 +00:00
renovate[bot]andWolfgang Walther 8628cb7cdf chore(deps): update actions/checkout action to v5 2025-08-11 15:31:50 +00:00
renovate[bot]andWolfgang Walther 0f2470c79d chore(deps): update actions/download-artifact action to v5 2025-08-06 07:44:53 +00:00
renovate[bot]andWolfgang Walther dc4e225b20 chore(deps): update docker/login-action action to v3.5.0 2025-08-04 17:11:07 +00:00
renovate[bot]andWolfgang Walther d201c5cac3 chore(deps): update haskell-actions/setup action to v2.8.1 2025-08-03 16:27:21 +00:00
Taimoor ZaeemandWolfgang Walther 16d59e825c test: adjust jwt claims error test to avoid failure
The JWT claims IO test fails too often. This breaks our
workflows. This commit adjusts the edge cases in test to
prevent flakiness.
2025-07-19 14:22:51 +02:00
renovate[bot]andWolfgang Walther dba6bda214 chore(deps): update ubuntu:noble docker digest to a08e551 2025-07-19 12:17:00 +00:00
Laurence Isla 9897ad2d9c chore: update sponsor 2025-07-17 11:56:34 -05:00
renovate[bot]andWolfgang Walther f82a11df49 chore(deps): update ubuntu:noble docker digest to c4570d2 2025-07-16 09:05:25 +00:00
Wolfgang Walther a87833cc10 docs: adjust some redirects
Those were reported in the weekly linkcheck.
2025-07-16 11:04:34 +02:00
renovate[bot]andWolfgang Walther a64e6fe87f chore(deps): update ubuntu:noble docker digest to e356c06 2025-07-16 08:51:04 +00:00
renovate[bot]andWolfgang Walther be4662e96e chore(deps): update all dependencies 2025-07-09 07:09:10 +00:00
Wolfgang Walther 97f9898e97 chore: bump some hackage dependencies
All of these were tested via stackage 23.27 which required allow-newer
for them.
2025-07-08 22:02:07 +02:00
Wolfgang Walther 0b058d6934 chore: fix stack's nix integration
The pkgconfig package has been renamed to pkg-config in... February
2019. So this has not been working for quite some time.
2025-07-08 20:29:24 +02:00
Wolfgang Walther 70057e65a5 chore: remove left-over comment for Ubuntu (arm)
We have been building with GHC 9.6 on that platform for a while.
2025-07-08 20:18:18 +02:00
Wolfgang Walther 5030c665be chore: build with GHC 9.8.4 for cabal 2025-07-08 20:18:17 +02:00
Wolfgang Walther ac6dac64b8 chore: update hackage index-state 2025-07-08 20:18:16 +02:00
Wolfgang Walther 2f8dbdb183 chore: stackage 22.41 -> 22.44
Updates stackage to 22.44, which is also supported on FreeBSD, where GHC
9.6.7 is available again.
2025-07-08 20:18:15 +02:00
steve-chavezandWolfgang Walther 67379f1d5e docs: clarify health checks empty response body 2025-07-08 20:18:13 +02:00
Wolfgang Walther d30abd99ae chore: remove Paths_postgrest module
The dependency on it was removed in #3608 already, but we forgot to
remove it from postgrest.cabal, which caused it to still be built.

We didn't realize because all references of it were stripped away by
dead code elimination anyway.
2025-07-08 18:37:14 +02:00
Joel JakobssonandWolfgang Walther 73d335e976 fix(openapi): respect function volatility for GET methods
The OpenAPI specification was incorrectly exposing GET methods for
VOLATILE functions, even though such functions properly reject GET
requests at runtime with "405 Method Not Allowed".  This created
a mismatch between the advertised API specification and the actual
runtime behavior.

VOLATILE functions should only be callable via POST since they may
have side effects, while STABLE and IMMUTABLE functions can safely
be called via GET since they don't modify database state.

Fix by checking the pdVolatility field in makeProcPathItem() and
only including GET methods in the OpenAPI PathItem for non-volatile
functions.

The runtime behavior was already correct; this fixes only the
OpenAPI documentation generation.
2025-07-07 17:28:07 +02:00
Taimoor ZaeemandWolfgang Walther 81b25871ae docs: horizontal filtering on table-valued functions 2025-07-05 21:09:56 +02:00
Taimoor ZaeemandWolfgang Walther 8f4a200f33 fix: OpenAPI broken docs link 2025-06-28 07:58:14 +00:00
Wolfgang Walther 087b9ecca1 ci: fix loadtest job on stable branches
Previously, the loadtest would always compare against main and the
latest tag. This meant a loadtest running on the v13 release branch,
would compare to a *future* version of both. This is not useful, and
also not supported by `postgrest-loadtest-against`, which recently
introduced a change on `main`, that now breaks the tests on the back
branches.

With this change, the loadtest will always run against the current
target branch of a PR, so against the v13 branch for a PR against v13,
for example. Also, it will compare against the latest released tag *for
that branch*.

Thus, when running this for v12, it will test against the v12 branch and
the v12.2.12 tag.
2025-06-26 09:44:01 +00:00
Wolfgang Walther 59eaae10ca ci: update Nix to 2.29.1
Related:
https://discourse.nixos.org/t/security-advisory-privilege-escalations-in-nix-lix-and-guix/66017
2025-06-25 12:58:46 +02:00
Laurence Isla a34d37bb82 bump version to 13.0.4 2025-06-17 19:58:43 -05:00
Taimoor ZaeemandLaurence Isla 5b45113565 fix: jwt-aud config not failing when set to invalid URI (#4140)
The `jwt-aud` config was not validated when containing ':'
character according to RFC 3986. This fix validates it and
fails at startup if it is invalid.
2025-06-18 00:11:30 +00:00
Laurence Isla 733a896113 fix: regression that makes fts not work on domain types based on tsvector 2025-06-18 00:11:30 +00:00
Laurence Isla fc06471f9d bump version to 13.0.3 2025-06-16 21:47:28 -05:00
Laurence Isla 6dcb0e0b02 fix: detect the correct base type of recursive domains in schema cache for tables and views
In OpenAPI it shows the correct base type in properties' definitions (including enums).
2025-06-16 18:23:00 -05:00
Taimoor ZaeemandLaurence Isla f54aef4795 fix: max-affected preference not failing for rpc with strict handling 2025-06-16 16:11:13 -05:00
Taimoor ZaeemandLaurence Isla ea2d3aeb72 test: add missing tests for max-affected preference with rpc 2025-06-16 16:11:03 -05:00
Laurence Isla 703fdd949f chore: update UTM tracking for Euronodes 2025-06-06 12:57:37 -05:00
Laurence Isla d942ea8438 chore: update sponsor 2025-06-06 09:56:37 -05:00
Laurence Isla 97b6022f5a bump version to 13.0.2 2025-06-02 14:18:26 -05:00
Laurence Isla c413833ec8 fix: regression that makes order by with nulls order not work alongside limits 2025-06-02 14:17:35 -05:00
steve-chavez dee7d6f39c bump version to 13.0.1 2025-06-01 08:10:43 -05:00
Thilo HohltandWolfgang Walther 6fb8077950 Update ecosystem.rst
The repository has been moved from a dedicated organisation to my personal profile, so this old link will no longer work after 90 days.
2025-05-31 13:50:41 +02:00
steve-chavez d4f82919f7 docs: external authentication page in explanations
- Move page from External JWT generation
2025-05-30 18:08:40 -05:00
steve-chavez 610a8be9c4 docs: move jwt using SSL to tutorial 1
Removes jwt.io example in favor of a bash script
2025-05-30 18:08:34 -05:00
Taimoor ZaeemandWolfgang Walther ba42e4610a fix: log db-schemas and db-extra-search-path in schema cache load error (#4108) 2025-05-30 20:56:04 +02:00
Laurence IslaandWolfgang Walther 8531c502d7 docs: JWK kid parameter validation 2025-05-30 14:35:58 +02:00
Taimoor ZaeemandWolfgang Walther 0ba47180ae fix: allow db-extra-search-path to accept empty value 2025-05-30 14:35:57 +02:00
Taimoor ZaeemandWolfgang Walther b77605e0d3 refactor: remove unused optValue function from Config.hs module 2025-05-30 14:35:54 +02:00
steve-chavez 800b32a59c docs: remove Greenplum integration
They're not really a sponsor, so it's not fair to include them.
2025-05-28 17:44:00 -05:00
Wolfgang Walther c48b6bc85b docs: fix functions link in api/preferences
External link syntax was used instead of internal reference.
2025-05-28 09:15:50 +02:00
Wolfgang Walther f899957675 docs: apply redirects
Those redirect, so we might as well hardcode the them.
2025-05-28 09:15:50 +02:00
Wolfgang Walther 609289d2bb docs: remove outdated "in production" links
Moat has been bought by Oracle. The advertising subpage redirects to
oracle.com, but pointing at that would be a bit misleading.

No need to keep failing links either.
2025-05-28 09:15:47 +02:00
steve-chavez 9e58946ec6 docs: update architecture HTTP link
It links directly to warp page, which is confusing. Link to the
same page reference instead, which finally links to warp.
2025-05-26 13:45:33 -05:00
Taimoor Zaeemandsteve-chavez 48a7b8dee7 docs: explain use of max-affected preference with rpc 2025-05-22 12:49:53 -05:00
Wolfgang Walther 23a4573a9c docs: Update sphinx-rtd-theme to 3.0.2 2025-05-22 08:07:54 +02:00
Taimoor ZaeemandWolfgang Walther 91814cd4f8 docs: add note in JWT Role Extraction section
Add a note describing that the used JSPath DSL does not
strictly follow the JSONPath as described in RFC 9535
2025-05-21 21:02:00 +02:00
steve-chavez 8ef5263b04 docs: add warning for duplicate keys in spread 2025-05-15 17:42:47 -05:00
steve-chavez a71f938a44 docs: clarify spread join table 2025-05-15 15:51:14 -05:00
steve-chavez e6d4bfa465 docs: clarify fts 2025-05-15 11:57:23 -05:00
steve-chavez b90d26034e docs: clarify spread feature 2025-05-15 11:17:59 -05:00
Taimoor ZaeemandWolfgang Walther 0230a844b2 test: add test for duplicate entries in pg_description with same OID 2025-05-14 21:44:23 +02:00
Taimoor ZaeemandWolfgang Walther e0e60fa433 fix: jwt error return status 400 for invalid role (#4081) 2025-05-14 21:44:22 +02:00
Taimoor ZaeemandWolfgang Walther 7269630538 test: add test when using .aud in jwt-role-claim-key 2025-05-14 21:44:20 +02:00
Laurence Isla cc2550a58b docs: fix link to SQL Query Logs 2025-05-09 20:59:31 -05:00
steve-chavez e20dc60e83 docs: jwt role extraction link to config
The feature section was missing a link to its config.

Also shorten the section name.
2025-05-09 20:47:30 -05:00
Wolfgang Walther 89fb2878df chore: adjust changelog for db-extra-search-path
Came up in #4073
2025-05-09 14:57:24 +02:00
195 changed files with 3727 additions and 7785 deletions
+42
View File
@@ -0,0 +1,42 @@
freebsd_instance:
image_family: freebsd-14-3
build_task:
# Don't change this name without adjusting .github/workflows/build.yaml
name: Build FreeBSD (Stack)
install_script: pkg install -y postgresql16-client hs-stack git
only_if: |
$CIRRUS_TAG != '' || $CIRRUS_BRANCH == 'main' || $CIRRUS_BRANCH =~ 'v*' ||
changesInclude(
'.github/workflows/build.yaml',
'.github/actions/artifact-from-cirrus/**',
'.cirrus.yml',
'postgrest.cabal',
'stack.yaml*',
'**.hs'
)
stack_cache:
folders: /.stack
fingerprint_script:
- echo $CIRRUS_OS
- stack --version
- md5sum postgrest.cabal
- md5sum stack.yaml.lock
stack_work_cache:
folders: .stack-work
fingerprint_script:
- echo $CIRRUS_OS
- stack --version
- md5sum postgrest.cabal
- md5sum stack.yaml.lock
- find main src -type f -iname '*.hs' -exec md5sum "{}" +
build_script: |
stack build -j 1 --local-bin-path . --copy-bins
strip postgrest
bin_artifacts:
path: postgrest
+1 -1
View File
@@ -21,7 +21,7 @@ For questions on how to use PostgREST, please use
### Reporting an Issue ### Reporting an Issue
* Make sure you test against the latest [stable release](https://github.com/PostgREST/postgrest/releases/latest) * Make sure you test against the latest [stable release](https://github.com/PostgREST/postgrest/releases/latest)
and also against the latest [devel release](https://github.com/PostgREST/postgrest/releases/tag/devel). and also against the latest [nightly release](https://github.com/PostgREST/postgrest/releases/tag/nightly).
It is possible we already fixed the bug you're experiencing. It is possible we already fixed the bug you're experiencing.
* Provide steps to reproduce the issue, including your OS version and * Provide steps to reproduce the issue, including your OS version and
+11 -12
View File
@@ -4,18 +4,17 @@ When submitting a new feature or fix:
- Add a new entry to the CHANGELOG - https://github.com/PostgREST/postgrest/blob/main/CHANGELOG.md#unreleased - Add a new entry to the CHANGELOG - https://github.com/PostgREST/postgrest/blob/main/CHANGELOG.md#unreleased
- If relevant, update the docs - If relevant, update the docs
- Use a prefix for the PR title or commits, e.g. "fix: description of the fix". - Use a prefix for the PR title or commits, e.g. "fix: description of the fix".
+ `add`, Add a new feature + `fix`, bug fixes
+ `amend`, To amend an unrealease commit + `feat`, new features added
+ `change`, Breaking changes + `perf`, performance improvements
+ `chore`, Maintenance, update sponsors, changelog, readme etc + `docs`, updating the documentation
+ `ci`, CI configuration files and scripts + `nix`, related to the Nix development environment
+ `docs`, Documentation + `ci`, related to the Continuous Integration modules
+ `fix`, Bug fix + `test`, related to the testing modules
+ `nix`, Related to Nix + `refactor`, refactoring code
+ `perf`, Performance improvements + `deprecate`, deprecating a feature
+ `refactor`, Refactoring code + `changelog`, updating the CHANGELOG
+ `remove`, Remove a feature or fix + `chore`, maintenance (build process, updating sponsors, etc.)
+ `test`, Adding tests
+ Other prefixes may be used if necessary + Other prefixes may be used if necessary
- If there's a breaking change, add `BREAKING CHANGE` and an explanation to your commit message - If there's a breaking change, add `BREAKING CHANGE` and an explanation to your commit message
--> -->
-3
View File
@@ -2,7 +2,4 @@
# and made its way to us through nixpkgs. # and made its way to us through nixpkgs.
self-hosted-runner: self-hosted-runner:
labels: labels:
- macos-15-intel
- macos-26
- ubuntu-24.04-arm - ubuntu-24.04-arm
- ubuntu-slim
-35
View File
@@ -1,35 +0,0 @@
name: Cache on main
description: Stores caches on main and release branches only, but restores them on all branches.
inputs:
path:
description: Path(s) to cache
required: true
save-prs:
description: Whether to additionally store the cache in a pull request, too. Should only be used for very small caches.
type: boolean
prefix:
description: Cache key prefix to be used in both primary key and restore-keys.
required: true
suffix:
description: Cache key suffix to be used only in primary key.
required: true
runs:
using: composite
steps:
- uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
if: ${{ startsWith(github.ref, 'refs/heads/') || (inputs.save-prs && startsWith(github.ref, 'refs/pull/')) }}
with:
path: ${{ inputs.path }}
key: ${{ runner.os }}-${{ inputs.prefix }}-${{ inputs.suffix }}
restore-keys: |
${{ runner.os }}-${{ inputs.prefix }}-
- uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
if: ${{ !startsWith(github.ref, 'refs/heads/') && !(inputs.save-prs && startsWith(github.ref, 'refs/pull/')) }}
with:
path: ${{ inputs.path }}
key: ${{ runner.os }}-${{ inputs.prefix }}-${{ inputs.suffix }}
restore-keys: |
${{ runner.os }}-${{ inputs.prefix }}-
-35
View File
@@ -1,35 +0,0 @@
name: Run anywhere
description: Runs the same code either in a VM or on the bare machine
inputs:
vm:
description: Which VM to run on.
envs:
description: List of relevant environment variables, which might need to be copied into the VM.
prepare:
description: Code to run in a prepare step, e.g. installing dependencies.
run:
description: Code to run as the main action.
required: true
runs:
using: composite
steps:
- if: ${{ inputs.vm == 'freebsd' }}
uses: vmactions/freebsd-vm@83b151f58c6047089f4c80eb5ba2039d158ce093 # v1.5.3
with:
envs: ${{ inputs.envs }}
prepare: ${{ inputs.prepare }}
# Work around https://github.com/vmactions/freebsd-vm/issues/59
run: |
pw user add -n action -m
su action -c '${{ inputs.run }}'
- if: ${{ inputs.vm == '' }}
name: Prepare
shell: ${{ runner.os == 'Windows' && 'pwsh' || 'bash' }}
run: ${{ inputs.prepare }}
- if: ${{ inputs.vm == '' }}
name: Run
shell: ${{ runner.os == 'Windows' && 'pwsh' || 'bash' }}
run: ${{ inputs.run }}
+10 -39
View File
@@ -3,7 +3,7 @@
"extends": [ "extends": [
"config:best-practices" "config:best-practices"
], ],
"baseBranchPatterns": [ "baseBranches": [
"main", "main",
"/^v[0-9]+/" "/^v[0-9]+/"
], ],
@@ -13,56 +13,27 @@
}, },
"packageRules": [ "packageRules": [
{ {
"matchBaseBranches": [ "matchBaseBranches": [ "/^v[0-9]+/" ],
"/^v[0-9]+/" "matchManagers": ["haskell-cabal"],
],
"matchManagers": [
"haskell-cabal"
],
"enabled": false "enabled": false
}, },
{ {
"matchBaseBranches": [ "matchBaseBranches": [ "/^v[0-9]+/" ],
"/^v[0-9]+/"
],
"groupName": "all dependencies" "groupName": "all dependencies"
}, },
{ {
"matchManagers": [ "matchManagers": ["haskell-cabal"],
"haskell-cabal" "matchPackageNames": ["base", "bytestring", "containers", "directory", "mtl", "parsec", "process", "text"],
],
"matchPackageNames": [
"base",
"bytestring",
"containers",
"directory",
"mtl",
"parsec",
"process",
"text"
],
"groupName": "GHC dependencies" "groupName": "GHC dependencies"
}, },
{ {
"matchManagers": [ "matchManagers": ["haskell-cabal"],
"haskell-cabal" "matchPackageNames": ["hasql", "hasql-dynamic-statements", "hasql-notifications", "hasql-transaction", "hasql-pool"],
],
"matchPackageNames": [
"hasql",
"hasql-dynamic-statements",
"hasql-notifications",
"hasql-transaction",
"hasql-pool"
],
"groupName": "hasql" "groupName": "hasql"
}, },
{ {
"matchManagers": [ "matchManagers": ["haskell-cabal"],
"haskell-cabal" "matchPackageNames": ["fuzzyset"],
],
"matchPackageNames": [
"fuzzyset"
],
"allowedVersions": "<0.3" "allowedVersions": "<0.3"
} }
] ]
-52
View File
@@ -1,52 +0,0 @@
name: Backport
on:
pull_request_target:
types:
- closed
- labeled
jobs:
backport:
name: Backport
runs-on: ubuntu-slim
# It triggers only when PR is already merged on either:
#
# - The merge event itself (action != labeled) or
# - A label event with the right label (backport ...).
#
# The result will be that we can add the label before or after merge,
# but the workflow will only run once the PR had been merged.
if: >
github.event.pull_request.merged &&
(
github.event.action != 'labeled' ||
startsWith(github.event.label.name, 'backport')
)
steps:
# This actions creates the github token using the postgrest app secrets
- name: Create Github App Token
id: app-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with:
client-id: ${{ vars.POSTGREST_CI_APP_ID }}
private-key: ${{ secrets.POSTGREST_CI_PRIVATE_KEY }}
permission-contents: write
permission-pull-requests: write
permission-workflows: write # required when backporting CI changes
# This is required for backport action to cherry-pick the PR
- name: Fetch PR ref
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.event.pull_request.head.sha }}
token: ${{ steps.app-token.outputs.token }}
# Backport action that creates the PR with given settings
- name: Create backport PR
uses: korthout/backport-action@2e830a1d0b8269505846ddd407a70876913ad1f8 # v4.6
with:
github_token: ${{ steps.app-token.outputs.token }}
pull_description: 'Backport for #${pull_number}.'
pull_title: '${target_branch}: ${pull_title}'
-189
View File
@@ -1,189 +0,0 @@
name: Build
on:
workflow_call:
secrets:
CACHIX_AUTH_TOKEN:
required: false
pull_request:
branches:
- main
- v[0-9]+
paths:
- .github/workflows/build.yaml
- .github/actions/**
- .github/scripts/**
- .github/*
- '*.nix'
- nix/**
- flake.lock
- .cirrus.yml
- cabal.project*
- postgrest.cabal
- stack.yaml*
- '**.hs'
- '!**.md'
concurrency:
# Terminate all previous runs of the same workflow for pull requests
group: build-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
jobs:
static:
name: Nix - Linux x86-64 static
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment
uses: ./.github/actions/setup-nix
with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
- name: Build static executable
run: nix-build -A postgrestStatic -A postgrestStatic.tests
- name: Save built executable as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: postgrest-linux-static-x86-64
path: result/bin/postgrest
if-no-files-found: error
- name: Build Docker image
run: nix-build -A docker.image --out-link postgrest-docker.tar.gz
- name: Save built Docker image as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: postgrest-docker-x86-64
path: postgrest-docker.tar.gz
if-no-files-found: error
macos:
name: Nix - MacOS
runs-on: macos-26
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment
uses: ./.github/actions/setup-nix
with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
- name: Install nix-build-uncached
run: nix-env -f default.nix -iA nix-build-uncached
- name: Build everything (default.nix)
run: nix-build-uncached
- name: Build everything (shell.nix)
run: nix-build-uncached shell.nix
stack:
strategy:
fail-fast: false
matrix:
include:
- name: FreeBSD x86-64
runs-on: ubuntu-24.04
vm: freebsd
artifact: postgrest-freebsd-x86-64
deps: pkg install -y git postgresql16-client hs-stack
- name: Linux aarch64
runs-on: ubuntu-24.04-arm
artifact: postgrest-ubuntu-aarch64
deps: sudo apt-get update && sudo apt-get install libpq-dev
- name: MacOS aarch64
runs-on: macos-14
artifact: postgrest-macos-aarch64
deps: brew link --force libpq
- name: MacOS x86-64
runs-on: macos-15-intel
artifact: postgrest-macos-x86-64
deps: brew link --force libpq
- name: Windows
runs-on: windows-2022
deps: Add-Content $env:GITHUB_PATH $env:PGBIN
artifact: postgrest-windows-x86-64
name: Stack - ${{ matrix.name }}
runs-on: ${{ matrix.runs-on }}
env:
# Putting .stack in the working directory helps with moving this in and out of the FreeBSD VM.
STACK_ROOT: ${{ github.workspace }}/.stack
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- if: ${{ !matrix.vm }}
uses: haskell-actions/setup@6037f33647c3f17758a2356c80fc4a53d7e0685d # v2.12.0
with:
# This must match the version in stack.yaml's resolver
ghc-version: 9.6.7
enable-stack: true
stack-no-global: true
stack-setup-ghc: true
- name: Cache .stack
uses: ./.github/actions/cache-on-main
with:
path: .stack
prefix: ${{ matrix.vm }}${{ matrix.vm && '-' }}stack
suffix: ${{ hashFiles('postgrest.cabal', 'stack.yaml.lock') }}
- name: Cache .stack-work
uses: ./.github/actions/cache-on-main
with:
path: .stack-work
save-prs: true
prefix: ${{ matrix.vm }}${{ matrix.vm && '-' }}stack-work-${{ hashFiles('postgrest.cabal', 'stack.yaml.lock') }}
suffix: ${{ hashFiles('main/**/*.hs', 'src/**/*.hs') }}
- name: Build with Stack
uses: ./.github/actions/run-anywhere
with:
vm: ${{ matrix.vm }}
envs: STACK_ROOT
prepare: ${{ matrix.deps }}
run: |
stack build --lock-file error-on-write --local-bin-path result --copy-bins
strip result/postgrest*
- name: Save built executable as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}
path: |
result/postgrest
result/postgrest.exe
if-no-files-found: error
cabal:
strategy:
matrix:
ghc: ['9.6.7', '9.8.4']
fail-fast: false
name: Cabal - Linux x86-64 - GHC ${{ matrix.ghc }}
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: haskell-actions/setup@6037f33647c3f17758a2356c80fc4a53d7e0685d # v2.12.0
with:
ghc-version: ${{ matrix.ghc }}
- name: Cache .cabal
uses: ./.github/actions/cache-on-main
with:
path: |
~/.cabal/packages
~/.cabal/store
prefix: cabal-${{ matrix.ghc }}-${{ hashFiles('cabal.project.freeze') }}
suffix: ${{ hashFiles('postgrest.cabal', 'cabal.project') }}
- name: Cache dist-newstyle
uses: ./.github/actions/cache-on-main
with:
path: dist-newstyle
save-prs: true
prefix: cabal-${{ matrix.ghc }}-dist-newstyle-${{ hashFiles('postgrest.cabal', 'cabal.project', 'cabal.project.freeze') }}
suffix: ${{ hashFiles('**/*.hs') }}
- name: Install dependencies
run: cabal build --only-dependencies --enable-tests --enable-benchmarks
- name: Build
run: cabal build --enable-tests --enable-benchmarks all
-53
View File
@@ -1,53 +0,0 @@
name: Check
on:
workflow_call:
secrets:
CACHIX_AUTH_TOKEN:
required: false
pull_request:
branches:
- main
- v[0-9]+
concurrency:
# Terminate all previous runs of the same workflow for pull requests
group: style-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
jobs:
lint-style:
name: Lint & Style
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment
uses: ./.github/actions/setup-nix
with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
tools: style.lint.bin style.styleCheck.bin
- name: Run linter (check locally with `nix-shell --run postgrest-lint`)
run: postgrest-lint
- name: Run style check (auto-format with `nix-shell --run postgrest-style`)
run: postgrest-style-check
commit:
if: github.event_name != 'push' # we don't run this on a push, a failure on push disrupts the release workflow
name: Commit
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 100 # fetch history (last 100 commits) instead of default shallow clone history, this is deemed enough for a PR history
- name: Setup Nix Environment
uses: ./.github/actions/setup-nix
with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
tools: gitTools.commitCheck.bin
- name: Run commitlint (check locally with `nix-shell --run postgrest-commitlint`)
run: |
# Fetch target branch explicitly
git fetch origin ${{ github.base_ref }}
# Run commitlint
postgrest-commitlint --from origin/${{ github.base_ref }} --to HEAD
-69
View File
@@ -1,69 +0,0 @@
name: CI
on:
push:
branches:
- main
- v[0-9]+
jobs:
check:
name: Check
uses: ./.github/workflows/check.yaml
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
docs:
name: Docs
uses: ./.github/workflows/docs.yaml
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
test:
name: Test
uses: ./.github/workflows/test.yaml
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
build:
name: Build
uses: ./.github/workflows/build.yaml
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
tag:
name: Tag
concurrency:
# Never tag outdated commits on the main branch by skipping superseded commits
group: ci-tag-${{ (github.ref == 'refs/heads/main' && github.ref) || github.run_id }}
cancel-in-progress: true
if: vars.RELEASE_ENABLED
runs-on: ubuntu-slim
needs:
- docs
- test
- build
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ssh-key: ${{ secrets.POSTGREST_SSH_KEY }}
- name: Tag latest commit
run: |
cabal_version="$(grep -oP '^version:\s*\K.*' postgrest.cabal)"
if [[ "$cabal_version" == *.* ]]; then
git fetch --tags
if [ -z "$(git tag --list "v$cabal_version")" ]; then
git tag "v$cabal_version"
git push origin "v$cabal_version"
fi
else
git tag -f "devel"
git push -f origin "devel"
fi
-1
View File
@@ -14,7 +14,6 @@ on:
- .github/actions/setup-nix/** - .github/actions/setup-nix/**
- default.nix - default.nix
- nix/** - nix/**
- flake.lock
- docs/** - docs/**
- '!**.md' - '!**.md'
-43
View File
@@ -1,43 +0,0 @@
name: Linkcheck
on:
schedule:
- cron: '1 2 * * 3'
workflow_dispatch:
jobs:
linkcheck:
name: Linkcheck
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment
uses: ./.github/actions/setup-nix
with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
tools: docs.linkcheck.bin
- name: Run Linkcheck
id: linkcheck
run: postgrest-docs-linkcheck
# This actions creates the github token using the postgrest app secrets
- name: Create Github App Token (Runs only on linkcheck failure)
id: app-token
if: ${{ failure() && steps.linkcheck.outcome == 'failure' }} # only create the token on linkcheck failure
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with:
client-id: ${{ vars.POSTGREST_CI_APP_ID }}
private-key: ${{ secrets.POSTGREST_CI_PRIVATE_KEY }}
permission-issues: write # required for commenting on issues
- name: Notify on linkcheck failure by commenting on GH Issue 4106
if: ${{ failure() && steps.linkcheck.outcome == 'failure' }}
uses: peter-evans/create-or-update-comment@e8674b075228eee787fea43ef493e45ece1004c9 # v5.0.0
with:
token: ${{ steps.app-token.outputs.token }}
issue-number: 4106
body: |
**Linkcheck Job Failed!**
A broken link was detected in the docs. Please check the [failed run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details.
-183
View File
@@ -1,183 +0,0 @@
name: Release
on:
push:
tags:
- devel
- v*
concurrency:
# Terminate all previous runs of the same workflow for the same tag.
group: release-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
name: Build
uses: ./.github/workflows/build.yaml
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
github:
name: GitHub
permissions:
contents: write
runs-on: ubuntu-slim
needs:
- build
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Check the version to be released
run: |
cabal_version="$(grep -oP '^version:\s*\K.*' postgrest.cabal)"
if [ "${GITHUB_REF_NAME}" != "devel" ] && [ "${GITHUB_REF_NAME}" != "v$cabal_version" ]; then
echo "Tagged version ($GITHUB_REF_NAME) does not match the one in postgrest.cabal (v$cabal_version). Aborting release..."
exit 1
fi
- name: Identify changes from CHANGELOG.md
run: |
if [ "${GITHUB_REF_NAME}" == "devel" ]; then
echo "Getting unreleased changes..."
sed -n "1,/## Unreleased/d;/## \[/q;p" CHANGELOG.md > CHANGES.md
else
version="$(grep -oP '^version:\s*\K.*' postgrest.cabal)"
echo "Propper release, getting changes for version $version ..."
sed -n "1,/## \[$version\]/d;/## \[/q;p" CHANGELOG.md > CHANGES.md
fi
echo "Relevant extract from CHANGELOG.md:"
cat CHANGES.md
- name: Download all artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: artifacts
- name: Create release bundle with archives for all builds
run: |
find artifacts -type f -iname postgrest -exec chmod +x {} \;
mkdir -p release-bundle
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-linux-static-x86-64.tar.xz" \
-C artifacts/postgrest-linux-static-x86-64 postgrest
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-macos-aarch64.tar.xz" \
-C artifacts/postgrest-macos-aarch64 postgrest
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-macos-x86-64.tar.xz" \
-C artifacts/postgrest-macos-x86-64 postgrest
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-freebsd-x86-64.tar.xz" \
-C artifacts/postgrest-freebsd-x86-64 postgrest
tar cJvf "release-bundle/postgrest-${GITHUB_REF_NAME}-ubuntu-aarch64.tar.xz" \
-C artifacts/postgrest-ubuntu-aarch64 postgrest
zip --junk-paths "release-bundle/postgrest-${GITHUB_REF_NAME}-windows-x86-64.zip" \
artifacts/postgrest-windows-x86-64/postgrest.exe
- name: Save release bundle
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-bundle
path: release-bundle
if-no-files-found: error
- name: Publish release on GitHub
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
echo "Releasing version ${GITHUB_REF_NAME} on GitHub..."
if [ "${GITHUB_REF_NAME}" == "devel" ]; then
# To replace the existing release, we must first delete the old assets,
# then modify the release, then add the new assets.
gh release view devel --json assets \
| jq -r '.assets[] | .name' \
| xargs -rn1 \
gh release delete-asset -y devel
gh release edit devel \
-t devel \
--verify-tag \
-F CHANGES.md \
--prerelease
gh release upload --clobber devel release-bundle/*
else
gh release create "${GITHUB_REF_NAME}" \
-t "${GITHUB_REF_NAME}" \
--verify-tag \
-F CHANGES.md \
release-bundle/*
fi
docker:
name: Docker Hub
runs-on: ubuntu-24.04-arm
needs:
- github
if: |
vars.DOCKER_REPO && vars.DOCKER_USER
env:
DOCKER_REPO: ${{ vars.DOCKER_REPO }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Download x86-64 Docker image
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: postgrest-docker-x86-64
- name: Download aarch64 binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: postgrest-ubuntu-aarch64
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
username: ${{ vars.DOCKER_USER }}
password: ${{ secrets.DOCKER_PASS }}
- name: Build aarch64 Docker image
run: |
# This only pushes the image via digest, not a tag. This will not appear
# in the image list on Docker Hub, yet. It will be later added to the main
# tag's manifest.
docker buildx build \
-t "$DOCKER_REPO/postgrest" \
--platform linux/arm64 \
--output push-by-digest=true,type=image,push=true \
--metadata-file metadata.json \
.
echo "SHA256_ARM=$(jq -r '."containerimage.digest"' metadata.json)" >> "$GITHUB_ENV"
- name: Publish images on Docker Hub
run: |
docker load -i postgrest-docker.tar.gz
docker tag postgrest:latest "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}"
docker push "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}"
docker buildx imagetools create --append \
-t "$DOCKER_REPO/postgrest:${GITHUB_REF_NAME}" \
"$DOCKER_REPO/postgrest@$SHA256_ARM"
# Only tag 'latest' for full releases
if [ "${GITHUB_REF_NAME}" != "devel" ]; then
echo "Pushing to 'latest' tag for full release of ${GITHUB_REF_NAME} ..."
docker tag postgrest:latest "$DOCKER_REPO"/postgrest:latest
docker push "$DOCKER_REPO"/postgrest:latest
docker buildx imagetools create --append \
-t "$DOCKER_REPO/postgrest:latest" \
"$DOCKER_REPO/postgrest@$SHA256_ARM"
else
echo "Skipping push to 'latest' tag for pre-release..."
fi
- uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0
if: github.ref == 'refs/tags/devel'
name: Docker Hub Description
with:
username: ${{ vars.DOCKER_USER }}
password: ${{ secrets.DOCKER_PASS }}
repository: ${{ vars.DOCKER_REPO }}/postgrest
short-description: ${{ github.event.repository.description }}
readme-filepath: ./docker-hub-readme.md
-178
View File
@@ -1,178 +0,0 @@
name: Test
on:
workflow_call:
secrets:
CACHIX_AUTH_TOKEN:
required: false
CODECOV_TOKEN:
required: false
pull_request:
branches:
- main
- v[0-9]+
paths:
- .github/workflows/test.yaml
- .github/workflows/report.yaml
- .github/actions/setup-nix/**
- default.nix
- nix/**
- flake.lock
- .stylish-haskell.yaml
- cabal.project
- postgrest.cabal
- '**.hs'
- test/**
- '!**.md'
concurrency:
# Terminate all previous runs of the same workflow for pull requests
group: test-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
jobs:
coverage:
name: Coverage
runs-on: ubuntu-24.04
defaults:
run:
# Hack for enabling color output, see:
# https://github.com/actions/runner/issues/241#issuecomment-842566950
shell: script -qec "bash --noprofile --norc -eo pipefail {0}"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment
uses: ./.github/actions/setup-nix
with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
tools: tests.coverage.bin tests.testDoctests.bin tests.testSpecIdempotence.bin cabalTools.update.bin
- run: postgrest-cabal-update
- name: Run coverage (IO tests and Spec tests against PostgreSQL 15)
run: postgrest-coverage
- name: Upload coverage to codecov
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
files: ./coverage/codecov.json
token: ${{ secrets.CODECOV_TOKEN }}
- name: Run doctests
if: always()
run: postgrest-test-doctests
- name: Check the spec tests for idempotence
if: always()
run: postgrest-test-spec-idempotence
postgres:
strategy:
fail-fast: false
matrix:
# Latest version is tested via `coverage` above.
pgVersion: [13, 14, 15, 16]
name: PG ${{ matrix.pgVersion }}
runs-on: ubuntu-24.04
defaults:
run:
# Hack for enabling color output, see:
# https://github.com/actions/runner/issues/241#issuecomment-842566950
shell: script -qec "bash --noprofile --norc -eo pipefail {0}"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment
uses: ./.github/actions/setup-nix
with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
tools: tests.testSpec.bin tests.testObservability.bin tests.testIO.bin tests.testBigSchema.bin withTools.pg-${{ matrix.pgVersion }}.bin cabalTools.update.bin
- run: postgrest-cabal-update
- name: Run spec tests
if: always()
run: postgrest-with-pg-${{ matrix.pgVersion }} postgrest-test-spec
- name: Run observability tests
if: always()
run: postgrest-with-pg-${{ matrix.pgVersion }} postgrest-test-observability
- name: Run IO tests
if: always()
run: postgrest-with-pg-${{ matrix.pgVersion }} postgrest-test-io -vv
- name: Run IO tests on a big schema
if: always()
run: postgrest-with-pg-${{ matrix.pgVersion }} postgrest-test-big-schema -vv
memory:
name: Memory
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Setup Nix Environment
uses: ./.github/actions/setup-nix
with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
tools: tests.testMemory.bin cabalTools.update.bin
- run: postgrest-cabal-update
- name: Run memory tests
run: postgrest-test-memory
loadtest:
strategy:
fail-fast: false
matrix:
kind: ['mixed', 'jwt-hs', 'jwt-hs-cache', 'jwt-hs-cache-worst', 'jwt-rsa', 'jwt-rsa-cache', 'jwt-rsa-cache-worst']
name: Loadtest
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
- name: Setup Nix Environment
uses: ./.github/actions/setup-nix
with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
tools: loadtest.loadtestAgainst.bin loadtest.report.bin cabalTools.update.bin
- run: postgrest-cabal-update
- name: Run loadtest
env:
TARGET_BRANCH: ${{ github.base_ref || github.ref_name }}
run: |
if [ "$TARGET_BRANCH" = "main" ]; then
latest_tag=$(git tag --sort=-creatordate --list "v*" | head -n1)
else
latest_tag=$(git tag --merged HEAD --sort=-creatordate "v*" | head -n1)
fi
postgrest-loadtest-against -k ${{ matrix.kind }} "$TARGET_BRANCH" "$latest_tag"
postgrest-loadtest-report -g ${{ matrix.kind }} >> "$GITHUB_STEP_SUMMARY"
flake:
strategy:
fail-fast: false
matrix:
runs-on:
- macos-15-intel # x86_64-darwin
- macos-14 # aarch64-darwin
- ubuntu-24.04 # x86_64-linux
- ubuntu-24.04-arm # aarch64-linux
name: Flake Check
runs-on: ${{ matrix.runs-on }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
- name: Setup Nix Environment
uses: ./.github/actions/setup-nix
with:
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
- name: Run flake check
run: |
nix flake check
-2
View File
@@ -25,5 +25,3 @@ loadtest
.history .history
.docs-build .docs-build
gen_targets.http gen_targets.http
gen_jwk.json
.ghc.environment.*
+8 -13
View File
@@ -13,26 +13,26 @@ PostgREST ongoing development is only possible thanks to our Sponsors and Backer
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank"> <a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="static/supabase.svg"> <img width="296px" src="static/neon.jpg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://www.euronodes.com/postgrest" target="_blank"> <a href="https://tembo.io/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="static/euronodes.svg"> <img width="296px" src="static/tembo.png">
</a> </a>
</td> </td>
</tr> </tr>
<tr></tr> <tr></tr>
<tr> <tr>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://www.euronodes.com/postgrest" target="_blank">
<img width="296px" src="static/neon.jpg"> <img width="296px" src="static/euronodes.svg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://www.bytebase.com/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank">
<img width="296px" src="static/bytebase.svg"> <img width="296px" src="static/supabase.svg">
</a> </a>
</td> </td>
</tr> </tr>
@@ -93,11 +93,6 @@ PostgREST ongoing development is only possible thanks to our Sponsors and Backer
<img width="222px" src="static/code-build.png"> <img width="222px" src="static/code-build.png">
</a> </a>
</td> </td>
<td align="center" valign="middle">
<a href="https://tembo.io/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="static/tembo.png">
</a>
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
+5 -160
View File
@@ -1,162 +1,10 @@
# Change Log # Change Log
All notable changes to this project will be documented in this file. From version `14.0` onwards PostgREST follows a `MAJOR.PATCH` two-part versioning. Only even-numbered MAJOR versions will be released, reserving odd-numbered MAJOR versions for development. All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).
## Unreleased ## Unreleased
## [14.17] - 2026-08-13
### Fixed
- JWT validation uses wrong current time due to a bug in auto-update by @mkleczek in #5159
## [14.16] - 2026-07-27
### Fixed
- Fix admin server crashing without a way to recover by @taimoorzaeem in #5096
## [14.15] - 2026-07-13
### Fixed
- Fix admin server dying silently by @Vlix, @mkleczek, @steve-chavez in #5012
## [14.14] - 2026-06-29
### Fixed
- Fix admin server not logging cause of failure by @taimoorzaeem in #5012
## [14.13] - 2026-06-04
### Fixed
- Fix connection retrying message in `PGRST000` error by @netqo in #4980
+ Remove redundant "Retrying the connection." from message because it is logged separately
- Fix request failures when `work_mem` is set on a role by @laurenceisla in #4955
## [14.12] - 2026-05-20
### Fixed
- Fix race condition in pool_available metric causing negative values during network instability by @mkleczek in #4622
## [14.11] - 2026-05-04
### Fixed
- Fix login with uppercase and mixed case role names by @taimoorzaeem in #4678
- Restore Listener query shape so it can be found in `pg_stat_activity` by @mkleczek in #4857 #4859
- The LISTEN channel now automatically recovers when it stops working due to a PostgreSQL bug @laurenceisla in #3147
- Fix misleading "Functions" name on schema cache summary in startup logs by @taimoorzaeem in #4821
## [14.10] - 2026-04-16
### Added
- Log when the pool is released during schema cache reload on `log-level=debug` by @mkleczek in #4668
### Fixed
- Fix unnecessary connection pool flushes during schema cache reloading by @mkleczek in #4645
## [14.9] - 2026-04-10
### Added
- Log host, port and pg version of listener database connection by @mkleczek in #4617 #4618
### Fixed
- Remove red herring warp logs on default log-level, only emit them on `log-level=debug` by @steve-chavez in #4799
## [14.8] - 2026-04-03
### Added
- Log a `HINT` when the LISTEN channel stops working due to a PostgreSQL bug by @laurenceisla in #4581
### Fixed
- Fix invalid OpenAPI 2.0 format for integer types (`smallint`, `integer`, `bigint`) by @arturbent0 in #4641
## [14.7] - 2026-03-20
### Fixed
- Fix not logging SIGTERM and SIGINT by @steve-chavez in #4728
## [14.6] - 2026-03-06
### Fixed
- Fix leaking table and function names when calculating error hint by @taimoorzaeem in #4675
## [14.5] - 2026-02-12
### Fixed
- Don't hide async exceptions in logs by @stevechavez in #4646
## [14.4] - 2026-01-29
### Fixed
- Ensure Listener connections are released by @mkleczek in #4614
- Fix incorrectly filtering the returned representation for PATCH requests when using `or/and` filters by @laurenceisla in #3707
- Fix listener running with exception masked after first failure by @mkleczek #4615
## [14.3] - 2026-01-03
### Fixed
- Fix performance and high memory usage of relation hint calculation by @mkleczek in #4462, #4463
## [14.2] - 2025-12-18
### Fixed
- Fix `hasSingleUnnamedParam` incorrectly matching functions with named parameters by @joelonsql in #4553
+ Functions with a single named parameter (e.g., `foo(data json)`) no longer incorrectly match the single-param fallback, returning a clean `PGRST202` error instead of a confusing PostgreSQL `42883` error.
- Fix misleading logs on unsupported PostgreSQL versions by @taimoorzaeem in #4519
- Fix regression where the `PGRST103` error response was truncated by @laurenceisla in #4455
+ Happened when an `offset` was greater than the rows requested and `Prefer: count=exact` was sent.
- Fix not returning `Content-Length` on empty HTTP `201` responses by @laurenceisla in #4518
- Fix inaccurate Server-Timing header durations by @steve-chavez in #4522
- Fix inaccurate "Schema cache queried" logs by @steve-chavez in #4522
## [14.1] - 2025-11-05
## Fixed
- Fix `db-pre-config` function failing when function names are pg reserved words by @taimoorzaeem in #4380
- Fix `server-host=!6` incorrectly binds to IPv4 address by @taimoorzaeem in #3202
## [14.0] - 2025-10-24
### Added
- Bounded JWT cache using the SIEVE algorithm by @mkleczek in #4084
+ It now uses a fixed size cache instead of arbitrary sized cache.
- Add `--ready` flag for postgrest healthcheck by @taimoorzaeem in #4239
### Fixed
- Fix not logging OpenAPI queries when `log-query=main-query` is enabled by @steve-chavez in #4226
- Fix not logging explain query when `log-query=main-query` is enabled by @steve-chavez in #4319
- Fix not logging transaction variables and db-pre-request function when `log-query=main-query` is enabled by @steve-chavez in #3934
- Fix not logging the JSON message to stderr on a `PGRST002` error by @laurenceisla in #4129
- Fix reloading the Schema Cache unnecessarily on a `PGRST002` error by @laurenceisla in #4367
- Fix schema cache loading taking a long time for large schemas by @mkleczek in #4360, #3704
### Changed
- Drop support for PostgreSQL EOL version 12 by @wolfgangwalther in #3865
- From now on PostgREST will follow a `MAJOR.PATCH` two-part versioning. Only even-numbered MAJOR versions will be released, reserving odd-numbered MAJOR versions for development.
- Replaced `jwt-cache-max-lifetime` config with `jwt-cache-max-entries` by @mkleczek in #4084
- `log-query` config now takes a boolean instead of a string value by @steve-chavez in #3934
## [13.0.8] - 2025-10-24 ## [13.0.8] - 2025-10-24
### Fixed ### Fixed
@@ -184,7 +32,7 @@ All notable changes to this project will be documented in this file. From versio
### Fixed ### Fixed
- Fix OpenAPI broken docs link by @taimoorzaeem in #4080 - Fix OpenAPI broken docs link by @taimoorzaeem in #4048
- Fix OpenAPI specification incorrectly exposing GET methods for volatile functions by @joelonsql in #4174 - Fix OpenAPI specification incorrectly exposing GET methods for volatile functions by @joelonsql in #4174
- Fix empty spread embeddings return unexpected SQL error by @taimoorzaeem in #3887 - Fix empty spread embeddings return unexpected SQL error by @taimoorzaeem in #3887
- Fix `/metrics` endpoint not responding with `Content-Type` header by @taimoorzaeem in #4271 - Fix `/metrics` endpoint not responding with `Content-Type` header by @taimoorzaeem in #4271
@@ -198,11 +46,11 @@ All notable changes to this project will be documented in this file. From versio
## [13.0.3] - 2025-06-16 ## [13.0.3] - 2025-06-16
### Fixed
- Fix `max-affected` preference not failing with RPC when `handling=strict` by @taimoorzaeem in #4100 - Fix `max-affected` preference not failing with RPC when `handling=strict` by @taimoorzaeem in #4100
- Fix a property definition's type in OpenAPI not showing the correct base type of a recursive domain by @laurenceisla in #4136 - Fix a property definition's type in OpenAPI not showing the correct base type of a recursive domain by @laurenceisla in #4136
### Fixed
## [13.0.2] - 2025-06-02 ## [13.0.2] - 2025-06-02
### Fixed ### Fixed
@@ -270,9 +118,6 @@ All notable changes to this project will be documented in this file. From versio
- #3956, Drop `/config` endpoint of admin server - @steve-chavez - #3956, Drop `/config` endpoint of admin server - @steve-chavez
+ The endpoint was at risk of being left unprotected when exposing it. + The endpoint was at risk of being left unprotected when exposing it.
+ The accompanying `admin-server-config-enabled` config was also dropped. + The accompanying `admin-server-config-enabled` config was also dropped.
- #3598, PostgREST now validates the `kid` parameter of the JWT - @wolfgangwalther
+ If the JWT contains a ``kid`` parameter, then PostgREST will look for the JSON Web Key in the `jwt-secret`.
+ If the JWT doesn't contain a `kid`, the behavior should be backwards compatible. PostgREST will try each key in the `jwt-secret` one by one until it finds one that works.
- #3697, #3602, Querying non-existent table now returns `PGRST205` error instead of empty json - @taimoorzaeem - #3697, #3602, Querying non-existent table now returns `PGRST205` error instead of empty json - @taimoorzaeem
- #3600, #3926, Improve JWT errors - @taimoorzaeem - #3600, #3926, Improve JWT errors - @taimoorzaeem
+ Return `PGRST301` error when `Bearer` in auth header is sent empty + Return `PGRST301` error when `Bearer` in auth header is sent empty
-21
View File
@@ -1,21 +0,0 @@
# PostgREST Docker Hub image for aarch64.
# The x86-64 is a single-static-binary image built via Nix, see:
# nix/tools/docker/README.md
FROM ubuntu:resolute@sha256:678c6550cc43645e08669028bc177f50be4e7c5b8cca677067b1914d4afc7a03 AS postgrest
RUN apt-get update -y \
&& apt install -y --no-install-recommends libpq-dev zlib1g-dev jq gcc libnuma-dev \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
COPY postgrest /usr/bin/postgrest
RUN chmod +x /usr/bin/postgrest
EXPOSE 3000
USER 1000
# Use the array form to avoid running the command using bash, which does not handle `SIGTERM` properly.
# See https://docs.docker.com/compose/faq/#why-do-my-services-take-10-seconds-to-recreate-or-stop
CMD ["postgrest"]
+10 -14
View File
@@ -22,26 +22,26 @@ API than you are likely to write from scratch.
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank"> <a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="static/supabase.svg"> <img width="296px" src="static/neon.jpg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://www.euronodes.com/postgrest" target="_blank"> <a href="https://tembo.io/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="static/euronodes.svg"> <img width="296px" src="static/tembo.png">
</a> </a>
</td> </td>
</tr> </tr>
<tr></tr> <tr></tr>
<tr> <tr>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://www.euronodes.com/postgrest" target="_blank">
<img width="296px" src="static/neon.jpg"> <img width="296px" src="static/euronodes.svg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://www.bytebase.com/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank">
<img width="296px" src="static/bytebase.svg"> <img width="296px" src="static/supabase.svg">
</a> </a>
</td> </td>
</tr> </tr>
@@ -52,8 +52,8 @@ Big thanks to our sponsors! You can join them by supporting PostgREST on [Patreo
## Usage ## Usage
1. See the docs for [how to install PostgREST on your platform](https://docs.postgrest.org/en/stable/explanations/install.html). You can also [use Docker](https://docs.postgrest.org/en/stable/explanations/install.html#docker). 1. Download the binary ([latest release](https://github.com/PostgREST/postgrest/releases/latest))
for your platform.
2. Invoke for help: 2. Invoke for help:
```bash ```bash
@@ -142,10 +142,6 @@ You can help PostgREST ongoing maintenance and development by making a regular d
Every donation will be spent on making PostgREST better for the whole community. Every donation will be spent on making PostgREST better for the whole community.
## Contributing
Contributions are always welcome and appreciated. Please see the [Contributing guidelines](https://github.com/PostgREST/postgrest/blob/main/CONTRIBUTING.md).
## Thanks ## Thanks
The PostgREST organization is grateful to: The PostgREST organization is grateful to:
+2
View File
@@ -1,2 +1,4 @@
packages: postgrest.cabal packages: postgrest.cabal
tests: true tests: true
package *
ghc-options: -split-sections
+1 -1
View File
@@ -1 +1 @@
index-state: hackage.haskell.org 2026-08-10T16:58:32Z index-state: hackage.haskell.org 2025-10-13T04:53:27Z
+12 -28
View File
@@ -43,6 +43,7 @@ let
allOverlays.build-toolbox allOverlays.build-toolbox
allOverlays.checked-shell-script allOverlays.checked-shell-script
allOverlays.gitignore allOverlays.gitignore
allOverlays.postgresql-libpq
(allOverlays.haskell-packages { inherit compiler; }) (allOverlays.haskell-packages { inherit compiler; })
allOverlays.slocat allOverlays.slocat
]; ];
@@ -53,17 +54,16 @@ let
postgresqlVersions = postgresqlVersions =
[ [
{ name = "pg-17"; postgresql = pkgs.postgresql_17.withPackages (p: [ p.postgis p.pg_safeupdate ]); } { name = "postgresql-17"; postgresql = pkgs.postgresql_17.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "pg-16"; postgresql = pkgs.postgresql_16.withPackages (p: [ p.postgis p.pg_safeupdate ]); } { name = "postgresql-16"; postgresql = pkgs.postgresql_16.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "pg-15"; postgresql = pkgs.postgresql_15.withPackages (p: [ p.postgis p.pg_safeupdate ]); } { name = "postgresql-15"; postgresql = pkgs.postgresql_15.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "pg-14"; postgresql = pkgs.postgresql_14.withPackages (p: [ p.postgis p.pg_safeupdate ]); } { name = "postgresql-14"; postgresql = pkgs.postgresql_14.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "pg-13"; postgresql = pkgs.postgresql_13.withPackages (p: [ p.postgis p.pg_safeupdate ]); } { name = "postgresql-13"; postgresql = pkgs.postgresql_13.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
{ name = "postgresql-12"; postgresql = pkgs.postgresql_12.withPackages (p: [ p.postgis p.pg_safeupdate ]); }
]; ];
haskellPackages = pkgs.haskell.packages."${compiler}";
# Dynamic derivation for PostgREST # Dynamic derivation for PostgREST
postgrest = pkgs.lib.pipe (haskellPackages.callCabal2nix name src { }) [ postgrest = pkgs.lib.pipe (pkgs.haskell.packages."${compiler}".callCabal2nix name src { }) [
# To allow ghc-datasize to be used. # To allow ghc-datasize to be used.
lib.disableLibraryProfiling lib.disableLibraryProfiling
# We are never going to use dynamic haskell libraries anyway. "Dynamic" refers to how # We are never going to use dynamic haskell libraries anyway. "Dynamic" refers to how
@@ -75,7 +75,7 @@ let
# Options passed to cabal in dev tools and tests # Options passed to cabal in dev tools and tests
devCabalOptions = devCabalOptions =
"-f dev --test-show-detail=direct"; "-f dev --test-show-detail=direct --disable-shared";
inherit (pkgs.haskell) lib; inherit (pkgs.haskell) lib;
in in
@@ -84,14 +84,9 @@ rec {
# Derivation for the PostgREST Haskell package, including the executable, # Derivation for the PostgREST Haskell package, including the executable,
# libraries and documentation. We disable running the test suite on Nix # libraries and documentation. We disable running the test suite on Nix
# builds, as they require a database to be set up. We split the binary # builds, as they require a database to be set up.
# into a separate output, so that the default distribution via flake.nix postgrestPackage =
# has a much smaller closure size. lib.dontCheck postgrest;
postgrestPackage = pkgs.lib.pipe postgrest [
lib.dontCheck
lib.enableSeparateBinOutput
(haskellPackages.generateOptparseApplicativeCompletions [ "postgrest" ])
];
# Profiled dynamic executable. # Profiled dynamic executable.
postgrestProfiled = pkgs.lib.pipe postgrestPackage [ postgrestProfiled = pkgs.lib.pipe postgrestPackage [
@@ -108,9 +103,6 @@ rec {
inherit (pkgs.haskell.packages."${compiler}") ghcWithPackages; inherit (pkgs.haskell.packages."${compiler}") ghcWithPackages;
}; };
# Used by CI on MacOS
inherit (pkgs) nix-build-uncached;
### Tools ### Tools
cabalTools = cabalTools =
@@ -127,18 +119,10 @@ rec {
docs = docs =
pkgs.callPackage nix/tools/docs.nix { }; pkgs.callPackage nix/tools/docs.nix { };
# Git tools.
gitTools =
pkgs.callPackage nix/tools/gitTools.nix { };
# Load testing tools. # Load testing tools.
loadtest = loadtest =
pkgs.callPackage nix/tools/loadtest.nix { inherit withTools; }; pkgs.callPackage nix/tools/loadtest.nix { inherit withTools; };
# Utility for updating the pinned version of Nixpkgs.
nixpkgsTools =
pkgs.callPackage nix/tools/nixpkgsTools.nix { };
# Scripts for publishing new releases. # Scripts for publishing new releases.
release = release =
pkgs.callPackage nix/tools/release.nix { }; pkgs.callPackage nix/tools/release.nix { };
+8 -8
View File
@@ -19,26 +19,26 @@ write from scratch.
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank"> <a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/supabase.svg"> <img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/neon.jpg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://www.euronodes.com/postgrest" target="_blank"> <a href="https://tembo.io/?utm_source=sponsor&utm_campaign=postgrest" target="_blank">
<img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/euronodes.svg"> <img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/tembo.png">
</a> </a>
</td> </td>
</tr> </tr>
<tr></tr> <tr></tr>
<tr> <tr>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://neon.tech/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://www.euronodes.com/postgrest" target="_blank">
<img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/neon.jpg"> <img width="296px" src="static/euronodes.svg">
</a> </a>
</td> </td>
<td align="center" valign="middle"> <td align="center" valign="middle">
<a href="https://www.bytebase.com/?utm_source=sponsor&utm_campaign=postgrest" target="_blank"> <a href="https://supabase.io?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage" target="_blank">
<img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/bytebase.svg"> <img width="296px" src="https://raw.githubusercontent.com/PostgREST/postgrest/main/static/supabase.svg">
</a> </a>
</td> </td>
</tr> </tr>
+3 -8
View File
@@ -40,14 +40,9 @@ database "PostgreSQL" {
:user: :user:
hexagon Proxy hexagon Proxy
:user: .r-> Proxy : request with JWT :user: .r-> Proxy
HTTPAPI <.l- Proxy HTTPAPI <.l- Proxy
hexagon ExternalAuth
ExternalAuth -u[hidden]- Proxy
:user: .r-> ExternalAuth : login
:user: <.r- ExternalAuth : JWT
:operator: .d-> HTTPADMIN :operator: .d-> HTTPADMIN
:operator: .d-> CLI :operator: .d-> CLI
@@ -56,8 +51,9 @@ PostgreSQL <.developer : "\t"
Listener -r.> "PostgreSQL" Listener -r.> "PostgreSQL"
"Connection Pool" -r.> "PostgreSQL" : "\t\t" "Connection Pool" -r.> "PostgreSQL" : "\t\t"
note bottom of Auth note bottom of Auth
Validates the JWT Authenticates the user request
end note end note
note bottom of ApiRequest note bottom of ApiRequest
@@ -76,7 +72,6 @@ note top of Listener
LISTEN session LISTEN session
end note end note
url of ExternalAuth is [[../explanations/external_auth.html]]
url of Admin is [[../references/admin_server.html#admin-server]] url of Admin is [[../references/admin_server.html#admin-server]]
url of API is [[../explanations/schema_isolation.html]] url of API is [[../explanations/schema_isolation.html]]
url of Auth is [[../references/auth.html#authn]] url of Auth is [[../references/auth.html#authn]]
+1 -1
View File
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 28 KiB

After

Width:  |  Height:  |  Size: 26 KiB

+1 -1
View File
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 29 KiB

After

Width:  |  Height:  |  Size: 27 KiB

+2 -2
View File
@@ -12,6 +12,7 @@
# All configuration values have a default; values that are commented out # All configuration values have a default; values that are commented out
# serve to show the default. # serve to show the default.
import sys
import os import os
# If extensions (or modules to document with autodoc) are in another directory, # If extensions (or modules to document with autodoc) are in another directory,
@@ -49,7 +50,7 @@ source_suffix = ".rst"
master_doc = "index" master_doc = "index"
# This is overriden by readthedocs with the version tag anyway # This is overriden by readthedocs with the version tag anyway
version = "14" version = "13.0"
# To avoid repetition in <title> we set this to an empty string. # To avoid repetition in <title> we set this to an empty string.
release = "" release = ""
@@ -299,7 +300,6 @@ linkcheck_ignore = [
# 403 only in CI / GitHub Actions # 403 only in CI / GitHub Actions
r"https://www.patreon.com/postgrest", r"https://www.patreon.com/postgrest",
r"https://blog.frankel.ch/poor-man-api", r"https://blog.frankel.ch/poor-man-api",
r"https://www.cybertec-postgresql.com/.*",
# Odd SSL error # Odd SSL error
r"https://www.dripdepot.com", r"https://www.dripdepot.com",
r"https://www.euronodes.com", r"https://www.euronodes.com",
+1 -1
View File
@@ -34,7 +34,7 @@ Templates
Example Apps Example Apps
------------ ------------
* `archtika <https://github.com/thiloho/archtika>`_ - self-hosted CMS * `archtika <https://github.com/thiloho/archtika>`_ - selfhosted CMS
* `delibrium-postgrest <https://gitlab.com/delibrium/delibrium-postgrest/>`_ - example school API and front-end in Vue.js * `delibrium-postgrest <https://gitlab.com/delibrium/delibrium-postgrest/>`_ - example school API and front-end in Vue.js
* `ETH-transactions-storage <https://github.com/Adamant-im/ETH-transactions-storage>`_ - indexer for Ethereum to get transaction list by ETH address * `ETH-transactions-storage <https://github.com/Adamant-im/ETH-transactions-storage>`_ - indexer for Ethereum to get transaction list by ETH address
* `general <https://github.com/PierreRochard/general>`_ - example auth back-end * `general <https://github.com/PierreRochard/general>`_ - example auth back-end
+11 -11
View File
@@ -31,58 +31,58 @@ This section talks briefly about various important modules.
Main Main
---- ----
The starting point of the program is `Main.hs <https://github.com/PostgREST/postgrest/blob/v14/main/Main.hs>`_. The starting point of the program is `Main.hs <https://github.com/PostgREST/postgrest/blob/main/main/Main.hs>`_.
CLI CLI
--- ---
Main then calls `CLI.hs <https://github.com/PostgREST/postgrest/blob/v14/src/PostgREST/CLI.hs>`_, which is in charge of :ref:`cli`. Main then calls `CLI.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/CLI.hs>`_, which is in charge of :ref:`cli`.
App App
--- ---
`App.hs <https://github.com/PostgREST/postgrest/blob/v14/src/PostgREST/App.hs>`_ is then in charge of composing the different modules. `App.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/App.hs>`_ is then in charge of composing the different modules.
Auth Auth
---- ----
`Auth.hs <https://github.com/PostgREST/postgrest/blob/v14/src/PostgREST/Auth.hs>`_ is in charge of :ref:`authn`. `Auth.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Auth.hs>`_ is in charge of :ref:`authn`.
Api Request Api Request
----------- -----------
`ApiRequest.hs <https://github.com/PostgREST/postgrest/blob/v14/src/PostgREST/ApiRequest.hs>`_ is in charge of parsing the URL query string (following PostgREST syntax), the request headers, and the request body. `ApiRequest.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/ApiRequest.hs>`_ is in charge of parsing the URL query string (following PostgREST syntax), the request headers, and the request body.
A request might be rejected at this level if it's invalid. For example when providing an unknown media type to PostgREST or using an unknown HTTP method. A request might be rejected at this level if it's invalid. For example when providing an unknown media type to PostgREST or using an unknown HTTP method.
Plan Plan
---- ----
Using the Schema Cache, `Plan.hs <https://github.com/PostgREST/postgrest/blob/v14/src/PostgREST/Plan.hs>`_ generates an internal AST, filling out-of-band SQL details (like an ``ON CONFLICT (pk)`` clause) required to complete the user request. Using the Schema Cache, `Plan.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Plan.hs>`_ generates an internal AST, filling out-of-band SQL details (like an ``ON CONFLICT (pk)`` clause) required to complete the user request.
A request might be rejected at this level if it's invalid. For example when doing resource embedding on a nonexistent resource. A request might be rejected at this level if it's invalid. For example when doing resource embedding on a nonexistent resource.
Query Query
----- -----
`Query.hs <https://github.com/PostgREST/postgrest/blob/v14/src/PostgREST/Query.hs>`_ generates the SQL queries (parametrized and prepared) required to satisfy the user request. `Query.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Query.hs>`_ generates the SQL queries (parametrized and prepared) required to satisfy the user request.
Only at this stage a connection from the pool might be used. Only at this stage a connection from the pool might be used.
Schema Cache Schema Cache
------------ ------------
`SchemaCache.hs <https://github.com/PostgREST/postgrest/blob/v14/src/PostgREST/SchemaCache.hs>`_ is in charge of :ref:`schema_cache`. `SchemaCache.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/SchemaCache.hs>`_ is in charge of :ref:`schema_cache`.
Config Config
------ ------
`Config.hs <https://github.com/PostgREST/postgrest/blob/v14/src/PostgREST/Config.hs>`_ is in charge of :ref:`configuration`. `Config.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Config.hs>`_ is in charge of :ref:`configuration`.
Admin Admin
----- -----
`Admin.hs <https://github.com/PostgREST/postgrest/blob/v14/src/PostgREST/Admin.hs>`_ is in charge of the :ref:`admin_server`. `Admin.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Admin.hs>`_ is in charge of the :ref:`admin_server`.
HTTP HTTP
---- ----
@@ -92,4 +92,4 @@ The HTTP server is provided by `Warp <https://aosabook.org/en/posa/warp.html>`_.
Listener Listener
-------- --------
`Listener.hs <https://github.com/PostgREST/postgrest/blob/v14/src/PostgREST/Listener.hs>`_ is in charge of the :ref:`listener`. `Listener.hs <https://github.com/PostgREST/postgrest/blob/main/src/PostgREST/Listener.hs>`_ is in charge of the :ref:`listener`.
+1 -1
View File
@@ -13,7 +13,7 @@ A role can be thought of as either a database user, or a group of database users
Roles for Each Web User Roles for Each Web User
----------------------- -----------------------
PostgREST can accommodate either viewpoint. If you treat a role as a single user then :ref:`user_impersonation` does most of what you need. When an authenticated user makes a request PostgREST will switch into the database role for that user, which in addition to restricting queries, is available to SQL through the :code:`current_user` variable. PostgREST can accommodate either viewpoint. If you treat a role as a single user then the :ref:`jwt_impersonation` does most of what you need. When an authenticated user makes a request PostgREST will switch into the database role for that user, which in addition to restricting queries, is available to SQL through the :code:`current_user` variable.
You can use row-level security to flexibly restrict visibility and access for the current user. Here is an `example <https://www.enterprisedb.com:443/blog/application-users-vs-row-level-security>`_ from Tomas Vondra, a chat table storing messages sent between users. Users can insert rows into it to send messages to other users, and query it to see messages sent to them by other users. You can use row-level security to flexibly restrict visibility and access for the current user. Here is an `example <https://www.enterprisedb.com:443/blog/application-users-vs-row-level-security>`_ from Tomas Vondra, a chat table storing messages sent between users. Users can insert rows into it to send messages to other users, and query it to see messages sent to them by other users.
+1 -6
View File
@@ -16,7 +16,7 @@ Supported PostgreSQL versions
============================= =============================
=============== ================================= =============== =================================
**Supported** PostgreSQL >= 13 **Supported** PostgreSQL >= 12
=============== ================================= =============== =================================
PostgREST works with all PostgreSQL versions still `officially supported <https://www.postgresql.org/support/versioning/>`_. PostgREST works with all PostgreSQL versions still `officially supported <https://www.postgresql.org/support/versioning/>`_.
@@ -75,12 +75,8 @@ You can get the `official PostgREST Docker image <https://hub.docker.com/r/postg
.. code-block:: bash .. code-block:: bash
# pull the latest version
docker pull postgrest/postgrest docker pull postgrest/postgrest
# to pull a particular version, use one of the versions on https://hub.docker.com/r/postgrest/postgrest/tags
docker pull postgrest/postgrest:<version>
To configure the container image, use :ref:`env_variables_config`. To configure the container image, use :ref:`env_variables_config`.
There are two ways to run the PostgREST container: with an existing external database, or through docker-compose. There are two ways to run the PostgREST container: with an existing external database, or through docker-compose.
@@ -146,7 +142,6 @@ To avoid having to install the database at all, you can run both it and the serv
ports: ports:
- "3000:3000" - "3000:3000"
environment: environment:
PGRST_SERVER_HOST: 0.0.0.0 # necessary for `postgrest --ready` flag to work
PGRST_DB_URI: postgres://app_user:password@db:5432/app_db PGRST_DB_URI: postgres://app_user:password@db:5432/app_db
PGRST_OPENAPI_SERVER_PROXY_URI: http://127.0.0.1:3000 PGRST_OPENAPI_SERVER_PROXY_URI: http://127.0.0.1:3000
depends_on: depends_on:
@@ -43,7 +43,7 @@ As in :ref:`sql_user_management`, we create the :code:`pgcrypto` and :code:`pgjw
CREATE EXTENSION pgcrypto WITH SCHEMA ext_pgcrypto; CREATE EXTENSION pgcrypto WITH SCHEMA ext_pgcrypto;
Concerning the `pgjwt extension <https://github.com/michelp/pgjwt>`_, please cf. to :ref:`jwt-from-sql`. Concerning the `pgjwt extension <https://github.com/michelp/pgjwt>`_, please cf. to :ref:`client_auth`.
.. code-block:: postgres .. code-block:: postgres
@@ -157,7 +157,7 @@ Here we use the username instead of the email address to identify a user.
Logins Logins
~~~~~~ ~~~~~~
As described in :ref:`jwt-from-sql`, we'll create a JWT token inside our login function. Note that you'll need to adjust the secret key which is hard-coded in this example to a secure (at least thirty-two character) secret of your choosing. As described in :ref:`client_auth`, we'll create a JWT token inside our login function. Note that you'll need to adjust the secret key which is hard-coded in this example to a secure (at least thirty-two character) secret of your choosing.
.. code-block:: postgres .. code-block:: postgres
+1 -3
View File
@@ -3,7 +3,7 @@
SQL User Management SQL User Management
=================== ===================
As mentioned on :ref:`jwt_generation`, an external service can provide user management and coordinate with the PostgREST server using JWT. It's also possible to support logins entirely through SQL. It's a fair bit of work, so get ready. As mentioned on :ref:`jwt_generation`, an external service can provide user management and coordinate with the PostgREST server using JWT. Its also possible to support logins entirely through SQL. Its a fair bit of work, so get ready.
Storing Users and Passwords Storing Users and Passwords
--------------------------- ---------------------------
@@ -110,8 +110,6 @@ Then, add ``db-anon-role`` to the configuration file to allow anonymous requests
db-anon-role = "anon" db-anon-role = "anon"
.. _jwt-from-sql:
JWT from SQL JWT from SQL
~~~~~~~~~~~~ ~~~~~~~~~~~~
@@ -318,6 +318,144 @@ You can insert a new product using a JSON object for the ``extra_info`` column:
To query and filter the data see :ref:`json_columns` for a complete reference. To query and filter the data see :ref:`json_columns` for a complete reference.
.. _ww_postgis:
PostGIS
-------
You can use the string representation for `PostGIS <https://postgis.net/>`_ data types such as ``geometry`` or ``geography`` (you need to `install PostGIS <https://postgis.net/documentation/getting_started/>`_ first).
.. code-block:: postgres
-- Activate the postgis module in the current database
create extension if not exists postgis;
create table coverage (
id int primary key,
name text unique,
area geometry
);
To add areas in polygon format, you can use string representation:
.. code-block:: bash
curl "http://localhost:3000/coverage" \
-X POST -H "Content-Type: application/json" \
-d @- << EOF
[
{ "id": 1, "name": "small", "area": "SRID=4326;POLYGON((0 0, 1 0, 1 1, 0 1, 0 0))" },
{ "id": 2, "name": "big", "area": "SRID=4326;POLYGON((0 0, 10 0, 10 10, 0 10, 0 0))" }
]
EOF
Now, when you request the information, PostgREST will automatically cast the ``area`` column into a ``Polygon`` geometry type. Although this is useful, you may need the whole output to be in `GeoJSON <https://geojson.org/>`_ format out of the box, which can be done by including the ``Accept: application/geo+json`` in the request. This will work for PostGIS versions 3.0.0 and up and will return the output as a `FeatureCollection Object <https://www.rfc-editor.org/rfc/rfc7946#section-3.3>`_:
.. code-block:: bash
curl "http://localhost:3000/coverage" \
-H "Accept: application/geo+json"
.. code-block:: json
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[1,0],[1,1],[0,1],[0,0]]
]
},
"properties": {
"id": 1,
"name": "small"
}
},
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[10,0],[10,10],[0,10],[0,0]]
]
},
"properties": {
"id": 2,
"name": "big"
}
}
]
}
If you need to add an extra property, like the area in square units by using ``st_area(area)``, you could add a generated column to the table and it will appear in the ``properties`` key of each ``Feature``.
.. code-block:: postgres
alter table coverage
add square_units double precision generated always as ( st_area(area) ) stored;
In the case that you are using older PostGIS versions, then creating a function is your best option:
.. code-block:: postgres
create or replace function coverage_geo_collection() returns json as $$
select
json_build_object(
'type', 'FeatureCollection',
'features', json_agg(
json_build_object(
'type', 'Feature',
'geometry', st_AsGeoJSON(c.area)::json,
'properties', json_build_object('id', c.id, 'name', c.name)
)
)
)
from coverage c;
$$ language sql;
Now this query will return the same results:
.. code-block:: bash
curl "http://localhost:3000/rpc/coverage_geo_collection"
.. code-block:: json
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[1,0],[1,1],[0,1],[0,0]]
]
},
"properties": {
"id": 1,
"name": "small"
}
},
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[10,0],[10,10],[0,10],[0,0]]
]
},
"properties": {
"id": 2,
"name": "big"
}
}
]
}
Ranges Ranges
------ ------
@@ -471,20 +609,3 @@ You can use other comparative filters and also all the `PostgreSQL special date/
"due_date": "2022-02-27T06:00:00-05:00" "due_date": "2022-02-27T06:00:00-05:00"
} }
] ]
.. raw:: html
<script type="text/javascript">
let hash = window.location.hash;
const redirects = {
// PostGIS
'#postgis': '../integrations/postgis.html#postgis',
};
let willRedirectTo = redirects[hash];
if (willRedirectTo) {
window.location.href = willRedirectTo;
}
</script>
+23 -54
View File
@@ -38,28 +38,6 @@ Sponsors
.. image:: ../static/cybertec.svg .. image:: ../static/cybertec.svg
:target: https://www.cybertec-postgresql.com/en/?utm_source=postgrest.org&utm_medium=referral&utm_campaign=postgrest :target: https://www.cybertec-postgresql.com/en/?utm_source=postgrest.org&utm_medium=referral&utm_campaign=postgrest
.. container:: img-dark
.. image:: ../static/supabase-dark.svg
:target: https://supabase.com/?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage
.. container:: img-light
.. image:: ../static/supabase.svg
:target: https://supabase.com/?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage
.. container:: img-dark
.. image:: ../static/euronodes.svg
:target: https://www.euronodes.com/postgrest
.. container:: img-light
.. image:: ../static/euronodes.svg
:target: https://www.euronodes.com/postgrest
|
.. container:: img-dark .. container:: img-dark
.. image:: ../static/neon-dark.jpg .. image:: ../static/neon-dark.jpg
@@ -70,15 +48,30 @@ Sponsors
.. image:: ../static/neon.jpg .. image:: ../static/neon.jpg
:target: https://neon.com/?utm_source=sponsor&utm_campaign=postgrest :target: https://neon.com/?utm_source=sponsor&utm_campaign=postgrest
.. image:: ../static/tembo.png
:target: https://www.tembo.io/?utm_source=sponsor&utm_campaign=postgrest
|
.. container:: img-dark .. container:: img-dark
.. image:: ../static/bytebase-dark.svg .. image:: ../static/euronodes.svg
:target: https://www.bytebase.com/?utm_source=sponsor&utm_campaign=postgrest :target: https://www.euronodes.com/postgrest
.. container:: img-light .. container:: img-light
.. image:: ../static/bytebase.svg .. image:: ../static/euronodes.svg
:target: https://www.bytebase.com/?utm_source=sponsor&utm_campaign=postgrest :target: https://www.euronodes.com/postgrest
.. container:: img-dark
.. image:: ../static/supabase-dark.svg
:target: https://supabase.com/?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage
.. container:: img-light
.. image:: ../static/supabase.svg
:target: https://supabase.com/?utm_source=postgrest%20backers&utm_medium=open%20source%20partner&utm_campaign=postgrest%20backers%20github&utm_term=homepage
.. The static/empty.png(created with `convert -size 320x95 xc:#fcfcfc empty.png`) is an ugly workaround .. The static/empty.png(created with `convert -size 320x95 xc:#fcfcfc empty.png`) is an ugly workaround
to create space and center the logos. It's not easy to layout with restructuredText. to create space and center the logos. It's not easy to layout with restructuredText.
@@ -113,17 +106,10 @@ Getting Support
The project has a friendly and growing community. For discussions, use the Github `discussions page <https://github.com/PostgREST/postgrest/discussions>`_. You can also report or search for bugs/features on the Github `issues <https://github.com/PostgREST/postgrest/issues>`_ page. The project has a friendly and growing community. For discussions, use the Github `discussions page <https://github.com/PostgREST/postgrest/discussions>`_. You can also report or search for bugs/features on the Github `issues <https://github.com/PostgREST/postgrest/issues>`_ page.
Releases Release Notes
-------- -------------
PostgREST follows ``MAJOR.PATCH`` two-part versioning: The release notes are published on `PostgREST's GitHub release page <https://github.com/PostgREST/postgrest/releases>`_.
- ``MAJOR``: feature release, may deprecate or remove things.
- ``PATCH``: fix/security release only; no features, no behavior changes.
Starting from ``v14.0``, only even-numbered MAJOR versions will be released, reserving odd-numbered MAJOR versions for development.
All the releases are published on `PostgREST's GitHub release page <https://github.com/PostgREST/postgrest/releases>`_.
Tutorials Tutorials
--------- ---------
@@ -271,21 +257,4 @@ Testimonials
Contributing Contributing
------------ ------------
Please see the `Contributing guidelines <https://github.com/PostgREST/postgrest/blob/main/CONTRIBUTING.md>`_ in the main PostgREST repository. Please see the `Contributing guidelines <https://github.com/PostgREST/postgrest/blob/main/.github/CONTRIBUTING.md>`_ in the main PostgREST repository.
.. raw:: html
<script type="text/javascript">
let hash = window.location.hash;
const redirects = {
// Tables and Views
'#release-notes': '#releases',
};
let willRedirectTo = redirects[hash];
if (willRedirectTo) {
window.location.href = willRedirectTo;
}
</script>
-154
View File
@@ -1,154 +0,0 @@
.. _ww_postgis:
PostGIS
=======
To work with `PostGIS <https://postgis.net/>`_ data types such as ``geometry`` or ``geography``, you'll need to `install PostGIS <https://postgis.net/documentation/getting_started/>`_ first.
.. code-block:: postgres
-- Activate the postgis module in the current database
create extension if not exists postgis;
create table coverage (
id int primary key,
name text unique,
area geometry
);
insert into coverage (id, name, area) values
(1, 'small', ST_GeomFromText('POLYGON((0 0, 1 0, 1 1, 0 1, 0 0))',4326)),
(2, 'big', ST_GeomFromText('POLYGON((0 0, 10 0, 10 10, 0 10, 0 0))', 4326);
.. _application/geo+json:
``application/geo+json``
------------------------
PostgREST supports the `standard <https://www.iana.org/assignments/media-types/application/geo+json>`_ ``application/geo+json`` media type which can be used to get the output in `GeoJSON <https://geojson.org/>`_ format. This will work for PostGIS versions 3.0.0 and up and will return the output as a `FeatureCollection Object <https://www.rfc-editor.org/rfc/rfc7946#section-3.3>`_:
.. code-block:: bash
curl "http://localhost:3000/coverage" \
-H "Accept: application/geo+json"
.. code-block:: json
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[1,0],[1,1],[0,1],[0,0]]
]
},
"properties": {
"id": 1,
"name": "small"
}
},
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[10,0],[10,10],[0,10],[0,0]]
]
},
"properties": {
"id": 2,
"name": "big"
}
}
]
}
Using generated columns
-----------------------
If you need to add an extra property, like the area in square units by using ``st_area(area)``, you could add a generated column to the table and it will appear in the ``properties`` key of each ``Feature``.
.. code-block:: postgres
alter table coverage
add square_units double precision generated always as ( st_area(area) ) stored;
In the case that you are using older PostGIS versions, then creating a function is your best option:
.. code-block:: postgres
create or replace function coverage_geo_collection() returns json as $$
select
json_build_object(
'type', 'FeatureCollection',
'features', json_agg(
json_build_object(
'type', 'Feature',
'geometry', st_AsGeoJSON(c.area)::json,
'properties', json_build_object('id', c.id, 'name', c.name)
)
)
)
from coverage c;
$$ language sql;
Now this query will return the same results:
.. code-block:: bash
curl "http://localhost:3000/rpc/coverage_geo_collection"
.. code-block:: json
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[1,0],[1,1],[0,1],[0,0]]
]
},
"properties": {
"id": 1,
"name": "small"
}
},
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[[0,0],[10,0],[10,10],[0,10],[0,0]]
]
},
"properties": {
"id": 2,
"name": "big"
}
}
]
}
Using string representation
---------------------------
To insert areas in polygon format, you can use string representation:
.. code-block:: bash
curl "http://localhost:3000/coverage" \
-X POST -H "Content-Type: application/json" \
-d @- << EOF
[
{ "id": 3, "name": "strip", "area": "SRID=4326;POLYGON((0 0, 50 0, 50 2, 0 2, 0 0))" },
{ "id": 4, "name": "diamond", "area": "SRID=4326;POLYGON((5 0, 10 5, 5 10, 0 5, 5 0))" }
]
EOF
PostgREST will automatically cast the ``area`` column into a ``Polygon`` geometry type.
-2
View File
@@ -194,5 +194,3 @@ Websockets
webuser webuser
wfts wfts
www www
debouncing
deduplicates
@@ -173,4 +173,4 @@ Domain Representations avoid all the above drawbacks. Their only drawback is tha
Why not create a `base type <https://www.postgresql.org/docs/current/sql-createtype.html#id-1.9.3.94.5.8>`_ instead? ``CREATE TYPE app_uuid (INTERNALLENGTH = 22, INPUT = app_uuid_parser, OUTPUT = app_uuid_formatter)``. Why not create a `base type <https://www.postgresql.org/docs/current/sql-createtype.html#id-1.9.3.94.5.8>`_ instead? ``CREATE TYPE app_uuid (INTERNALLENGTH = 22, INPUT = app_uuid_parser, OUTPUT = app_uuid_formatter)``.
Creating base types need superuser, which is restricted on cloud hosted databases. Additionally this way lets "how the data is presented" dictate "how the data is stored" which would be backwards. Creating base types need superuser, which is restricted on cloud hosted databases. Additionally this way lets how the data is presented dictate how the data is stored which would be backwards.
+9 -13
View File
@@ -143,7 +143,7 @@ Since the table name is plural, we can be more accurate by making it singular wi
One-to-many relationships One-to-many relationships
------------------------- -------------------------
The **foreign key reference** establishes the inverse one-to-many relationship. In this case, ``films`` returns as a JSON array because of the "to-many" end. The **foreign key reference** establishes the inverse one-to-many relationship. In this case, ``films`` returns as a JSON array because of the to-many end.
.. code-block:: bash .. code-block:: bash
@@ -251,12 +251,6 @@ Computed Relationships
You can manually define relationships by using functions. This is useful for database objects that can't define foreign keys, like `Foreign Data Wrappers <https://wiki.postgresql.org/wiki/Foreign_data_wrappers>`_. You can manually define relationships by using functions. This is useful for database objects that can't define foreign keys, like `Foreign Data Wrappers <https://wiki.postgresql.org/wiki/Foreign_data_wrappers>`_.
Computed relationships have good performance as their intended design enable `function inlining <https://wiki.postgresql.org/wiki/Inlining_of_SQL_functions#Inlining_conditions_for_table_functions>`_.
.. important::
- Always use ``SETOF`` when creating computed relationships. Functions can return a table without using ``SETOF``, but bear in mind that PostgreSQL will not inline them. e.g. ``RETURNS <table_name>`` is not inlinable.
Assuming there's a foreign table ``premieres`` that we want to relate to ``films``. Assuming there's a foreign table ``premieres`` that we want to relate to ``films``.
.. code-block:: postgres .. code-block:: postgres
@@ -289,10 +283,6 @@ The name of the function ``film`` is arbitrary and can be used to do the embeddi
".." ".."
] ]
.. warning::
- Make sure to correctly label the ``to-one`` part of the relationship. When using the ``ROWS 1`` estimation, PostgREST will expect a single row to be returned. If that is not the case, it will unnest the embedding and return repeated values for the top level resource.
Now let's define the opposite one-to-many relationship. Now let's define the opposite one-to-many relationship.
.. code-block:: postgres .. code-block:: postgres
@@ -341,6 +331,12 @@ Thanks to overloaded functions, you can use the same function name for different
Computed relationships have good performance as their intended design enable `function inlining <https://wiki.postgresql.org/wiki/Inlining_of_SQL_functions#Inlining_conditions_for_table_functions>`_. Computed relationships have good performance as their intended design enable `function inlining <https://wiki.postgresql.org/wiki/Inlining_of_SQL_functions#Inlining_conditions_for_table_functions>`_.
.. warning::
- Always use ``SETOF`` when creating computed relationships. Functions can return a table without using ``SETOF``, but bear in mind that PostgreSQL will not inline them.
- Make sure to correctly label the ``to-one`` part of the relationship. When using the ``ROWS 1`` estimation, PostgREST will expect a single row to be returned. If that is not the case, it will unnest the embedding and return repeated values for the top level resource.
.. _embed_disamb: .. _embed_disamb:
.. _target_disamb: .. _target_disamb:
.. _hint_disamb: .. _hint_disamb:
@@ -1210,7 +1206,7 @@ Spread on resources forming :ref:`one-to-many <one-to-many>` and :ref:`many-to-m
} }
] ]
Note that ``films`` is no longer an array of objects, unlike regularly embedding :ref:`one-to-many`. The embedded columns become arrays and they're correlated-in the above result, we can say that "Pulp Fiction" premiered in 1994 and "Reservoir Dogs" in 1992. Note that ``films`` is no longer an array of objects, unlike regularly embedding :ref:`one-to-many`. The embedded columns become arrays and they're correlatedin the above result, we can say that "Pulp Fiction" premiered in 1994 and "Reservoir Dogs" in 1992.
Order in spread to-many Order in spread to-many
~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
@@ -1306,7 +1302,7 @@ You can use multiple spreads at any level. For example, let's spread ``technical
Note that: Note that:
- All the ``film_*`` arrays are correlated-"Reservoir Dogs" premiered in 1992, its runtime is 1:39:00 and it has the following characters: ``[ "Mr. Pink", "Mr. White" ]``. - All the ``film_*`` arrays are correlated"Reservoir Dogs" premiered in 1992, its runtime is 1:39:00 and it has the following characters: ``[ "Mr. Pink", "Mr. White" ]``.
- The ``film_*`` arrays are ordered by ``year`` (due to ``films.order=year``). - The ``film_*`` arrays are ordered by ``year`` (due to ``films.order=year``).
- The bottom level array ``film_characters`` is ordered (due to ``films.roles.order=character``). - The bottom level array ``film_characters`` is ordered (due to ``films.roles.order=character``).
@@ -51,7 +51,7 @@ Builtin handlers are offered for common standard media types.
* ``text/csv`` and ``application/json``, for all API endpoints. See :ref:`tables_views` and :ref:`functions`. * ``text/csv`` and ``application/json``, for all API endpoints. See :ref:`tables_views` and :ref:`functions`.
* ``application/openapi+json``, for the root endpoint. See :ref:`open-api`. * ``application/openapi+json``, for the root endpoint. See :ref:`open-api`.
* ``application/geo+json``, see :ref:`application/geo+json`. * ``application/geo+json``, see :ref:`ww_postgis`.
* ``*/*``, resolves to ``application/json`` for API endpoints and to ``application/openapi+json`` for the root endpoint. * ``*/*``, resolves to ``application/json`` for API endpoints and to ``application/openapi+json`` for the root endpoint.
The following vendor media types handlers are also supported. The following vendor media types handlers are also supported.
-9
View File
@@ -217,15 +217,6 @@ This allows using the ``fts`` operator on ``text`` and ``json`` types out of the
curl --get "http://localhost:3000/people" \ curl --get "http://localhost:3000/people" \
-d "my_json_column=not.phfts(english).The%20Fat%20Cats" -d "my_json_column=not.phfts(english).The%20Fat%20Cats"
.. important::
To ensure this operation is fast, you need to create an index on the expression:
.. code-block:: postgres
CREATE INDEX idx_people_col ON people
USING GIN (to_tsvector('french', my_text_column));
.. _v_filter: .. _v_filter:
Vertical Filtering Vertical Filtering
+1 -1
View File
@@ -14,7 +14,7 @@ Custom Queries
The PostgREST URL grammar limits the kinds of queries clients can perform. It prevents arbitrary, potentially poorly constructed and slow client queries. It's good for quality of service, but means database administrators must create custom views and functions to provide richer endpoints. The most common causes for custom endpoints are The PostgREST URL grammar limits the kinds of queries clients can perform. It prevents arbitrary, potentially poorly constructed and slow client queries. It's good for quality of service, but means database administrators must create custom views and functions to provide richer endpoints. The most common causes for custom endpoints are
* SET operators like `UNION, INTERSECT and EXCEPT <https://www.postgresql.org/docs/current/queries-union.html>`_. * Table unions
* More complicated joins than those provided by :ref:`resource_embedding`. * More complicated joins than those provided by :ref:`resource_embedding`.
* Geo-spatial queries that require an argument, like "points near (lat,lon)" * Geo-spatial queries that require an argument, like "points near (lat,lon)"
+42 -80
View File
@@ -31,7 +31,7 @@ The authenticator role is used for connecting to the database and should be conf
.. _user_impersonation: .. _user_impersonation:
User Impersonation User Impersonation
~~~~~~~~~~~~~~~~~~ ------------------
The picture below shows how the server handles authentication. If auth succeeds, it switches into the user role specified by the request, otherwise it switches into the anonymous role (if it's set in :ref:`db-anon-role`). The picture below shows how the server handles authentication. If auth succeeds, it switches into the user role specified by the request, otherwise it switches into the anonymous role (if it's set in :ref:`db-anon-role`).
@@ -43,13 +43,12 @@ This role switching mechanism is called **user impersonation**. In PostgreSQL it
The impersonated roles will have their settings applied. See :ref:`impersonated_settings`. The impersonated roles will have their settings applied. See :ref:`impersonated_settings`.
.. _jwt_auth: .. _jwt_impersonation:
JWT Authentication JWT-Based User Impersonation
------------------ ----------------------------
We use `JSON Web Tokens <https://datatracker.ietf.org/doc/html/rfc7519/>`_ to authenticate API requests, this allows us to be stateless and not require database lookups for verification. We use `JSON Web Tokens <https://jwt.io/>`_ to authenticate API requests, this allows us to be stateless and not require database lookups for verification. As you'll recall a JWT contains a list of cryptographically signed claims. All claims are allowed but PostgREST cares specifically about a claim called role.
As you'll recall a JWT contains a list of cryptographically signed claims. All claims are allowed but PostgREST cares specifically about a claim called role (configurable with :ref:`jwt_role_extract`).
.. code:: json .. code:: json
@@ -73,10 +72,17 @@ Note that the database administrator must allow the authenticator role to switch
If the client included no JWT (or one without a role claim) then PostgREST switches into the anonymous role. The database administrator must set the anonymous role permissions correctly to prevent anonymous users from seeing or changing things they shouldn't. If the client included no JWT (or one without a role claim) then PostgREST switches into the anonymous role. The database administrator must set the anonymous role permissions correctly to prevent anonymous users from seeing or changing things they shouldn't.
.. _bearer_auth: .. _jwt_generation:
Bearer Authentication JWT Generation
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
You can create a valid JWT either from inside your database (see :ref:`sql_user_management`) or via an external service (see :ref:`external_auth`).
.. _client_auth:
Client Auth
~~~~~~~~~~~
To make an authenticated request the client must include an :code:`Authorization` HTTP header with the value :code:`Bearer <jwt>`. For instance: To make an authenticated request the client must include an :code:`Authorization` HTTP header with the value :code:`Bearer <jwt>`. For instance:
@@ -87,29 +93,24 @@ To make an authenticated request the client must include an :code:`Authorization
The ``Bearer`` header value can be used with or without capitalization(``bearer``). The ``Bearer`` header value can be used with or without capitalization(``bearer``).
.. _jwt_generation: .. _jwt_caching:
JWT Generation JWT Caching
~~~~~~~~~~~~~~ -----------
You can create a valid JWT either from inside your database (see :ref:`sql_user_management`) or via an external service (see :ref:`external_auth`). PostgREST validates ``JWTs`` on every request. We can cache ``JWTs`` to avoid this performance overhead.
.. _jwt_signature: To enable JWT caching, the config :code:`jwt-cache-max-lifetime` is to be set. It is the maximum number of seconds for which the cache stores the JWT validation results. The cache uses the :code:`exp` claim to set the cache entry lifetime. If the JWT does not have an :code:`exp` claim, it uses the config value. See :ref:`jwt-cache-max-lifetime` for more details.
JWT Signature Verification .. note::
--------------------------
PostgREST supports both symmetric and asymmetric keys for verifying the signature of the token. You can use the :ref:`server-timing_header` to see the effect of JWT caching.
Symmetric Keys Symmetric Keys
~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
In the case of symmetric cryptography the signer and verifier share the same secret passphrase, which can be configured with :ref:`jwt-secret`. Each token is cryptographically signed with a secret key. In the case of symmetric cryptography the signer and verifier share the same secret passphrase, which can be configured with :ref:`jwt-secret`.
If it is set to a simple string then PostgREST interprets it as an HMAC-SHA256 passphrase. If it is set to a simple string value like “reallyreallyreallyreallyverysafe” then PostgREST interprets it as an HMAC-SHA256 passphrase.
.. code-block:: ini
jwt-secret = "reallyreallyreallyreallyverysafe"
.. _asym_keys: .. _asym_keys:
@@ -155,39 +156,39 @@ You can specify the literal value as we saw earlier, or reference a filename to
jwt-secret = "@rsa.jwk.pub" jwt-secret = "@rsa.jwk.pub"
``kid`` verification JWK ``kid`` validation
^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^
PostgREST has built-in verification of the `key ID parameter <https://www.rfc-editor.org/rfc/rfc7517#section-4.5>`_, useful when working with a JSON Web Key Set. PostgREST has built-in validation of the `key ID parameter <https://www.rfc-editor.org/rfc/rfc7517#section-4.5>`_, useful when working with a JWK Set.
It goes as follows: It goes as follows:
- If the JWT contains a ``kid`` parameter, then PostgREST will look for the JSON Web Key in the :ref:`jwt-secret`. - If the JWT contains a ``kid`` parameter, then PostgREST will look for the JWK in the :ref:`jwt-secret`.
+ If no key has a matching ``kid`` (or if they don't have one defined), the token will be rejected with a :ref:`401 Unauthorized <pgrst301>` error. + If no JWK matches the same ``kid`` value (or if they do not have a ``kid``), then the token will be rejected with a :ref:`401 Unauthorized <pgrst301>` error.
+ If a key matches the ``kid`` value then it will validate the token against that key accordingly. + If a JWK matches the ``kid`` value then it will validate the token against that JWK accordingly.
- If the JWT doesn't have a ``kid``, PostgREST will try each key in the :ref:`jwt-secret` one by one until it finds one that works. - If the JWT does not have a ``kid`` parameter, then PostgREST will validate the token against each JWK in the :ref:`jwt-secret`.
.. _jwt_claims_validation: .. _jwt_claims_validation:
JWT Claims Validation JWT Claims Validation
--------------------- ~~~~~~~~~~~~~~~~~~~~~
Time-Based claims validation PostgREST honors the following `JWT claims <https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.4>`_:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The time-based JWT claims specified in `RFC 7519 <https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.4>`_ are validated:
- ``exp`` Expiration Time - ``exp`` Expiration Time
- ``iat`` Issued At - ``iat`` Issued At
- ``nbf`` Not Before - ``nbf`` Not Before
- ``aud`` :ref:`Audience <jwt_aud_validation>`
We allow a 30-second clock skew when validating the above claims. In other words, we give an extra 30 seconds before the JWT is rejected if there is a slight discrepancy in the timestamps. .. note::
PostgREST allows for a 30-second clock skew when validating the ``exp``, ``iat`` and ``nbf`` claims.
In other words, it gives an extra 30 seconds before the token is rejected if there is a slight discrepancy in the timestamps.
.. _jwt_aud: .. _jwt_aud_validation:
``aud`` validation JWT ``aud`` Claim Validation
~~~~~~~~~~~~~~~~~~ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PostgREST has built-in validation of the `JWT audience claim <https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.3>`_. PostgREST has built-in validation of the `JWT audience claim <https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.3>`_.
It works this way: It works this way:
@@ -200,29 +201,10 @@ It works this way:
+ If the match fails or if the ``aud`` value is not a string or array of strings, then the token will be rejected with a :ref:`401 Unauthorized <pgrst303>` error. + If the match fails or if the ``aud`` value is not a string or array of strings, then the token will be rejected with a :ref:`401 Unauthorized <pgrst303>` error.
+ If the ``aud`` key **is not present** or if its value is ``null`` or ``[]``, PostgREST will interpret this token as allowed for all audiences and will complete the request. + If the ``aud`` key **is not present** or if its value is ``null`` or ``[]``, PostgREST will interpret this token as allowed for all audiences and will complete the request.
.. _jwt_caching:
JWT Cache
---------
JWT signature validation (specially :ref:`asym_keys` such as RSA) is slow, we can cache ``JWT`` validation results to avoid this performance overhead.
The JWT cache is bounded and uses the `SIEVE algorithm <https://cachemon.github.io/SIEVE-website>`_ for efficient eviction. The cache is enabled by default and can be configured with :ref:`jwt-cache-max-entries`.
It's recommended to leave the JWT cache enabled as our load tests indicate ~20% more throughput for simple GET requests when using it. This while reducing CPU utilization in exchange for a bit more memory.
:ref:`jwt_cache_metrics` are available.
.. note::
- If the ``jwt-secret`` is changed and the config is reloaded, the JWT cache will reset.
- JWTs that pass :ref:`jwt_signature` are cached, regardless if they pass :ref:`jwt_claims_validation`. We do this to ensure responses stays fast under common failure cases (such as expired JWTs).
- You can use the :ref:`server-timing_header` to see the peformance benefit of JWT caching.
.. _jwt_role_extract: .. _jwt_role_extract:
JWT Role Extraction JWT Role Extraction
------------------- ~~~~~~~~~~~~~~~~~~~
A JSPath DSL that specifies the location of the :code:`role` key in the JWT claims. It's configured by :ref:`jwt-role-claim-key`. This can be used to consume a JWT provided by a third party service like Auth0, Okta, Microsoft Entra or Keycloak. A JSPath DSL that specifies the location of the :code:`role` key in the JWT claims. It's configured by :ref:`jwt-role-claim-key`. This can be used to consume a JWT provided by a third party service like Auth0, Okta, Microsoft Entra or Keycloak.
@@ -260,7 +242,7 @@ Usage examples:
The string comparison operators are implemented as a custom extension to the JSPath and does not strictly follow the `RFC 9535 <https://www.rfc-editor.org/rfc/rfc9535.html>`_. The string comparison operators are implemented as a custom extension to the JSPath and does not strictly follow the `RFC 9535 <https://www.rfc-editor.org/rfc/rfc9535.html>`_.
JWT Security JWT Security
------------ ~~~~~~~~~~~~
There are at least three types of common critiques against using JWT: 1) against the standard itself, 2) against using libraries with known security vulnerabilities, and 3) against using JWT for web sessions. We'll briefly explain each critique, how PostgREST deals with it, and give recommendations for appropriate user action. There are at least three types of common critiques against using JWT: 1) against the standard itself, 2) against using libraries with known security vulnerabilities, and 3) against using JWT for web sessions. We'll briefly explain each critique, how PostgREST deals with it, and give recommendations for appropriate user action.
@@ -296,23 +278,3 @@ doing custom logic based on the web user info.
END IF; END IF;
END END
$$ LANGUAGE plpgsql; $$ LANGUAGE plpgsql;
.. raw:: html
<script type="text/javascript">
let hash = window.location.hash;
const redirects = {
'#jwt-based-user-impersonation': '#jwt-authentication',
'#client-auth': '#bearer-authentication',
'#jwt-caching': '#jwt-cache',
'#jwk-kid-validation': '#kid-verification',
'#jwt-aud-claim-validation': '#aud-validation',
};
let willRedirectTo = redirects[hash];
if (willRedirectTo) {
window.location.href = willRedirectTo;
}
</script>
+8 -46
View File
@@ -3,47 +3,23 @@
CLI CLI
=== ===
PostgREST provides a CLI with the options listed below: PostgREST provides a CLI with the commands listed below:
.. code:: text
Usage: postgrest [-v|--version] [-e|--example] [--dump-config | --dump-schema | --ready]
[FILENAME]
PostgREST / create a REST API to an existing Postgres
database
Available options:
-h,--help Show this help text
-v,--version Show the version information
-e,--example Show an example configuration file
--dump-config Dump loaded configuration and exit
--dump-schema Dump loaded schema as JSON and exit (for debugging,
output structure is unstable)
--ready Checks the health of PostgREST by doing a request on
the admin server /ready endpoint
FILENAME Path to configuration file
FILENAME
--------
Runs PostgREST with the given :ref:`file_config`.
Help Help
---- ----
.. code:: bash .. code:: bash
$ postgrest --help $ postgrest [-h|--help]
Shows all the options available. Shows all the commands available.
Version Version
------- -------
.. code:: bash .. code:: bash
$ postgrest --version $ postgrest [-v|--version]
Prints the PostgREST version. Prints the PostgREST version.
@@ -52,16 +28,16 @@ Example
.. code:: bash .. code:: bash
$ postgrest --example $ postgrest [-e|--example]
Shows example configuration settings. Shows example configuration options.
Dump Config Dump Config
----------- -----------
.. code:: bash .. code:: bash
$ postgrest --dump-config $ postgrest [--dump-config]
Dumps the loaded :ref:`configuration` values, considering the configuration file, environment variables and :ref:`in_db_config`. Dumps the loaded :ref:`configuration` values, considering the configuration file, environment variables and :ref:`in_db_config`.
@@ -70,20 +46,6 @@ Dump Schema
.. code:: bash .. code:: bash
$ postgrest --dump-schema $ postgrest [--dump-schema]
Dumps the schema cache in JSON format. Dumps the schema cache in JSON format.
Ready Flag
----------
Makes a request to the ``/ready`` endpoint of the :ref:`admin_server`. It exits with a return code of ``0`` on success and ``1`` on failure.
.. code-block:: bash
$ postgrest --ready
OK: http://localhost:3001/ready
.. note::
The ``--ready`` flag cannot be used when :ref:`server-host` is configured with special hostnames. We suggest to change it to ``localhost``.
+21 -12
View File
@@ -316,7 +316,7 @@ db-extra-search-path
Multiple schemas can be added in a comma-separated string, e.g. ``public, extensions``. Multiple schemas can be added in a comma-separated string, e.g. ``public, extensions``.
.. important:: .. important::
We default this config to ``public`` because it is the most common schema used to install PostgreSQL extensions such as :ref:`PostGIS <ww_postgis>`. You can disable this by setting this config to ``""``. We default this config to ``public`` because it is the most common schema used to install PostgreSQL extensions such as :ref:`PostGIS <ww_postgis>`. You can disable this by setting this config to ``""``.
.. _db-hoisted-tx-settings: .. _db-hoisted-tx-settings:
@@ -409,7 +409,7 @@ db-pool-max-idletime
**In-Database** `n/a` **In-Database** `n/a`
=============== ================================= =============== =================================
*For backwards compatibility, this config parameter is also available as "db-pool-timeout".* *For backwards compatibility, this config parameter is also available as db-pool-timeout.*
Time in seconds to close idle pool connections. Time in seconds to close idle pool connections.
@@ -603,7 +603,7 @@ jwt-aud
**In-Database** pgrst.jwt_aud **In-Database** pgrst.jwt_aud
=============== ================================= =============== =================================
Specifies an audience for the JWT ``aud`` claim. See :ref:`jwt_aud`. Specifies an audience for the JWT ``aud`` claim. See :ref:`jwt_aud_validation`.
.. _jwt-role-claim-key: .. _jwt-role-claim-key:
@@ -658,20 +658,20 @@ jwt-secret-is-base64
When this is set to :code:`true`, the value derived from :code:`jwt-secret` will be treated as a base64 encoded secret. When this is set to :code:`true`, the value derived from :code:`jwt-secret` will be treated as a base64 encoded secret.
.. _jwt-cache-max-entries: .. _jwt-cache-max-lifetime:
jwt-cache-max-entries jwt-cache-max-lifetime
---------------------- ----------------------
=============== ================================= =============== =================================
**Type** Int **Type** Int
**Default** 1000 **Default** 0
**Reloadable** Y **Reloadable** Y
**Environment** PGRST_JWT_CACHE_MAX_ENTRIES **Environment** PGRST_JWT_CACHE_MAX_LIFETIME
**In-Database** pgrst.jwt_cache_max_entries **In-Database** pgrst.jwt_cache_max_lifetime
=============== ================================= =============== =================================
Maximum number of entries in JWT cache. The value :code:`0` disables JWT caching. See :ref:`jwt_caching`. Maximum number of seconds of lifetime for cached entries. The default :code:`0` disables caching. See :ref:`jwt_caching`.
.. _log-level: .. _log-level:
@@ -714,14 +714,23 @@ log-query
--------- ---------
=============== ================================= =============== =================================
**Type** Boolean **Type** String
**Default** False **Default** "disabled"
**Reloadable** Y **Reloadable** Y
**Environment** PGRST_LOG_QUERY **Environment** PGRST_LOG_QUERY
**In-Database** `n/a` **In-Database** `n/a`
=============== ================================= =============== =================================
Logs the SQL query for the corresponding request at the current :ref:`log-level`. See :ref:`sql_query_logs`. Logs the SQL query for the corresponding request at the current :ref:`log-level`.
See :ref:`sql_query_logs`.
.. code:: bash
# Logs the main SQL query
log-query = "main-query"
# Disables logging the SQL query
log-query = "disabled"
.. _openapi-mode: .. _openapi-mode:
+2 -2
View File
@@ -15,7 +15,7 @@ Dynamic Connection Pool
To conserve system resources, PostgREST uses a dynamic connection pool. This enables the number of connections in the pool to increase and decrease depending on request traffic. To conserve system resources, PostgREST uses a dynamic connection pool. This enables the number of connections in the pool to increase and decrease depending on request traffic.
- If all the connections are being used, a new connection is added. The pool can grow until it reaches the :ref:`db-pool` size. Note that it's pointless to set this higher than the ``max_connections`` setting in your database. - If all the connections are being used, a new connection is added. The pool can grow until it reaches the :ref:`db-pool` size. Note that its pointless to set this higher than the ``max_connections`` setting in your database.
- If a connection is unused for a period of time (:ref:`db-pool-max-idletime`), it will be released. - If a connection is unused for a period of time (:ref:`db-pool-max-idletime`), it will be released.
- For connecting to the database, the :ref:`authenticator <roles>` role is used. You can configure this using :ref:`db-uri`. - For connecting to the database, the :ref:`authenticator <roles>` role is used. You can configure this using :ref:`db-uri`.
@@ -106,4 +106,4 @@ Also set :ref:`db-channel-enabled` to ``false`` since ``LISTEN`` is not compatib
.. note:: .. note::
It's not recommended to use an external connection pooler. `Our benchmarks <https://github.com/PostgREST/postgrest/issues/2294#issuecomment-1139148672>`_ indicate it provides much lower performance than PostgREST built-in pool. Its not recommended to use an external connection pooler. `Our benchmarks <https://github.com/PostgREST/postgrest/issues/2294#issuecomment-1139148672>`_ indicate it provides much lower performance than PostgREST built-in pool.
+1 -1
View File
@@ -329,7 +329,7 @@ Related to the authentication process using JWT. You can follow the :ref:`tut1`
| PGRST301 | | | | PGRST301 | | |
+---------------+-------------+-------------------------------------------------------------+ +---------------+-------------+-------------------------------------------------------------+
| .. _pgrst302: | 401 | Attempted to do a request without | | .. _pgrst302: | 401 | Attempted to do a request without |
| | | :ref:`bearer_auth` when the anonymous role | | | | :ref:`authentication <client_auth>` when the anonymous role |
| PGRST302 | | is disabled by not setting it in :ref:`db-anon-role`. | | PGRST302 | | is disabled by not setting it in :ref:`db-anon-role`. |
+---------------+-------------+-------------------------------------------------------------+ +---------------+-------------+-------------------------------------------------------------+
| .. _pgrst303: | 401 | :ref:`JWT claims validation <jwt_claims_validation>` | | .. _pgrst303: | 401 | :ref:`JWT claims validation <jwt_claims_validation>` |
+2 -4
View File
@@ -4,7 +4,7 @@ Listener
######## ########
PostgREST uses `LISTEN <https://www.postgresql.org/docs/current/sql-listen.html>`_ to reload its :ref:`Schema Cache <schema_reloading_notify>` and :ref:`Configuration <config_reloading_notify>` via `NOTIFY <https://www.postgresql.org/docs/current/sql-notify.html>`_. PostgREST uses `LISTEN <https://www.postgresql.org/docs/current/sql-listen.html>`_ to reload its :ref:`Schema Cache <schema_reloading_notify>` and :ref:`Configuration <config_reloading_notify>` via `NOTIFY <https://www.postgresql.org/docs/current/sql-notify.html>`_.
This is useful in environments where you can't send SIGUSR1 or SIGUSR2 Unix Signals. This is useful in environments where you cant send SIGUSR1 or SIGUSR2 Unix Signals.
Like on cloud managed containers or on Windows systems. Like on cloud managed containers or on Windows systems.
.. code:: postgresql .. code:: postgresql
@@ -46,9 +46,7 @@ This will cause the :ref:`connection_pool` to connect to the read replica host a
.. note:: .. note::
- Under the hood, PostgREST forces `target_session_attrs=read-write <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS>`_ for the ``LISTEN`` session. Under the hood, PostgREST forces `target_session_attrs=read-write <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS>`_ for the ``LISTEN`` session.
So if you specify ``target_session_attrs=read-only`` as mentioned above, PostgREST will override it for the ``LISTEN``.
- ``read-only`` is only available on libpq >= 14, if you use a lower version you will get an error like ``invalid target_session_attrs value: \"read-only\"``.
.. _listener_automatic_recovery: .. _listener_automatic_recovery:
+6 -38
View File
@@ -39,7 +39,7 @@ For diagnostic information about the server itself, PostgREST logs to ``stderr``
06/May/2024:08:16:11 -0500: Listening for database notifications on the "pgrst" channel 06/May/2024:08:16:11 -0500: Listening for database notifications on the "pgrst" channel
06/May/2024:08:16:11 -0500: Config reloaded 06/May/2024:08:16:11 -0500: Config reloaded
06/May/2024:08:16:11 -0500: Schema cache queried in 3.8 milliseconds 06/May/2024:08:16:11 -0500: Schema cache queried in 3.8 milliseconds
06/May/2024:08:16:11 -0500: Schema cache loaded 15 Relations, 8 Relationships, 8 RPCs, 0 Domain Representations, 4 Media Type Handlers 06/May/2024:08:16:11 -0500: Schema cache loaded 15 Relations, 8 Relationships, 8 Functions, 0 Domain Representations, 4 Media Type Handlers
06/May/2024:14:11:27 -0500: Received a config reload message on the "pgrst" channel 06/May/2024:14:11:27 -0500: Received a config reload message on the "pgrst" channel
06/May/2024:14:11:27 -0500: Config reloaded 06/May/2024:14:11:27 -0500: Config reloaded
@@ -52,12 +52,14 @@ For diagnostic information about the server itself, PostgREST logs to ``stderr``
SQL Query Logs SQL Query Logs
-------------- --------------
To log the SQL queries executed for a request, set the :ref:`log-query` to ``true``. It will be logged based on the current :ref:`log-level` setting. To log the :ref:`main SQL query <main_query>` executed for a request, set the :ref:`log-query` to ``main-query``.
It will be logged based on the current :ref:`log-level` setting.
For example, with this configuration:
.. code-block:: bash .. code-block:: bash
log-level = "warn" log-level = "warn"
log-query = "true" log-query = "main-query"
The SQL queries will only be logged on ``400`` HTTP errors and up. The SQL queries will only be logged on ``400`` HTTP errors and up.
So, if the user requests a resource without sufficient privileges: So, if the user requests a resource without sufficient privileges:
@@ -204,40 +206,6 @@ pgrst_db_pool_max
Max pool connections. Max pool connections.
.. _jwt_cache_metrics:
JWT Cache Metrics
-----------------
Metrics related to the :ref:`jwt_caching`.
pgrst_jwt_cache_requests_total
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
======== =======
**Type** Counter
======== =======
The total number of JWT cache lookups.
pgrst_jwt_cache_hits_total
~~~~~~~~~~~~~~~~~~~~~~~~~~
======== =======
**Type** Counter
======== =======
The total number of JWT cache hits.
pgrst_jwt_cache_evictions_total
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
======== =======
**Type** Counter
======== =======
The total number of JWT cache evictions.
Traces Traces
====== ======
@@ -297,7 +265,7 @@ This header communicates metrics of the different phases in the request-response
Server-Timing: jwt;dur=14.9, parse;dur=71.1, plan;dur=109.0, transaction;dur=353.2, response;dur=4.4 Server-Timing: jwt;dur=14.9, parse;dur=71.1, plan;dur=109.0, transaction;dur=353.2, response;dur=4.4
- All the durations (``dur``) are in milliseconds. - All the durations (``dur``) are in milliseconds.
- The ``jwt`` stage is when :ref:`jwt_auth` is done. This duration can be lowered with :ref:`jwt_caching`. - The ``jwt`` stage is when :ref:`jwt_impersonation` is done. This duration can be lowered with :ref:`jwt_caching`.
- On the ``parse`` stage, the :ref:`url_grammar` is parsed. - On the ``parse`` stage, the :ref:`url_grammar` is parsed.
- On the ``plan`` stage, the :ref:`schema_cache` is used to generate the :ref:`main_query` of the transaction. - On the ``plan`` stage, the :ref:`schema_cache` is used to generate the :ref:`main_query` of the transaction.
- The ``transaction`` stage corresponds to the database transaction. See :ref:`transactions`. - The ``transaction`` stage corresponds to the database transaction. See :ref:`transactions`.
-13
View File
@@ -53,19 +53,6 @@ To reload the schema cache from within the database, you can use the ``NOTIFY``
NOTIFY pgrst, 'reload schema' NOTIFY pgrst, 'reload schema'
Debouncing
~~~~~~~~~~
PostgREST does not reload the schema cache for each notification when several ``NOTIFY pgrst`` events are generated quickly after one another.
There are two cases to consider: when notifications are sent within a single transaction and when they are sent across multiple transactions.
In the first case, PostgreSQL deduplicates identical ``NOTIFY`` events within the same transaction. This means that even if multiple ``NOTIFY pgrst`` statements are executed before a ``COMMIT``, only a single notification is delivered to PostgREST.
In the second case, when notifications are sent from separate transactions in a short time span, PostgREST applies a debouncing mechanism to avoid excessive schema cache reloads.
Instead of reloading the schema cache for each notification, events are grouped within a small time window of 100 milliseconds. The reload function is executed once immediately when the first notification is received and once more after the burst of events settles, resulting in at most two executions within that time window.
.. _auto_schema_reloading: .. _auto_schema_reloading:
Automatic Schema Cache Reloading Automatic Schema Cache Reloading
+2 -2
View File
@@ -1,7 +1,7 @@
# This file is auto-generated by postgrest-nixpkgs-upgrade # This file is auto-generated by postgrest-nixpkgs-upgrade
sphinx==8.2.3 sphinx==7.4.7
sphinx-copybutton==0.5.2 sphinx-copybutton==0.5.2
sphinx-rtd-dark-mode==1.3.0 sphinx-rtd-dark-mode==1.3.0
sphinx-rtd-theme==3.0.2 sphinx-rtd-theme==3.0.2
sphinx-tabs==3.4.7 sphinx-tabs==3.4.7
sphinxext-opengraph==0.9.1 sphinxext-opengraph==0.9.1
Generated
+4 -4
View File
@@ -2,16 +2,16 @@
"nodes": { "nodes": {
"nixpkgs": { "nixpkgs": {
"locked": { "locked": {
"lastModified": 1752006229, "lastModified": 1731165248,
"narHash": "sha256-BeuAPwNM2RBc5bvUTb0j4GRs2yBkDeRCw/8Y3v9Xesc=", "narHash": "sha256-DiHFKIdBmMx5/DUARhVqaxvEIiy4EE6Eqs9Qs4oxme8=",
"owner": "nixos", "owner": "nixos",
"repo": "nixpkgs", "repo": "nixpkgs",
"rev": "c80edd02003fe3d8af527215a3ac069be9cfd47f", "rev": "a90280100f41a10914edfe729a4053e60c92b8e3",
"type": "github" "type": "github"
}, },
"original": { "original": {
"owner": "nixos", "owner": "nixos",
"ref": "nixpkgs-25.05-darwin", "ref": "nixpkgs-unstable",
"repo": "nixpkgs", "repo": "nixpkgs",
"type": "github" "type": "github"
} }
+4 -4
View File
@@ -2,7 +2,7 @@
description = "REST API for any Postgres database"; description = "REST API for any Postgres database";
inputs = { inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-25.05-darwin"; nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
}; };
nixConfig = { nixConfig = {
@@ -33,8 +33,8 @@
in in
{ {
packages = genSystems (attrs: { packages = genSystems (attrs: {
default = attrs.postgrestPackage.bin; default = attrs.postgrestPackage;
profiled = attrs.postgrestProfiled.bin; profiled = attrs.postgrestProfiled;
} // nixpkgs.lib.optionalAttrs (attrs ? postgrestStatic) { } // nixpkgs.lib.optionalAttrs (attrs ? postgrestStatic) {
static = attrs.postgrestStatic; static = attrs.postgrestStatic;
}); });
@@ -42,7 +42,7 @@
apps = genSystems (attrs: { apps = genSystems (attrs: {
default = { default = {
type = "app"; type = "app";
program = "${attrs.postgrestStatic or attrs.postgrestPackage.bin}/bin/postgrest"; program = "${attrs.postgrestStatic or attrs.postgrestPackage}/bin/postgrest";
meta.description = "REST API for any Postgres database"; meta.description = "REST API for any Postgres database";
}; };
}); });
+15 -15
View File
@@ -72,10 +72,9 @@ The PostgREST utilities available in `nix-shell` all have names that begin with
```bash ```bash
# Note: The utilities listed here might not be up to date. # Note: The utilities listed here might not be up to date.
[nix-shell]$ postgrest-<tab> [nix-shell]$ postgrest-<tab>
postgrest-build postgrest-parallel-curl postgrest-build postgrest-profiled-run
postgrest-check postgrest-profiled-run postgrest-check postgrest-push-cachix
postgrest-clean postgrest-push-cachix postgrest-clean postgrest-release
postgrest-commitlint postgrest-release
postgrest-coverage postgrest-repl postgrest-coverage postgrest-repl
postgrest-coverage-draft-overlay postgrest-run postgrest-coverage-draft-overlay postgrest-run
postgrest-docs-build postgrest-style postgrest-docs-build postgrest-style
@@ -91,14 +90,15 @@ postgrest-gen-ctags postgrest-watch
postgrest-gen-jwt postgrest-with-all postgrest-gen-jwt postgrest-with-all
postgrest-gen-secret postgrest-with-git postgrest-gen-secret postgrest-with-git
postgrest-git-hooks postgrest-with-pgrst postgrest-git-hooks postgrest-with-pgrst
postgrest-hsie-graph-modules postgrest-with-pg-13 postgrest-hsie-graph-modules postgrest-with-postgresql-12
postgrest-hsie-graph-symbols postgrest-with-pg-14 postgrest-hsie-graph-symbols postgrest-with-postgresql-13
postgrest-hsie-minimal-imports postgrest-with-pg-15 postgrest-hsie-minimal-imports postgrest-with-postgresql-14
postgrest-lint postgrest-with-pg-16 postgrest-lint postgrest-with-postgresql-15
postgrest-loadtest postgrest-with-pg-17 postgrest-loadtest postgrest-with-postgresql-16
postgrest-loadtest-against postgrest-with-slow-pg postgrest-loadtest-against postgrest-with-postgresql-17
postgrest-loadtest-report postgrest-with-slow-postgrest postgrest-loadtest-report postgrest-with-slow-pg
postgrest-nixpkgs-upgrade postgrest-nixpkgs-upgrade postgrest-with-slow-postgrest
postgrest-parallel-curl
... ...
[nix-shell]$ [nix-shell]$
@@ -174,7 +174,7 @@ $ nix-shell --run "postgrest-with-all postgrest-test-spec"
# Run the tests against a specific version of PostgreSQL (use tab-completion in # Run the tests against a specific version of PostgreSQL (use tab-completion in
# nix-shell to see all available versions): # nix-shell to see all available versions):
$ nix-shell --run "postgrest-with-pg-13 postgrest-test-spec" $ nix-shell --run "postgrest-with-postgresql-13 postgrest-test-spec"
``` ```
@@ -284,7 +284,7 @@ Tools like `postgrest-build`, `postgrest-run`, `postgrest-repl` etc. are simple
also run in CI, with the exception of the IO and Memory checks that need to be run also run in CI, with the exception of the IO and Memory checks that need to be run
separately. separately.
`postgrest-with-pg-*` take a command as an argument and will run it `postgrest-with-postgresql-*` take a command as an argument and will run it
with a temporary database. `postgrest-with-all` will run the command against with a temporary database. `postgrest-with-all` will run the command against
all supported PostgreSQL versions. Tests run without `postgrest-with-*` are all supported PostgreSQL versions. Tests run without `postgrest-with-*` are
run against the latest PostgreSQL version by default. run against the latest PostgreSQL version by default.
@@ -380,7 +380,7 @@ that).
We also use `default.nix` to load our pinned version of the `nixpkgs` We also use `default.nix` to load our pinned version of the `nixpkgs`
repository. This set of packages will always be the same, independently from repository. This set of packages will always be the same, independently from
where or when you use it. The pinned version is taken from `flake.lock` and where or when you use it. The pinned version is taken from `flake.lock` and
can be updated with `postgrest-nixpkgs-upgrade`. can be updated with `nix flake update`.
### `shell.nix` ### `shell.nix`
+61
View File
@@ -0,0 +1,61 @@
# Creating a separate libpq package is is discussed in
# https://github.com/NixOS/nixpkgs/issues/61580, but nixpkgs has not moved
# forward, yet.
# This package is passed to postgresql-libpq (haskell) which needs to be
# cross-compiled to the static build and possibly other architectures as
# as well. To reduce the number of dependencies that need to be built with
# it, this derivation focuses on building the client libraries only. No
# server, no tests.
{ stdenv
, lib
, openssl
, zlib
, postgresql
, pkg-config
, tzdata
}:
stdenv.mkDerivation {
pname = "libpq";
inherit (postgresql) src version patches;
__structuredAttrs = true;
env.CFLAGS = "-fdata-sections -ffunction-sections"
+ (if stdenv.cc.isClang then " -flto" else " -fmerge-constants -Wl,--gc-sections");
configureFlags = [
"--without-gssapi"
"--without-icu"
"--without-readline"
"--with-openssl"
"--with-system-tzdata=${tzdata}/share/zoneinfo"
"--sysconfdir=/etc/postgresql"
];
nativeBuildInputs = [ pkg-config tzdata ];
buildInputs = [ openssl zlib ];
buildFlags = [ "submake-libpq" "submake-libpgport" ];
installPhase = ''
runHook preInstall
make -C src/bin/pg_config install
make -C src/common install
make -C src/include install
make -C src/interfaces/libpq install
make -C src/port install
rm -rfv $out/share
runHook postInstall
'';
outputs = [ "out" ];
meta = with lib; {
homepage = "https://www.postgresql.org";
description = "Client API library for PostgreSQL";
license = licenses.postgresql;
};
}
@@ -6,7 +6,6 @@
, coreutils , coreutils
, git , git
, lib , lib
, moreutils
, runCommand , runCommand
, shellcheck , shellcheck
, stdenv , stdenv
@@ -57,7 +56,7 @@ let
# Example: This way `postgrest-watch -h` will return the help output for watch, while # Example: This way `postgrest-watch -h` will return the help output for watch, while
# `postgrest-watch postgrest-test-spec -h` will return the help output for test-spec. # `postgrest-watch postgrest-test-spec -h` will return the help output for test-spec.
# Taken from: https://github.com/matejak/argbash/issues/114#issuecomment-557108274 # Taken from: https://github.com/matejak/argbash/issues/114#issuecomment-557108274
sed '/_positionals_count + 1/a\\t\t\t\tset -- "''${@:1:1}" "--" "''${@:2}"' $out | ${moreutils}/bin/sponge $out sed '/_positionals_count + 1/a\\t\t\t\tset -- "''${@:1:1}" "--" "''${@:2}"' -i $out
''; '';
bash-completion = bash-completion =
@@ -67,7 +66,7 @@ let
'' ''
+ lib.optionalString (positionalCompletion != "") '' + lib.optionalString (positionalCompletion != "") ''
sed 's#COMPREPLY.*compgen -o bashdefault .*$#${escape positionalCompletion}#' $out | ${moreutils}/bin/sponge $out sed 's#COMPREPLY.*compgen -o bashdefault .*$#${escape positionalCompletion}#' -i $out
'' ''
); );
@@ -104,7 +103,8 @@ let
'' ''
+ lib.optionalString withTmpDir '' + lib.optionalString withTmpDir ''
tmpdir="$(${coreutils}/bin/mktemp -d --tmpdir ${name}-XXX)" mkdir -p "''${TMPDIR:-/tmp}/postgrest"
tmpdir="$(${coreutils}/bin/mktemp -d --tmpdir postgrest/${name}-XXX)"
# we keep the tmpdir when an error occurs for debugging # we keep the tmpdir when an error occurs for debugging
trap 'echo Temporary directory kept at: $tmpdir' ERR trap 'echo Temporary directory kept at: $tmpdir' ERR
+1
View File
@@ -3,5 +3,6 @@
checked-shell-script = import ./checked-shell-script; checked-shell-script = import ./checked-shell-script;
gitignore = import ./gitignore.nix; gitignore = import ./gitignore.nix;
haskell-packages = import ./haskell-packages.nix; haskell-packages = import ./haskell-packages.nix;
postgresql-libpq = import ./postgresql-libpq.nix;
slocat = import ./slocat.nix; slocat = import ./slocat.nix;
} }
+25 -80
View File
@@ -50,16 +50,6 @@ let
# jailbreak, because hspec limit for tests # jailbreak, because hspec limit for tests
fuzzyset = prev.fuzzyset_0_2_4; fuzzyset = prev.fuzzyset_0_2_4;
# TODO: Remove once available in nixpkgs
auto-update =
prev.callHackageDirect
{
pkg = "auto-update";
ver = "0.2.7";
sha256 = "sha256-fHX/OqF/cB9rbpGpLUtA29bcEJS43HUWHcK55yUxKoo=";
}
{ };
# TODO: Remove once available in nixpkgs haskellPackages # TODO: Remove once available in nixpkgs haskellPackages
configurator-pg = configurator-pg =
prev.callHackageDirect prev.callHackageDirect
@@ -70,79 +60,34 @@ let
} }
{ }; { };
# TODO: Remove once available in nixpkgs haskellPackages hasql-pool = lib.dontCheck (prev.callHackageDirect
streaming-commons = {
prev.callHackageDirect pkg = "hasql-pool";
{ ver = "1.0.1";
pkg = "streaming-commons"; sha256 = "sha256-Hf1f7lX0LWkjrb25SDBovCYPRdmUP1H6pAxzi7kT4Gg=";
ver = "0.2.3.1"; }
sha256 = "sha256-Gl2eaJcWe1sxmcE/octWlH9uSnERguf+5H66K4fV87s="; { });
}
{ };
http2 = hasql-notifications = lib.dontCheck (prev.callHackageDirect
prev.callHackageDirect {
{ pkg = "hasql-notifications";
pkg = "http2"; ver = "0.2.2.2";
ver = "5.4.0"; sha256 = "sha256-myKwlug7OgTa/qP6mHfCD+5Q8IhM17JvpJBfSo+M01k=";
sha256 = "sha256-PeEWVd61bQ8G7LvfLeXklzXqNJFaAjE2ecRMWJZESPE="; }
} { });
{ };
http-semantics = # newer nixpkgs already has 0.10., so we fallback to default for forward compat
prev.callHackageDirect jose-jwt = prev.jose-jwt_0_10_0 or prev.jose-jwt;
{
pkg = "http-semantics";
ver = "0.4.0";
sha256 = "sha256-rh0z51EKvsu5rQd5n2z3fSRjjEObouNZSBPO9NFYOF0=";
}
{ };
jose-jwt = postgresql-libpq = lib.dontCheck (prev.callHackageDirect
prev.callHackageDirect {
{ pkg = "postgresql-libpq";
pkg = "jose-jwt"; ver = "0.10.1.0";
ver = "0.9.6"; sha256 = "sha256-tXOMqCO8opMilI9rx0D+njqjIjbZsH168Bzb8Aq8Ff4=";
sha256 = "sha256-FhBz5wzyNrDvmjHWOeNAHuVMyJUVSlm+DeQQuITSjaI="; }
} {
{ }; postgresql = super.libpq;
});
time-manager =
prev.callHackageDirect
{
pkg = "time-manager";
ver = "0.2.4";
sha256 = "sha256-sAt/331YLQ2IU3z90aKYSq1nxoazv87irsuJp7ZG3pw=";
}
{ };
network-run =
prev.callHackageDirect
{
pkg = "network-run";
ver = "0.5.0";
sha256 = "sha256-vbXh+CzxDsGApjqHxCYf/ijpZtUCApFbkcF5gyN0THU=";
}
{ };
warp =
lib.dontCheck
(prev.callHackageDirect
{
pkg = "warp";
ver = "3.4.14";
sha256 = "sha256-RnoOUlC6dOP0sK/tYAJCX1oLzVFG1GILUY+yVbmvW8Y=";
}
{ });
# Downgrade hasql and related packages while we are still on GHC 9.4 for the static build.
hasql = lib.dontCheck (lib.doJailbreak prev.hasql_1_6_4_4);
hasql-dynamic-statements = lib.dontCheck prev.hasql-dynamic-statements_0_3_1_5;
hasql-implicits = lib.dontCheck prev.hasql-implicits_0_1_1_3;
hasql-notifications = lib.dontCheck prev.hasql-notifications_0_2_2_2;
hasql-pool = lib.dontCheck prev.hasql-pool_1_0_1;
hasql-transaction = lib.dontCheck prev.hasql-transaction_1_1_0_1;
postgresql-binary = lib.dontCheck (lib.doJailbreak prev.postgresql-binary_0_13_1_3);
}; };
in in
{ {
+7
View File
@@ -0,0 +1,7 @@
_: super:
{
# Depending on which nixpkgs version is pinned, libpq might either be available already - or not.
libpq = super.libpq or (super.callPackage ../libpq.nix {
postgresql = super.postgresql_16;
});
}
+53 -13
View File
@@ -8,25 +8,65 @@ let
inherit (pkgs) pkgsStatic; inherit (pkgs) pkgsStatic;
inherit (pkgsStatic.haskell) lib; inherit (pkgsStatic.haskell) lib;
packagesStatic = pkgsStatic.haskell.packages.native-bignum."${compiler}"; packagesStatic =
pkgsStatic.haskell.packages."${compiler}".override (old: {
ghc = pkgsStatic.pkgsBuildHost.haskell.compiler."${compiler}".override {
# Using the bundled libffi generally works better for cross-compiling
libffi = null;
# Building sphinx fails on some platforms
enableDocs = false;
# Cross compiling with native bignum works better than with gmp
enableNativeBignum = true;
};
overrides = pkgs.lib.composeExtensions old.overrides (_: prev: {
postgresql-libpq = (lib.overrideCabal prev.postgresql-libpq {
# TODO: This section can be simplified when this PR has made it's way to us:
# https://github.com/NixOS/nixpkgs/pull/286370
# Additionally, we need to use the default version in nixpkgs, otherwise the
# override will not be active as well.
# Using use-pkg-config flag, because pg_config won't work when cross-compiling
configureFlags = [ "-fuse-pkg-config" ];
# postgresql doesn't build in the fully static overlay - but the default
# derivation is built with static libraries anyway.
libraryPkgconfigDepends = [ pkgsStatic.libpq ];
librarySystemDepends = [ ];
}).overrideAttrs (_: prevAttrs: {
buildInputs = prevAttrs.buildInputs ++ [ pkgsStatic.openssl ];
});
});
});
makeExecutableStatic = drv: pkgs.lib.pipe drv [ makeExecutableStatic = drv: pkgs.lib.pipe drv [
lib.compose.justStaticExecutables lib.compose.justStaticExecutables
# To successfully compile a redistributable, fully static executable we need to: # To successfully compile a redistributable, fully static executable we need to:
# 1. avoid any references to /nix/store to prevent blowing up the closure size. # 1. make executable really statically linked.
(drv: drv.overrideAttrs { # 2. avoid any references to /nix/store to prevent blowing up the closure size.
allowedReferences = [ # 3. be able to run the executable.
pkgsStatic.openssl.etc # When checking for references, we ignore the following:
]; # - eeee... are removed references which don't actually exist
}) # - openssl-etc references are purposely designed to be very small
(lib.compose.overrideCabal (drv: {
postFixup = drv.postFixup + ''
exe="$out/bin/postgrest"
# 2. be able to run the executable. if ! (file "$exe" | grep 'statically linked') then
(drv: drv.overrideAttrs { echo "not a static executable, ldd output:"
passthru.tests.version = pkgsStatic.testers.testVersion { ldd "$exe"
package = drv; exit 1
}; fi
})
echo "Checking for references to /nix/store..."
(${pkgsStatic.binutils}/bin/strings "$exe" \
| grep -v /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee \
| grep -v -etc/etc/ssl \
| grep /nix/store || exit 0 && exit 1)
echo "No references to /nix/store found"
"$exe" --help
'';
}))
]; ];
in in
+5 -16
View File
@@ -15,6 +15,7 @@ let
withEnv = postgrest.env; withEnv = postgrest.env;
} }
'' ''
${cabal-install}/bin/cabal v2-update
exec ${cabal-install}/bin/cabal v2-build ${devCabalOptions} "''${_arg_leftovers[@]}" exec ${cabal-install}/bin/cabal v2-build ${devCabalOptions} "''${_arg_leftovers[@]}"
''; '';
@@ -33,17 +34,6 @@ let
exec ${cabal-install}/bin/cabal v2-clean exec ${cabal-install}/bin/cabal v2-clean
''; '';
update =
checkedShellScript
{
name = "postgrest-cabal-update";
docs = "Update cabal's package list from hackage.haskell.org";
workingDir = "/";
}
''
exec ${cabal-install}/bin/cabal v2-update
'';
run = run =
checkedShellScript checkedShellScript
{ {
@@ -55,7 +45,6 @@ let
"ARG_USE_ENV([PGRST_DB_POOL], [1], [PostgREST pool size])" "ARG_USE_ENV([PGRST_DB_POOL], [1], [PostgREST pool size])"
"ARG_USE_ENV([PGRST_DB_POOL_ACQUISITION_TIMEOUT], [1], [PostgREST pool timeout])" "ARG_USE_ENV([PGRST_DB_POOL_ACQUISITION_TIMEOUT], [1], [PostgREST pool timeout])"
"ARG_USE_ENV([PGRST_JWT_SECRET], [reallyreallyreallyreallyverysafe], [PostgREST JWT secret])" "ARG_USE_ENV([PGRST_JWT_SECRET], [reallyreallyreallyreallyverysafe], [PostgREST JWT secret])"
"ARG_USE_ENV([PGRST_ADMIN_SERVER_PORT], [3001], [PostgREST admin server port])"
"ARG_LEFTOVERS([PostgREST arguments])" "ARG_LEFTOVERS([PostgREST arguments])"
]; ];
workingDir = "/"; workingDir = "/";
@@ -66,7 +55,6 @@ let
export PGRST_DB_POOL export PGRST_DB_POOL
export PGRST_DB_POOL_ACQUISITION_TIMEOUT export PGRST_DB_POOL_ACQUISITION_TIMEOUT
export PGRST_JWT_SECRET export PGRST_JWT_SECRET
export PGRST_ADMIN_SERVER_PORT
exec ${cabal-install}/bin/cabal v2-run ${devCabalOptions} --verbose=0 -- \ exec ${cabal-install}/bin/cabal v2-run ${devCabalOptions} --verbose=0 -- \
postgrest "''${_arg_leftovers[@]}" postgrest "''${_arg_leftovers[@]}"
@@ -95,8 +83,10 @@ let
export PGRST_DB_POOL_ACQUISITION_TIMEOUT export PGRST_DB_POOL_ACQUISITION_TIMEOUT
export PGRST_JWT_SECRET export PGRST_JWT_SECRET
exec ${cabal-install}/bin/cabal --builddir="dist-prof" v2-run --enable-profiling --disable-shared exe:postgrest -- \ ${cabal-install}/bin/cabal v2-update
+RTS -p -h -RTS "''${_arg_leftovers[@]}" ${cabal-install}/bin/cabal --builddir="dist-prof" v2-build --enable-profiling --disable-shared exe:postgrest
${cabal-install}/bin/cabal --builddir="dist-prof" v2-run -- \
postgrest +RTS -p -h -RTS "''${_arg_leftovers[@]}"
''; '';
repl = repl =
@@ -119,7 +109,6 @@ buildToolbox
inherit inherit
build build
clean clean
update
run run
runProfiled runProfiled
repl; repl;
+2 -7
View File
@@ -10,7 +10,6 @@
, hsie , hsie
, nix , nix
, silver-searcher , silver-searcher
, stdenv
, style , style
, tests , tests
, withTools , withTools
@@ -54,14 +53,10 @@ let
Requires authentication with `cachix authtoken ...`. Requires authentication with `cachix authtoken ...`.
''; '';
args =
[
"ARG_OPTIONAL_SINGLE([system], , [System], [${stdenv.system}])"
];
workingDir = "/"; workingDir = "/";
} }
'' ''
${nix}/bin/nix-instantiate --argstr system "$_arg_system" \ ${nix}/bin/nix-instantiate \
| xargs ${nix}/bin/nix-store -qR --include-outputs \ | xargs ${nix}/bin/nix-store -qR --include-outputs \
| ${cachix}/bin/cachix push postgrest | ${cachix}/bin/cachix push postgrest
''; '';
@@ -82,7 +77,6 @@ let
} }
'' ''
${tests}/bin/postgrest-test-spec ${tests}/bin/postgrest-test-spec
${tests}/bin/postgrest-test-observability
${tests}/bin/postgrest-test-doctests ${tests}/bin/postgrest-test-doctests
${tests}/bin/postgrest-test-io ${tests}/bin/postgrest-test-io
${tests}/bin/postgrest-test-big-schema ${tests}/bin/postgrest-test-big-schema
@@ -252,6 +246,7 @@ let
} }
'' ''
mkdir -p "$_arg_dumpdir" mkdir -p "$_arg_dumpdir"
${cabal-install}/bin/cabal v2-update
${cabal-install}/bin/cabal v2-build ${devCabalOptions} \ ${cabal-install}/bin/cabal v2-build ${devCabalOptions} \
--builddir="$tmpdir" \ --builddir="$tmpdir" \
--ghc-option=-ddump-minimal-imports \ --ghc-option=-ddump-minimal-imports \
+21 -21
View File
@@ -2,6 +2,7 @@
, aspellDicts , aspellDicts
, buildToolbox , buildToolbox
, checkedShellScript , checkedShellScript
, fetchPypi
, lib , lib
, plantuml , plantuml
, python3 , python3
@@ -10,14 +11,26 @@
, writers , writers
}: }:
let let
selectPythonPackages = ps: [ selectPythonPackages = ps:
ps.sphinx let
ps.sphinx-copybutton # TODO: Remove with next nixpkgs update
ps.sphinx-rtd-dark-mode sphinx-rtd-theme = assert ps.sphinx-rtd-theme.version == "2.0.0"; ps.sphinx-rtd-theme.overrideAttrs rec {
ps.sphinx-rtd-theme version = "3.0.2";
ps.sphinx-tabs src = fetchPypi {
ps.sphinxext-opengraph pname = "sphinx_rtd_theme";
]; inherit version;
hash = "sha256-t0V7wl3acjsgsIamcLmVPIWeq2CioD7o6yuyPhduX4U=";
};
};
in
[
ps.sphinx
ps.sphinx-copybutton
(ps.sphinx-rtd-dark-mode.override { inherit sphinx-rtd-theme; })
sphinx-rtd-theme
ps.sphinx-tabs
ps.sphinxext-opengraph
];
requirements = writeTextFile { requirements = writeTextFile {
name = "requirements.txt"; name = "requirements.txt";
@@ -35,9 +48,6 @@ let
workingDir = "/docs"; workingDir = "/docs";
} }
'' ''
# https://github.com/sphinx-doc/sphinx/issues/11739
export LC_ALL=C
function build() { function build() {
${python}/bin/sphinx-build --color -W -a -n . -b "$@" ${python}/bin/sphinx-build --color -W -a -n . -b "$@"
} }
@@ -122,10 +132,6 @@ let
workingDir = "/docs"; workingDir = "/docs";
} }
'' ''
echo "Checking spelling mistakes..."
export LC_ALL=C
FILES=$(find . -type f -iname '*.rst' | tr '\n' ' ') FILES=$(find . -type f -iname '*.rst' | tr '\n' ' ')
# shellcheck disable=SC2086 disable=SC2016 # shellcheck disable=SC2086 disable=SC2016
@@ -146,10 +152,6 @@ let
workingDir = "/docs"; workingDir = "/docs";
} }
'' ''
echo "Detecting obsolete dictionary entries..."
export LC_ALL=C
FILES=$(find . -type f -iname '*.rst' | tr '\n' ' ') FILES=$(find . -type f -iname '*.rst' | tr '\n' ' ')
tail -n+2 postgrest.dict \ tail -n+2 postgrest.dict \
@@ -168,8 +170,6 @@ let
workingDir = "/docs"; workingDir = "/docs";
} }
'' ''
export LC_ALL=C
${python}/bin/sphinx-build --color -b linkcheck . ../.docs-build ${python}/bin/sphinx-build --color -b linkcheck . ../.docs-build
''; '';
+49 -101
View File
@@ -1,53 +1,63 @@
# generates a file to be used by the vegeta load testing tool # generates a file to be used by the vegeta load testing tool
# It includes a worst case scenario for the JWT cache:
# - all requests will have a unique JWT so no cache hits
# - all jwts have an expiration that will be long enough to be
# valid at time of request but short enough that already
# validated jwts will expire later during the loadtest run
# - the above guarantees JWT cache purging will happen
# - we want this to track resource consumption in the worst case
# And a more normal scenario where non-expiring JWTs are picked
# from an array
import time import time
import hmac
import hashlib
import base64
import json
import argparse import argparse
import sys import sys
import random import random
import jwt
import jwcrypto.jwk as jwk
from typing import Optional
from pathlib import Path
SECRET = b"reallyreallyreallyreallyverysafe"
URL = "http://postgrest" URL = "http://postgrest"
JWT_DURATION = 120
secret_key = b"reallyreallyreallyreallyverysafe" TOTAL_TARGETS = 50000 # tuned by hand to reduce result variance
key = jwk.JWK.generate(kty="RSA", size=4096)
private_key = jwt.algorithms.RSAAlgorithm.from_jwk(key.export_private())
public_key = key.export_public()
def generate_jwt(now: int, exp_inc: Optional[int], is_hs: bool) -> str: def base64url_encode(data: bytes) -> str:
"""Generate an HS256 or RS256 JWT""" """URL-safe Base64 encode without padding."""
return base64.urlsafe_b64encode(data).rstrip(b"=").decode("ascii")
def generate_jwt(exp_inc: int) -> str:
"""Generate an HS256 JWT"""
# Header & payload
header = {"alg": "HS256", "typ": "JWT"}
now = int(time.time())
payload = { payload = {
"sub": f"user_{random.getrandbits(32)}", "sub": f"user_{random.getrandbits(32)}",
"iat": now, "iat": now,
"exp": now + exp_inc,
"role": "postgrest_test_author", "role": "postgrest_test_author",
} }
if exp_inc is not None: # Encode to JSON and then to Base64URL
payload["exp"] = now + exp_inc header_b = json.dumps(header, separators=(",", ":")).encode()
payload_b = json.dumps(payload, separators=(",", ":")).encode()
header_b64 = base64url_encode(header_b)
payload_b64 = base64url_encode(payload_b)
k = secret_key if is_hs else private_key # Sign (HMACSHA256) the "<header>.<payload>" string
alg = "HS256" if is_hs else "RS256" signing_input = f"{header_b64}.{payload_b64}".encode()
return jwt.encode(payload, k, alg) signature = hmac.new(SECRET, signing_input, hashlib.sha256).digest()
signature_b64 = base64url_encode(signature)
return f"{header_b64}.{payload_b64}.{signature_b64}"
def append_targets(lines: list[str], token: str): # We want to ensure 401 Unauthorized responses don't happen during
lines.append(f"OPTIONS {URL}/authors_only") # JWT validation, this can happen when the jwt `exp` is too short.
lines.append(f"Authorization: Bearer {token}") # At the same time, we want to ensure the `exp` is not too big,
lines.append("") # blank line to separate requests # so expires will occur and postgREST will have to clean cached expired JWTs.
def estimate_adequate_jwt_exp_increase(iteration: int) -> int:
# estimated time takes to build and run postgrest itself
build_run_postgrest_time = 2
# estimated time it takes to generate the targets file
file_generation_time = TOTAL_TARGETS // (10**-5)
# estimated exp time so some JWTs will expire
dynamic_exp_inc = iteration // 1000
return build_run_postgrest_time + file_generation_time + dynamic_exp_inc
def main(): def main():
@@ -58,77 +68,16 @@ def main():
"output", "output",
help="Path to write the generated targets file", help="Path to write the generated targets file",
) )
parser.add_argument(
"--worst",
dest="worst",
action=argparse.BooleanOptionalAction,
default=False,
help="Generate worst case targets for a JWT cache",
)
parser.add_argument(
"--rsa",
dest="jwk_path",
metavar="JWK_PATH",
type=Path,
default=None,
help="Path for generating a RSA JWK file to sign tokens with",
)
args = parser.parse_args() args = parser.parse_args()
is_hs = args.jwk_path is None lines = []
nsamples = 1000
if is_hs:
ntargets = 200000
else:
# The asymmetric targets take too long to compute so we reduce them
ntargets = 50000
if not is_hs:
try:
with open(args.jwk_path, "w") as jwk:
jwk.write(public_key)
print(f"Created {args.jwk_path} file containing the RSA JWK")
except IOError as e:
print(f"Error writing to {args.jwk_path}: {e}", file=sys.stderr)
sys.exit(1)
print(f"Generating {ntargets} targets...")
start_time = time.time() start_time = time.time()
now = int(start_time) for i in range(TOTAL_TARGETS):
token = generate_jwt(estimate_adequate_jwt_exp_increase(i))
lines = [] lines.append(f"OPTIONS {URL}/authors_only")
lines.append(f"Authorization: Bearer {token}")
# We want to ensure 401 Unauthorized responses don't happen during lines.append("") # blank line to separate requests
# JWT validation, this can happen when the jwt `exp` is too short.
# At the same time, we want to ensure the `exp` is not too big,
# so expires will occur and postgREST needs to
# clean cached expired JWTs
if args.worst:
# estimated time takes to build and run postgrest itself
build_run_postgrest_time = 2
# estimated time it takes to generate the targets file
# the division numbers are tuned by hand
if is_hs: # hs generation is much faster
gen_time = ntargets // 66666
else: # asymmetric is slower so the time is higher
gen_time = ntargets // 220
# estimated exp time so some JWTs will expire
inc = build_run_postgrest_time + gen_time
for i in range(ntargets):
token = generate_jwt(now, inc + i // 1000, is_hs)
append_targets(lines, token)
else:
tokens = [generate_jwt(now, None, is_hs) for _ in range(nsamples)]
for i in range(ntargets):
token = random.choice(tokens)
append_targets(lines, token)
try: try:
with open(args.output, "w") as f: with open(args.output, "w") as f:
@@ -138,8 +87,7 @@ def main():
sys.exit(1) sys.exit(1)
elapsed = time.time() - start_time elapsed = time.time() - start_time
print(f"Created {ntargets} targets", end=" ") print(f"Created {TOTAL_TARGETS} targets in {args.output} ({elapsed:.2f}s)")
print(f"in {args.output} ({elapsed:.2f}s)")
if __name__ == "__main__": if __name__ == "__main__":
-60
View File
@@ -1,60 +0,0 @@
{ buildToolbox
, checkedShellScript
, commitlint
, writeText
}:
let
# Rules format: [<severity>, <"always"/"never">, <value>]
commitlintConfig = writeText "commitlint.config.mjs" ''
export default {
rules: {
"type-enum": [2, "always", [
'add', // Add a new feature
'amend', // To amend an unrealease commit
'change', // Breaking changes
'chore', // Update sponsors, changelog, readme etc
'ci', // CI configuration files and scripts
'docs', // Documentation
'fix', // Bug fix
'nix', // Related to Nix
'perf', // Performance improvements
'refactor', // Refactoring code
'remove', // Remove a feature or fix
'test', // Adding tests
]],
'subject-case': [2, 'never', ['pascal-case', 'start-case']],
'subject-empty': [2, 'never'],
'subject-full-stop': [2, 'never', '.'],
'subject-max-length': [2, 'always', 80],
'subject-min-length': [2, 'always', 5],
'scope-case': [2, 'always', 'lower-case'],
'body-leading-blank': [2, 'always'],
},
};
'';
commitCheck =
checkedShellScript
{
name = "postgrest-commitlint";
docs = "Script to validate commit messages";
workingDir = "/";
args = [
"ARG_OPTIONAL_SINGLE([from],, [commit ref start from], [main])"
"ARG_OPTIONAL_SINGLE([to],, [commit ref end at], [HEAD])"
];
}
''
# Run commitlint with the given configuration
${commitlint}/bin/commitlint --config ${commitlintConfig} --from "$_arg_from" --to "$_arg_to"
'';
in
buildToolbox
{
name = "postgrest-commitlint";
tools = { inherit commitCheck; };
}
+23 -85
View File
@@ -41,9 +41,8 @@ let
args = [ args = [
"ARG_OPTIONAL_SINGLE([output], [o], [Filename to dump json output to], [./loadtest/result.bin])" "ARG_OPTIONAL_SINGLE([output], [o], [Filename to dump json output to], [./loadtest/result.bin])"
"ARG_OPTIONAL_SINGLE([testdir], [t], [Directory to load tests and fixtures from], [./test/load])" "ARG_OPTIONAL_SINGLE([testdir], [t], [Directory to load tests and fixtures from], [./test/load])"
"ARG_OPTIONAL_SINGLE([kind], [k], [Kind of loadtest], [mixed])" "ARG_OPTIONAL_SINGLE([kind], [k], [Kind of loadtest (mixed: repeat mixed requests, jwt: run once over many requests with unique jwts)], [mixed])"
"ARG_TYPE_GROUP_SET([KIND], [KIND], [kind], [mixed,jwt-hs,jwt-hs-cache,jwt-hs-cache-worst,jwt-rsa,jwt-rsa-cache,jwt-rsa-cache-worst])" "ARG_TYPE_GROUP_SET([KIND], [KIND], [kind], [mixed,jwt])"
"ARG_OPTIONAL_SINGLE([monitor], [m], [Monitoring file], [./loadtest/result.csv])"
"ARG_LEFTOVERS([additional vegeta arguments])" "ARG_LEFTOVERS([additional vegeta arguments])"
]; ];
workingDir = "/"; workingDir = "/";
@@ -59,67 +58,35 @@ let
export PGRST_DB_TX_END="rollback-allow-override" export PGRST_DB_TX_END="rollback-allow-override"
export PGRST_LOG_LEVEL="crit" export PGRST_LOG_LEVEL="crit"
export PGRST_JWT_SECRET="reallyreallyreallyreallyverysafe" export PGRST_JWT_SECRET="reallyreallyreallyreallyverysafe"
# set previous PGRST_JWT_CACHE_MAX_LIFETIME configuration so that
# load test works across branches
# TODO clean once PGRST_JWT_CACHE_MAX_ENTRIES merged and released
export PGRST_JWT_CACHE_MAX_LIFETIME="86400" export PGRST_JWT_CACHE_MAX_LIFETIME="86400"
mkdir -p "$(dirname "$_arg_output")" mkdir -p "$(dirname "$_arg_output")"
abs_output="$(realpath "$_arg_output")" abs_output="$(realpath "$_arg_output")"
case "$_arg_kind" in case "$_arg_kind" in
jwt-hs) jwt)
${genTargetsHS} "$_arg_testdir"/gen_targets.http
export PGRST_JWT_CACHE_MAX_ENTRIES="0"
export PGRST_JWT_CACHE_MAX_LIFETIME="0"
;;
jwt-hs-cache) ${genTargets} "$_arg_testdir"/gen_targets.http
${genTargetsHS} "$_arg_testdir"/gen_targets.http
;;
jwt-hs-cache-worst) # shellcheck disable=SC2145
${genTargetsHS} --worst "$_arg_testdir"/gen_targets.http ${withTools.withPg} -f "$_arg_testdir"/fixtures.sql \
;; ${withTools.withPgrst} \
sh -c "cd \"$_arg_testdir\" && ${runner} -lazy -targets gen_targets.http -output \"$abs_output\" \"''${_arg_leftovers[@]}\""
jwt-rsa) ${vegeta}/bin/vegeta report -type=text "$_arg_output"
${genTargetsHS} --rsa="$_arg_testdir"/gen_jwk.json "$_arg_testdir"/gen_targets.http
export PGRST_JWT_CACHE_MAX_ENTRIES="0"
export PGRST_JWT_CACHE_MAX_LIFETIME="0"
export PGRST_JWT_SECRET="@$_arg_testdir/gen_jwk.json"
;;
jwt-rsa-cache)
${genTargetsHS} --rsa="$_arg_testdir"/gen_jwk.json "$_arg_testdir"/gen_targets.http
export PGRST_JWT_SECRET="@$_arg_testdir/gen_jwk.json"
;;
jwt-rsa-cache-worst)
${genTargetsHS} --worst --rsa="$_arg_testdir"/gen_jwk.json "$_arg_testdir"/gen_targets.http
export PGRST_JWT_SECRET="@$_arg_testdir/gen_jwk.json"
;; ;;
*) *)
# shellcheck disable=SC2145
${withTools.withPg} -f "$_arg_testdir"/fixtures.sql \
${withTools.withSlowPg} \
${withTools.withPgrst} \
${withTools.withSlowPgrst} \
sh -c "cd \"$_arg_testdir\" && ${runner} -targets targets.http -output \"$abs_output\" \"''${_arg_leftovers[@]}\""
${vegeta}/bin/vegeta report -type=text "$_arg_output"
;; ;;
esac esac
if [ "$_arg_kind" == "mixed" ]; then
# shellcheck disable=SC2145
${withTools.withPg} -f "$_arg_testdir"/fixtures.sql \
${withTools.withSlowPg} \
${withTools.withPgrst} -m "$_arg_monitor" \
${withTools.withSlowPgrst} \
sh -c "cd \"$_arg_testdir\" && \
${runner} -targets targets.http -output \"$abs_output\" \"''${_arg_leftovers[@]}\""
else
# shellcheck disable=SC2145
${withTools.withPg} -f "$_arg_testdir"/fixtures.sql \
${withTools.withPgrst} -m "$_arg_monitor" \
sh -c "cd \"$_arg_testdir\" && \
${runner} -lazy -targets gen_targets.http -output \"$abs_output\" \"''${_arg_leftovers[@]}\""
fi
${vegeta}/bin/vegeta report -type=text "$_arg_output"
''; '';
loadtestAgainst = loadtestAgainst =
@@ -148,12 +115,11 @@ let
workingDir = "/"; workingDir = "/";
} }
'' ''
# run loadtest for every target
for tgt in "''${_arg_target[@]}"; do for tgt in "''${_arg_target[@]}"; do
cat << EOF cat << EOF
Running "$_arg_kind" loadtest on "$tgt"... Running loadtest on "$tgt"...
EOF EOF
@@ -162,7 +128,7 @@ let
# Save the results in the current working tree, too, # Save the results in the current working tree, too,
# otherwise they'd be lost in the temporary working tree # otherwise they'd be lost in the temporary working tree
# created by withTools.withGit. # created by withTools.withGit.
${withTools.withGit} "$tgt" ${loadtest} -k "$_arg_kind" -m "$PWD/loadtest/$tgt.csv" --output "$PWD/loadtest/$tgt.bin" --testdir "$PWD/test/load" ${withTools.withGit} "$tgt" ${loadtest} -k "$_arg_kind" --output "$PWD/loadtest/$tgt.bin" --testdir "$PWD/test/load"
cat << EOF cat << EOF
@@ -172,15 +138,13 @@ let
done done
# run loadtest once on HEAD
cat << EOF cat << EOF
Running "$_arg_kind" loadtest on HEAD... Running loadtest on HEAD...
EOF EOF
${loadtest} -k "$_arg_kind" -m "$PWD/loadtest/head.csv" --output "$PWD/loadtest/head.bin" --testdir "$PWD/test/load" ${loadtest} -k "$_arg_kind" --output "$PWD/loadtest/head.bin" --testdir "$PWD/test/load"
cat << EOF cat << EOF
@@ -217,7 +181,6 @@ let
pd.read_json(sys.stdin) \ pd.read_json(sys.stdin) \
.set_index('param') \ .set_index('param') \
.drop(['branch', 'earliest', 'end', 'latest']) \ .drop(['branch', 'earliest', 'end', 'latest']) \
.fillna("") \
.convert_dtypes() \ .convert_dtypes() \
.to_markdown(sys.stdout, floatfmt='.0f') .to_markdown(sys.stdout, floatfmt='.0f')
''; '';
@@ -228,41 +191,16 @@ let
{ {
name = "postgrest-loadtest-report"; name = "postgrest-loadtest-report";
docs = "Create a report of all loadtest reports as markdown."; docs = "Create a report of all loadtest reports as markdown.";
args = [
"ARG_OPTIONAL_SINGLE([group], [g], [Marker to group results])"
];
workingDir = "/"; workingDir = "/";
} }
'' ''
marker=''${_arg_group:+"($_arg_group)"}
echo -e "## Loadtest results $marker\n"
find loadtest -type f -iname '*.bin' -exec ${reporter} {} \; \ find loadtest -type f -iname '*.bin' -exec ${reporter} {} \; \
| ${jq}/bin/jq '[paths(scalars) as $path | {param: $path | join("."), (.branch): getpath($path)}]' \ | ${jq}/bin/jq '[paths(scalars) as $path | {param: $path | join("."), (.branch): getpath($path)}]' \
| ${jq}/bin/jq --slurp 'flatten | group_by(.param) | map(add)' \ | ${jq}/bin/jq --slurp 'flatten | group_by(.param) | map(add)' \
| ${toMarkdown} | ${toMarkdown}
echo -e "\n\n## Loadtest elapsed seconds vs CPU/MEM usage $marker\n"
find loadtest -type f -iname '*.csv' \
| sort -m \
| ${mergeMonitorResults}
''; '';
genTargetsHS = genTargets = writers.writePython3 "postgrest-gen-loadtest-targets" { } (builtins.readFile ./generate_targets.py);
writers.writePython3 "postgrest-gen-loadtest-targets-hs"
{
libraries = [ python3Packages.pyjwt python3Packages.jwcrypto ];
}
(builtins.readFile ./generate_targets.py);
mergeMonitorResults =
writers.writePython3 "postgrest-merge-monitor-results"
{
libraries = [ python3Packages.pandas python3Packages.tabulate ];
}
(builtins.readFile ./merge_monitor_result.py);
in in
buildToolbox { buildToolbox {
name = "postgrest-loadtest"; name = "postgrest-loadtest";
-44
View File
@@ -1,44 +0,0 @@
import os
import sys
import pandas as pd
KEY = "Elapsed seconds"
BASE_METRICS = ["CPU (%)", "Real (MB)"]
branch_order = []
merged = None
paths = [p.strip() for p in sys.stdin.read().split() if p.strip()]
for csv_path in paths:
# br is branch (variable shortened to pass linter)
br = os.path.splitext(os.path.basename(csv_path))[0]
branch_order.append(br)
df = pd.read_csv(csv_path)
if KEY not in df.columns:
sys.exit(f"{csv_path} is missing the {KEY} column")
for m in BASE_METRICS:
if m not in df.columns:
sys.exit(f"Error: '{csv_path}' missing required column '{m}'.")
# add branch marker to every metric column
df = df.rename(columns={c: f"{c} [{br}]" for c in df.columns if c != KEY})
# outer join so missing rows appear
merged = df if merged is None else merged.merge(df, on=KEY, how="outer")
# Re-order columns so related metrics are adjacent
ordered_cols = [KEY]
for metric in BASE_METRICS:
for br in branch_order:
col_name = f"{metric} [{br}]"
if col_name in merged.columns:
ordered_cols.append(col_name)
merged = merged[ordered_cols]
# replace nan with empty string
merged = merged.fillna("")
merged.to_markdown(sys.stdout, index=False, tablefmt="github")
-57
View File
@@ -1,57 +0,0 @@
# Monitor a process pid with psutil and emits a CSV.
import sys
import time
import psutil
import pandas as pd
KEY = "Elapsed seconds"
BASE_METRICS = ["CPU (%)", "Real (MB)"]
SAMPLE_INTERVAL_SECS = 1
if len(sys.argv) != 2 or not sys.argv[1].isdigit():
sys.exit(f"Usage: {sys.argv[0]} <PID>")
pid = int(sys.argv[1])
try:
proc = psutil.Process(pid)
except psutil.NoSuchProcess:
sys.exit(f"Error: process {pid} not found.")
print(f"Starting monitoring of {pid} pid", file=sys.stderr)
records = []
start = time.time()
# ignore first result as per docs recommendation
# https://psutil.readthedocs.io/en/latest/#psutil.cpu_percent
proc.cpu_percent(None)
while True:
try:
if not proc.is_running():
break
time.sleep(SAMPLE_INTERVAL_SECS)
elapsed_secs = int(time.time() - start)
cpu = proc.cpu_percent(None)
meminfo = proc.memory_info()
bytes_in_MB = 1024**2
rss_mb = meminfo.rss / bytes_in_MB
records.append(
[
str(elapsed_secs),
f"{cpu:.3f}",
f"{rss_mb:.3f}",
]
)
except psutil.NoSuchProcess:
break
end = time.time()
total_time = end - start
print(f"Finished {pid} pid monitoring in {total_time:.3f}", file=sys.stderr)
cols = [KEY] + BASE_METRICS
df = pd.DataFrame(records, columns=cols, dtype=str)
df.to_csv(sys.stdout, index=False)
-28
View File
@@ -1,28 +0,0 @@
{ buildToolbox
, checkedShellScript
}:
# Utility script for pinning the latest stable version of Nixpkgs.
# Instead of running `nix flake update` manually, we run this script
# to also pin readthedocs dependencies at the same time.
let
upgrade =
checkedShellScript
{
name = "postgrest-nixpkgs-upgrade";
docs = "Pin the newest version of Nixpkgs.";
workingDir = "/";
}
''
nix flake update
echo "# This file is auto-generated by postgrest-nixpkgs-upgrade" > docs/requirements.txt
cat "$(nix-build -A docs.requirements)" >> docs/requirements.txt
'';
in
buildToolbox
{
name = "postgrest-nixpkgs";
tools = { inherit upgrade; };
}
+24 -17
View File
@@ -20,24 +20,27 @@ let
git diff --exit-code HEAD postgrest.cabal > /dev/null git diff --exit-code HEAD postgrest.cabal > /dev/null
trap "" ERR trap "" ERR
# TODO: Support C+D bumps when implementing hackage releases
bump () { bump () {
current_version="$(grep -oP '^version:\s*\K.*' postgrest.cabal)" current_version="$(grep -oP '^version:\s*\K.*' postgrest.cabal)"
# shellcheck disable=SC2034 # shellcheck disable=SC2034
IFS=. read -r A B C D <<< "$current_version" IFS=. read -r major minor patch <<< "$current_version"
echo "Current version is $current_version" echo "Current version is $current_version"
case "$1" in case "$1" in
A) major)
new_version="$((A+1)).0" new_version="$((major+1)).0.0"
new_docs_version="$((A+1))" new_docs_version="$((major+1)).0"
;; ;;
B) minor)
new_version="$A.$((B+1))" new_version="$major.$((minor+1)).0"
new_docs_version="$A" new_docs_version="$major.$((minor+1))"
;;
patch)
new_version="$major.$minor.$((patch+1))"
new_docs_version="$major.$minor"
;; ;;
devel) devel)
new_version="$((A+1))" new_version="$major.$((minor+1))"
new_docs_version="devel" new_docs_version="devel"
;; ;;
esac esac
@@ -52,9 +55,13 @@ let
today_date_for_changelog="$(date '+%Y-%m-%d')" today_date_for_changelog="$(date '+%Y-%m-%d')"
if [[ "$current_branch" == "main" ]]; then if [[ "$current_branch" == "main" ]]; then
bump A if [[ "$_arg_major" == "on" ]]; then
bump major
else
bump minor
fi
else else
bump B bump patch
fi fi
echo "Updating CHANGELOG.md ..." echo "Updating CHANGELOG.md ..."
@@ -62,19 +69,19 @@ let
git add CHANGELOG.md > /dev/null git add CHANGELOG.md > /dev/null
echo "Committing ..." echo "Committing ..."
git commit -m "chore: bump version to $new_version" > /dev/null git commit -m "bump version to $new_version" > /dev/null
if [[ "$current_branch" == "main" ]]; then if [[ "$current_branch" == "main" ]]; then
bump devel bump devel
# The order of operations is important here: # The order of operations is important here:
# - bump devel is run and $A is upated to the new version # - bump devel is run and $major is upated to the new version
# - the branch is created with the new A, but the commit before the devel bump # - the branch is created with the new major, but the commit before the devel bump
# - the devel bump is committed # - the devel bump is committed
git branch "v$A" git branch -f "v$major"
echo "Committing (devel bump)..." echo "Committing (devel bump)..."
git commit -m "chore: bump version to $new_version" > /dev/null git commit -m "bump version to $new_version" > /dev/null
fi fi
trap "echo Remote not found. Please push manually ..." ERR trap "echo Remote not found. Please push manually ..." ERR
@@ -83,7 +90,7 @@ let
if [[ "$current_branch" == "main" ]]; then if [[ "$current_branch" == "main" ]]; then
push1="git push $remote $current_branch" push1="git push $remote $current_branch"
push2="git push $remote v$A" push2="git push $remote v$major --force"
else else
push1="git push $remote $current_branch" push1="git push $remote $current_branch"
push2="" push2=""
+1 -26
View File
@@ -7,12 +7,9 @@
, hlint , hlint
, hsie , hsie
, nixpkgs-fmt , nixpkgs-fmt
, python3Packages
, ruff
, silver-searcher , silver-searcher
, statix , statix
, stylish-haskell , stylish-haskell
, writeText
}: }:
let let
style = style =
@@ -52,20 +49,6 @@ let
${git}/bin/git diff-index --exit-code HEAD -- '*.hs' '*.lhs' '*.nix' '*.py' ${git}/bin/git diff-index --exit-code HEAD -- '*.hs' '*.lhs' '*.nix' '*.py'
''; '';
hlintConfig = writeText "hlintConfig.yml" ''
# Arguments passed to hlint
- arguments: [-j, -XQuasiQuotes, -XNoPatternSynonyms]
# Warnings
- warn: { lhs: "a == a", rhs: "True", note: "This comparison always evaluates to True" }
- warn: { lhs: "a /= a", rhs: "False", note: "This comparison always evaluates to False" }
- warn: { lhs: "a < a", rhs: "False", note: "This comparison always evaluates to False" }
- warn: { lhs: "a > a", rhs: "False", note: "This comparison always evaluates to False" }
- warn: { lhs: "a <= a", rhs: "True", note: "This comparison always evaluates to True" }
- warn: { lhs: "a >= a", rhs: "True", note: "This comparison always evaluates to True" }
'';
lint = lint =
checkedShellScript checkedShellScript
{ {
@@ -80,21 +63,13 @@ let
echo "Scanning nix files for unused code..." echo "Scanning nix files for unused code..."
${deadnix}/bin/deadnix -f ${deadnix}/bin/deadnix -f
# ruff has gaps in scanning for unused code, so we use vulture
echo "Scanning python files for unused code..."
${silver-searcher}/bin/ag -l --vimgrep -g '\.l?py$' . \
| xargs ${python3Packages.vulture}/bin/vulture --exclude docs/conf.py
echo "Linting python files..."
${ruff}/bin/ruff check .
echo "Checking consistency of import aliases in Haskell code..." echo "Checking consistency of import aliases in Haskell code..."
${hsie} check-aliases main src ${hsie} check-aliases main src
echo "Linting Haskell files..." echo "Linting Haskell files..."
# --vimgrep fixes a bug in ag: https://github.com/ggreer/the_silver_searcher/issues/753 # --vimgrep fixes a bug in ag: https://github.com/ggreer/the_silver_searcher/issues/753
${silver-searcher}/bin/ag -l --vimgrep -g '\.l?hs$' . \ ${silver-searcher}/bin/ag -l --vimgrep -g '\.l?hs$' . \
| xargs ${hlint}/bin/hlint --hint=${hlintConfig} | xargs ${hlint}/bin/hlint -X QuasiQuotes -X NoPatternSynonyms
''; '';
in in
+17 -31
View File
@@ -10,7 +10,6 @@
, hostPlatform , hostPlatform
, jq , jq
, lib , lib
, nginx
, postgrest , postgrest
, python3 , python3
, runtimeShell , runtimeShell
@@ -29,24 +28,11 @@ let
withEnv = postgrest.env; withEnv = postgrest.env;
} }
'' ''
${cabal-install}/bin/cabal v2-update
${withTools.withPg} -f test/spec/fixtures/load.sql \ ${withTools.withPg} -f test/spec/fixtures/load.sql \
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:spec -- "''${_arg_leftovers[@]}" ${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:spec -- "''${_arg_leftovers[@]}"
''; '';
testObservability =
checkedShellScript
{
name = "postgrest-test-observability";
docs = "Run the Haskell observability test suite.";
args = [ "ARG_LEFTOVERS([hspec arguments])" ];
workingDir = "/";
withEnv = postgrest.env;
}
''
${withTools.withPg} -f test/observability/fixtures/load.sql \
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:observability -- "''${_arg_leftovers[@]}"
'';
testDoctests = testDoctests =
checkedShellScript checkedShellScript
{ {
@@ -56,6 +42,7 @@ let
withEnv = postgrest.env; withEnv = postgrest.env;
} }
'' ''
${cabal-install}/bin/cabal v2-update
# This makes nix-env -iA tests.doctests.bin work. # This makes nix-env -iA tests.doctests.bin work.
export NIX_GHC=${postgrest.env.NIX_GHC} export NIX_GHC=${postgrest.env.NIX_GHC}
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:doctests ${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:doctests
@@ -70,6 +57,7 @@ let
withEnv = postgrest.env; withEnv = postgrest.env;
} }
'' ''
${cabal-install}/bin/cabal v2-update
${withTools.withPg} -f test/spec/fixtures/load.sql \ ${withTools.withPg} -f test/spec/fixtures/load.sql \
${runtimeShell} -c " \ ${runtimeShell} -c " \
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:spec && \ ${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:spec && \
@@ -95,11 +83,11 @@ let
args = [ "ARG_LEFTOVERS([pytest arguments])" ]; args = [ "ARG_LEFTOVERS([pytest arguments])" ];
workingDir = "/"; workingDir = "/";
withEnv = postgrest.env; withEnv = postgrest.env;
withPath = [ nginx ];
} }
'' ''
${cabal-install}/bin/cabal v2-update
${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest ${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest
${cabal-install}/bin/cabal v2-exec -- ${withTools.withPg} -f test/io/fixtures/load.sql \ ${cabal-install}/bin/cabal v2-exec -- ${withTools.withPg} -f test/io/fixtures.sql \
${ioTestPython}/bin/pytest --ignore=test/io/test_big_schema.py --ignore=test/io/test_replica.py -v test/io "''${_arg_leftovers[@]}" ${ioTestPython}/bin/pytest --ignore=test/io/test_big_schema.py --ignore=test/io/test_replica.py -v test/io "''${_arg_leftovers[@]}"
''; '';
@@ -113,8 +101,9 @@ let
withEnv = postgrest.env; withEnv = postgrest.env;
} }
'' ''
${cabal-install}/bin/cabal v2-update
${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest ${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest
${cabal-install}/bin/cabal v2-exec -- ${withTools.withPg} -f test/io/fixtures/big_schema.sql \ ${cabal-install}/bin/cabal v2-exec -- ${withTools.withPg} -f test/io/big_schema.sql \
${ioTestPython}/bin/pytest -v test/io/test_big_schema.py "''${_arg_leftovers[@]}" ${ioTestPython}/bin/pytest -v test/io/test_big_schema.py "''${_arg_leftovers[@]}"
''; '';
@@ -128,8 +117,9 @@ let
withEnv = postgrest.env; withEnv = postgrest.env;
} }
'' ''
${cabal-install}/bin/cabal v2-update
${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest ${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest
${cabal-install}/bin/cabal v2-exec -- ${withTools.withPg} --replica -f test/io/fixtures/replica.sql \ ${cabal-install}/bin/cabal v2-exec -- ${withTools.withPg} --replica -f test/io/replica.sql \
${ioTestPython}/bin/pytest -v test/io/test_replica.py "''${_arg_leftovers[@]}" ${ioTestPython}/bin/pytest -v test/io/test_replica.py "''${_arg_leftovers[@]}"
''; '';
@@ -143,6 +133,7 @@ let
withPath = [ jq ]; withPath = [ jq ];
} }
'' ''
${cabal-install}/bin/cabal v2-update
${withTools.withPg} -f test/spec/fixtures/load.sql \ ${withTools.withPg} -f test/spec/fixtures/load.sql \
${cabal-install}/bin/cabal v2-run ${devCabalOptions} --verbose=0 -- \ ${cabal-install}/bin/cabal v2-run ${devCabalOptions} --verbose=0 -- \
postgrest --dump-schema postgrest --dump-schema
@@ -158,7 +149,6 @@ let
redirectTixFiles = false; redirectTixFiles = false;
withEnv = postgrest.env; withEnv = postgrest.env;
withTmpDir = true; withTmpDir = true;
withPath = [ nginx ];
} }
( (
# required for `hpc markup` in CI; glibcLocales is not available e.g. on Darwin # required for `hpc markup` in CI; glibcLocales is not available e.g. on Darwin
@@ -172,7 +162,8 @@ let
rm -rf coverage/* rm -rf coverage/*
# build once before running all the tests # build once before running all the tests
${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest lib:postgrest test:spec test:observability ${cabal-install}/bin/cabal v2-update
${cabal-install}/bin/cabal v2-build ${devCabalOptions} exe:postgrest lib:postgrest test:spec
( (
trap 'echo Found dead code: Check file list above.' ERR ; trap 'echo Found dead code: Check file list above.' ERR ;
@@ -181,31 +172,26 @@ let
# collect all tests # collect all tests
HPCTIXFILE="$tmpdir"/io.tix \ HPCTIXFILE="$tmpdir"/io.tix \
${withTools.withPg} -f test/io/fixtures/load.sql \ ${withTools.withPg} -f test/io/fixtures.sql \
${cabal-install}/bin/cabal v2-exec ${devCabalOptions} -- ${ioTestPython}/bin/pytest --ignore=test/io/test_big_schema.py --ignore=test/io/test_replica.py -v test/io ${cabal-install}/bin/cabal v2-exec ${devCabalOptions} -- ${ioTestPython}/bin/pytest --ignore=test/io/test_big_schema.py --ignore=test/io/test_replica.py -v test/io
HPCTIXFILE="$tmpdir"/big_schema.tix \ HPCTIXFILE="$tmpdir"/big_schema.tix \
${withTools.withPg} -f test/io/fixtures/big_schema.sql \ ${withTools.withPg} -f test/io/big_schema.sql \
${cabal-install}/bin/cabal v2-exec ${devCabalOptions} -- ${ioTestPython}/bin/pytest -v test/io/test_big_schema.py ${cabal-install}/bin/cabal v2-exec ${devCabalOptions} -- ${ioTestPython}/bin/pytest -v test/io/test_big_schema.py
HPCTIXFILE="$tmpdir"/replica.tix \ HPCTIXFILE="$tmpdir"/replica.tix \
${withTools.withPg} --replica -f test/io/fixtures/replica.sql \ ${withTools.withPg} --replica -f test/io/replica.sql \
${cabal-install}/bin/cabal v2-exec ${devCabalOptions} -- ${ioTestPython}/bin/pytest -v test/io/test_replica.py ${cabal-install}/bin/cabal v2-exec ${devCabalOptions} -- ${ioTestPython}/bin/pytest -v test/io/test_replica.py
HPCTIXFILE="$tmpdir"/spec.tix \ HPCTIXFILE="$tmpdir"/spec.tix \
${withTools.withPg} -f test/spec/fixtures/load.sql \ ${withTools.withPg} -f test/spec/fixtures/load.sql \
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:spec ${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:spec
HPCTIXFILE="$tmpdir"/observability.tix \
${withTools.withPg} -f test/observability/fixtures/load.sql \
${cabal-install}/bin/cabal v2-run ${devCabalOptions} test:observability
# Note: No coverage for doctests, as doctests leverage GHCi and GHCi does not support hpc # Note: No coverage for doctests, as doctests leverage GHCi and GHCi does not support hpc
# collect all the tix files # collect all the tix files
${ghc}/bin/hpc sum --union --exclude=Paths_postgrest --output="$tmpdir"/tests.tix \ ${ghc}/bin/hpc sum --union --exclude=Paths_postgrest --output="$tmpdir"/tests.tix \
"$tmpdir"/io*.tix "$tmpdir"/big_schema*.tix "$tmpdir"/replica*.tix "$tmpdir"/spec.tix \ "$tmpdir"/io*.tix "$tmpdir"/big_schema*.tix "$tmpdir"/replica*.tix "$tmpdir"/spec.tix
"$tmpdir"/observability.tix
# prepare the overlay # prepare the overlay
${ghc}/bin/hpc overlay --output="$tmpdir"/overlay.tix test/coverage.overlay ${ghc}/bin/hpc overlay --output="$tmpdir"/overlay.tix test/coverage.overlay
@@ -260,6 +246,7 @@ let
withPath = [ curl ]; withPath = [ curl ];
} }
'' ''
${cabal-install}/bin/cabal v2-update
${cabal-install}/bin/cabal --builddir="dist-prof" v2-build --enable-profiling --disable-shared exe:postgrest ${cabal-install}/bin/cabal --builddir="dist-prof" v2-build --enable-profiling --disable-shared exe:postgrest
${cabal-install}/bin/cabal --builddir="dist-prof" v2-exec -- ${withTools.withPg} -f test/spec/fixtures/load.sql \ ${cabal-install}/bin/cabal --builddir="dist-prof" v2-exec -- ${withTools.withPg} -f test/spec/fixtures/load.sql \
test/memory/memory-tests.sh test/memory/memory-tests.sh
@@ -272,7 +259,6 @@ buildToolbox
tools = { tools = {
inherit inherit
testSpec testSpec
testObservability
testDoctests testDoctests
testSpecIdempotence testSpecIdempotence
testIO testIO
+6 -53
View File
@@ -5,10 +5,8 @@
, lib , lib
, postgresqlVersions , postgresqlVersions
, postgrest , postgrest
, python3Packages
, slocat , slocat
, writeText , writeText
, writers
}: }:
let let
withTmpDb = withTmpDb =
@@ -25,7 +23,7 @@ let
"ARG_OPTIONAL_SINGLE([fixtures], [f], [SQL file to load fixtures from])" "ARG_OPTIONAL_SINGLE([fixtures], [f], [SQL file to load fixtures from])"
"ARG_POSITIONAL_SINGLE([command], [Command to run])" "ARG_POSITIONAL_SINGLE([command], [Command to run])"
"ARG_LEFTOVERS([command arguments])" "ARG_LEFTOVERS([command arguments])"
"ARG_USE_ENV([PGUSER], [Postgrest_Test_Authenticator], [Authenticator PG role])" # user is written in mixed case to implicitly test that it is being properly quoted in schema cache queries "ARG_USE_ENV([PGUSER], [postgrest_test_authenticator], [Authenticator PG role])"
"ARG_USE_ENV([PGDATABASE], [postgres], [PG database name])" "ARG_USE_ENV([PGDATABASE], [postgres], [PG database name])"
"ARG_USE_ENV([PGRST_DB_SCHEMAS], [test], [Schema to expose])" "ARG_USE_ENV([PGRST_DB_SCHEMAS], [test], [Schema to expose])"
"ARG_USE_ENV([PGTZ], [utc], [Timezone to use])" "ARG_USE_ENV([PGTZ], [utc], [Timezone to use])"
@@ -46,7 +44,7 @@ let
} }
# Avoid starting multiple layers of withTmpDb, but make sure to have the last invocation # Avoid starting multiple layers of withTmpDb, but make sure to have the last invocation
# load fixtures. Otherwise postgrest-with-pg-xx postgrest-test-io would not be possible. # load fixtures. Otherwise postgrest-with-postgresql-xx postgrest-test-io would not be possible.
if ! test -v PGHOST; then if ! test -v PGHOST; then
mkdir -p "$tmpdir"/{db,socket} mkdir -p "$tmpdir"/{db,socket}
@@ -74,13 +72,7 @@ let
>> "$setuplog" >> "$setuplog"
log "Starting the database cluster..." log "Starting the database cluster..."
# Instead of listening on a local port, we will listen on a unix domain socket. # Instead of listening on a local port, we will listen on a unix domain socket.
# NOTE: unix domain socket filename name must remain under max limit.
# On Linux, it's 108 chars (including '\0' terminator)
# On MacOS, it's 104 chars
# See: https://serverfault.com/questions/641347/check-if-a-path-exceeds-maximum-for-unix-domain-socket
pg_ctl -l "$tmpdir/db.log" -w start -o "-F -c listen_addresses=\"\" -c hba_file=$HBA_FILE -k $PGHOST -c log_statement=\"all\" " \ pg_ctl -l "$tmpdir/db.log" -w start -o "-F -c listen_addresses=\"\" -c hba_file=$HBA_FILE -k $PGHOST -c log_statement=\"all\" " \
>> "$setuplog" >> "$setuplog"
@@ -335,22 +327,6 @@ let
done done
''; '';
# Broadcast SIGINT to any running postgrest instances on the host. Uses python for cross-platform compatibility.
signalPostgrest =
writers.writePython3 "postgrest-signal-int"
{ libraries = [ python3Packages.psutil ]; }
''
import psutil
import signal
for proc in psutil.process_iter(["name"]):
try:
if proc.info["name"] == "postgrest":
proc.send_signal(signal.SIGINT)
except (psutil.NoSuchProcess, psutil.AccessDenied):
continue
'';
withPgrst = withPgrst =
checkedShellScript checkedShellScript
{ {
@@ -360,7 +336,6 @@ let
[ [
"ARG_POSITIONAL_SINGLE([command], [Command to run])" "ARG_POSITIONAL_SINGLE([command], [Command to run])"
"ARG_LEFTOVERS([command arguments])" "ARG_LEFTOVERS([command arguments])"
"ARG_OPTIONAL_SINGLE([monitor], [m], [Enable CPU and memory monitoring of the PostgREST process and output to the designated file as markdown])"
]; ];
positionalCompletion = "_command"; positionalCompletion = "_command";
workingDir = "/"; workingDir = "/";
@@ -373,13 +348,12 @@ let
rm -f result rm -f result
if [ -z "''${PGRST_BUILD_CABAL:-}" ]; then if [ -z "''${PGRST_BUILD_CABAL:-}" ]; then
echo -n "Building postgrest (nix)... " echo -n "Building postgrest (nix)... "
# Using lib.getBin to also make this work with older checkouts, where .bin was not a thing, yet. nix-build -A postgrestPackage > "$tmpdir"/build.log 2>&1 || {
nix-build -E 'with import ./. {}; pkgs.lib.getBin postgrestPackage' > "$tmpdir"/build.log 2>&1 || {
echo "failed, output:" echo "failed, output:"
cat "$tmpdir"/build.log cat "$tmpdir"/build.log
exit 1 exit 1
} }
PGRST_CMD=$(echo ./result*/bin/postgrest) PGRST_CMD=./result/bin/postgrest
else else
echo -n "Building postgrest (cabal)... " echo -n "Building postgrest (cabal)... "
postgrest-build postgrest-build
@@ -387,22 +361,11 @@ let
fi fi
echo "done." echo "done."
ver=$($PGRST_CMD ${legacyConfig} --version) echo -n "Starting postgrest... "
echo -n "Starting $ver... "
$PGRST_CMD ${legacyConfig} > "$tmpdir"/run.log 2>&1 & $PGRST_CMD ${legacyConfig} > "$tmpdir"/run.log 2>&1 &
pid=$! pid=$!
# shellcheck disable=SC2317 # shellcheck disable=SC2317
cleanup() { cleanup() {
# Send INT to all postgrest processes.
# Workaround to trigger dumping postgrest.prof for postgrest-profiled-run
# Caveat: we cannot realistically limit this to the current process' tree,
# since pkill's --parent supports only direct children; therefore this
# would reap neighbor postgrest instances as well, because INT is asking
# the process to terminate too.
# TODO: consider cgroups to make this cleaner
${signalPostgrest}
kill "$pid" || true kill "$pid" || true
} }
trap cleanup EXIT trap cleanup EXIT
@@ -414,19 +377,9 @@ let
} }
echo "done." echo "done."
if [[ -n "$_arg_monitor" ]]; then
${monitorPid} "$pid" > "$_arg_monitor" &
fi
("$_arg_command" "''${_arg_leftovers[@]}") ("$_arg_command" "''${_arg_leftovers[@]}")
''; '';
monitorPid =
writers.writePython3 "postgrest-monitor-pid"
{
libraries = [ python3Packages.pandas python3Packages.tabulate python3Packages.psutil ];
}
(builtins.readFile ./monitor_pid.py);
in in
buildToolbox buildToolbox
{ {
@@ -439,7 +392,7 @@ buildToolbox
withSlowPg withSlowPg
withSlowPgrst; withSlowPgrst;
} // builtins.listToAttrs ( } // builtins.listToAttrs (
# Create a `postgrest-with-pg-` for each PostgreSQL version # Create a `postgrest-with-postgresql-` for each PostgreSQL version
builtins.map (pg: { inherit (pg) name; value = withTmpDb pg; }) postgresqlVersions builtins.map (pg: { inherit (pg) name; value = withTmpDb pg; }) postgresqlVersions
); );
# make latest withPg available for other nix files # make latest withPg available for other nix files
+12 -56
View File
@@ -1,5 +1,5 @@
name: postgrest name: postgrest
version: 14.17 version: 13.0.8
synopsis: REST API for any Postgres database synopsis: REST API for any Postgres database
description: Reads the schema of a PostgreSQL database and creates RESTful routes description: Reads the schema of a PostgreSQL database and creates RESTful routes
for tables, views, and functions, supporting all HTTP methods that security for tables, views, and functions, supporting all HTTP methods that security
@@ -26,7 +26,7 @@ tested-with:
source-repository head source-repository head
type: git type: git
location: https://github.com/PostgREST/postgrest.git location: git://github.com/PostgREST/postgrest.git
flag dev flag dev
default: False default: False
@@ -47,12 +47,9 @@ library
PostgREST.App PostgREST.App
PostgREST.AppState PostgREST.AppState
PostgREST.Auth PostgREST.Auth
PostgREST.Auth.Jwt
PostgREST.Auth.JwtCache PostgREST.Auth.JwtCache
PostgREST.Auth.Types PostgREST.Auth.Types
PostgREST.Cache.Sieve
PostgREST.CLI PostgREST.CLI
PostgREST.Client
PostgREST.Config PostgREST.Config
PostgREST.Config.Database PostgREST.Config.Database
PostgREST.Config.JSPath PostgREST.Config.JSPath
@@ -68,13 +65,11 @@ library
PostgREST.Error PostgREST.Error
PostgREST.Listener PostgREST.Listener
PostgREST.Logger PostgREST.Logger
PostgREST.MainTx
PostgREST.MediaType PostgREST.MediaType
PostgREST.Metrics PostgREST.Metrics
PostgREST.Network PostgREST.Network
PostgREST.Observation PostgREST.Observation
PostgREST.Query PostgREST.Query
PostgREST.Query.PreQuery
PostgREST.Query.QueryBuilder PostgREST.Query.QueryBuilder
PostgREST.Query.SqlFragment PostgREST.Query.SqlFragment
PostgREST.Query.Statements PostgREST.Query.Statements
@@ -88,23 +83,23 @@ library
PostgREST.ApiRequest PostgREST.ApiRequest
PostgREST.ApiRequest.Preferences PostgREST.ApiRequest.Preferences
PostgREST.ApiRequest.QueryParams PostgREST.ApiRequest.QueryParams
PostgREST.ApiRequest.Payload
PostgREST.ApiRequest.Types PostgREST.ApiRequest.Types
PostgREST.Response PostgREST.Response
PostgREST.Response.OpenAPI PostgREST.Response.OpenAPI
PostgREST.Response.GucHeader PostgREST.Response.GucHeader
PostgREST.Response.Performance PostgREST.Response.Performance
PostgREST.TimeIt
PostgREST.Version PostgREST.Version
build-depends: base >= 4.9 && < 4.20 build-depends: base >= 4.9 && < 4.20
, HTTP >= 4000.3.7 && < 4000.5 , HTTP >= 4000.3.7 && < 4000.5
, Ranged-sets >= 0.3 && < 0.5 , Ranged-sets >= 0.3 && < 0.5
, aeson >= 2.0.3 && < 2.3 , aeson >= 2.0.3 && < 2.3
, auto-update >= 0.2.7 && < 0.3 , auto-update >= 0.1.4 && < 0.3
, base64-bytestring >= 1 && < 1.3 , base64-bytestring >= 1 && < 1.3
, bytestring >= 0.10.8 && < 0.13 , bytestring >= 0.10.8 && < 0.13
, cache >= 0.1.3 && < 0.2.0
, case-insensitive >= 1.2 && < 1.3 , case-insensitive >= 1.2 && < 1.3
, cassava >= 0.4.5 && < 0.6 , cassava >= 0.4.5 && < 0.6
, clock >= 0.8.3 && < 0.9.0
, configurator-pg >= 0.2.11 && < 0.3 , configurator-pg >= 0.2.11 && < 0.3
, containers >= 0.5.7 && < 0.7 , containers >= 0.5.7 && < 0.7
, cookie >= 0.4.2 && < 0.6 , cookie >= 0.4.2 && < 0.6
@@ -117,10 +112,11 @@ library
, hasql-notifications >= 0.2.2.2 && < 0.2.3 , hasql-notifications >= 0.2.2.2 && < 0.2.3
, hasql-pool >= 1.0.1 && < 1.1 , hasql-pool >= 1.0.1 && < 1.1
, hasql-transaction >= 1.0.1 && < 1.2 , hasql-transaction >= 1.0.1 && < 1.2
, http-client >= 0.7.19 && < 0.8 , heredoc >= 0.2 && < 0.3
, http-types >= 0.12.2 && < 0.13 , http-types >= 0.12.2 && < 0.13
, insert-ordered-containers >= 0.2.2 && < 0.3 , insert-ordered-containers >= 0.2.2 && < 0.3
, jose-jwt >= 0.9.6 && < 0.10 , iproute >= 1.7.0 && < 1.8
, jose-jwt >= 0.9.6 && < 0.11
, lens >= 4.14 && < 5.4 , lens >= 4.14 && < 5.4
, lens-aeson >= 1.0.1 && < 1.3 , lens-aeson >= 1.0.1 && < 1.3
, mtl >= 2.2.2 && < 2.4 , mtl >= 2.2.2 && < 2.4
@@ -136,10 +132,11 @@ library
, regex-tdfa >= 1.2.2 && < 1.4 , regex-tdfa >= 1.2.2 && < 1.4
, retry >= 0.7.4 && < 0.10 , retry >= 0.7.4 && < 0.10
, scientific >= 0.3.4 && < 0.4 , scientific >= 0.3.4 && < 0.4
, streaming-commons >= 0.2.3.1 && < 0.3 , streaming-commons >= 0.1.1 && < 0.3
, swagger2 >= 2.4 && < 2.9 , swagger2 >= 2.4 && < 2.9
, text >= 1.2.2 && < 2.2 , text >= 1.2.2 && < 2.2
, time >= 1.6 && < 1.13 , time >= 1.6 && < 1.13
, timeit >= 2.0 && < 2.1
, unordered-containers >= 0.2.8 && < 0.3 , unordered-containers >= 0.2.8 && < 0.3
, unix-compat >= 0.5.4 && < 0.8 , unix-compat >= 0.5.4 && < 0.8
, vault >= 0.3.1.5 && < 0.4 , vault >= 0.3.1.5 && < 0.4
@@ -152,12 +149,7 @@ library
-- for unix sockets; this is tested in test/io/test_io.py. See -- for unix sockets; this is tested in test/io/test_io.py. See
-- https://github.com/kazu-yamamoto/logger/commit/3a71ca70afdbb93d4ecf0083eeba1fbbbcab3fc3 -- https://github.com/kazu-yamamoto/logger/commit/3a71ca70afdbb93d4ecf0083eeba1fbbbcab3fc3
, wai-logger >= 2.4.0 , wai-logger >= 2.4.0
, warp >= 3.4.14 && < 3.5 , warp >= 3.3.19 && < 3.5
, stm >= 2.5 && < 3
, stm-hamt >= 1.2 && < 2
, focus >= 1.0 && < 2
, some >= 1.0.4.1 && < 2
, uuid >= 1.3 && < 2
-- -fno-spec-constr may help keep compile time memory use in check, -- -fno-spec-constr may help keep compile time memory use in check,
-- see https://gitlab.haskell.org/ghc/ghc/issues/16017#note_219304 -- see https://gitlab.haskell.org/ghc/ghc/issues/16017#note_219304
-- -optP-Wno-nonportable-include-path -- -optP-Wno-nonportable-include-path
@@ -269,17 +261,15 @@ test-suite spec
, hasql-transaction >= 1.0.1 && < 1.2 , hasql-transaction >= 1.0.1 && < 1.2
, heredoc >= 0.2 && < 0.3 , heredoc >= 0.2 && < 0.3
, hspec >= 2.3 && < 2.12 , hspec >= 2.3 && < 2.12
, hspec-expectations >= 0.8.4 && < 0.9
, hspec-wai >= 0.10 && < 0.12 , hspec-wai >= 0.10 && < 0.12
, hspec-wai-json >= 0.10 && < 0.12 , hspec-wai-json >= 0.10 && < 0.12
, http-types >= 0.12.3 && < 0.13 , http-types >= 0.12.3 && < 0.13
, jose-jwt >= 0.9.6 && < 0.10 , jose-jwt >= 0.9.6 && < 0.11
, lens >= 4.14 && < 5.4 , lens >= 4.14 && < 5.4
, lens-aeson >= 1.0.1 && < 1.3 , lens-aeson >= 1.0.1 && < 1.3
, monad-control >= 1.0.1 && < 1.1 , monad-control >= 1.0.1 && < 1.1
, postgrest , postgrest
, process >= 1.4.2 && < 1.7 , process >= 1.4.2 && < 1.7
, prometheus-client >= 1.1.1 && < 1.2.0
, protolude >= 0.3.1 && < 0.4 , protolude >= 0.3.1 && < 0.4
, regex-tdfa >= 1.2.2 && < 1.4 , regex-tdfa >= 1.2.2 && < 1.4
, scientific >= 0.3.4 && < 0.4 , scientific >= 0.3.4 && < 0.4
@@ -294,40 +284,6 @@ test-suite spec
-- https://github.com/PostgREST/postgrest/issues/387 -- https://github.com/PostgREST/postgrest/issues/387
-with-rtsopts=-K33K -with-rtsopts=-K33K
test-suite observability
type: exitcode-stdio-1.0
default-language: Haskell2010
default-extensions: OverloadedStrings
QuasiQuotes
NoImplicitPrelude
hs-source-dirs: test/observability
main-is: Main.hs
other-modules: ObsHelper
Observation.JwtCache
Observation.MetricsSpec
Observation.SchemaCacheSpec
build-depends: base >= 4.9 && < 4.20
, base64-bytestring >= 1 && < 1.3
, bytestring >= 0.10.8 && < 0.13
, hasql-pool >= 1.0.1 && < 1.1
, hasql-transaction >= 1.0.1 && < 1.2
, hspec >= 2.3 && < 2.12
, hspec-expectations >= 0.8.4 && < 0.9
, hspec-wai >= 0.10 && < 0.12
, hspec-wai-json >= 0.10 && < 0.12
, http-types >= 0.12.3 && < 0.13
, jose-jwt >= 0.9.6 && < 0.10
, postgrest
, prometheus-client >= 1.1.1 && < 1.2.0
, protolude >= 0.3.1 && < 0.4
, text >= 1.2.2 && < 2.2
, wai >= 3.2.1 && < 3.3
ghc-options: -threaded -O0 -Werror -Wall -fwarn-identities
-fno-spec-constr -optP-Wno-nonportable-include-path
-fwrite-ide-info
-- https://github.com/PostgREST/postgrest/issues/387
-with-rtsopts=-K33K
test-suite doctests test-suite doctests
type: exitcode-stdio-1.0 type: exitcode-stdio-1.0
default-language: Haskell2010 default-language: Haskell2010
-2
View File
@@ -21,9 +21,7 @@ let
postgrest.cabalTools postgrest.cabalTools
postgrest.devTools postgrest.devTools
postgrest.docs postgrest.docs
postgrest.gitTools
postgrest.loadtest postgrest.loadtest
postgrest.nixpkgsTools
postgrest.release postgrest.release
postgrest.style postgrest.style
postgrest.tests postgrest.tests
+19 -25
View File
@@ -1,3 +1,5 @@
{-# LANGUAGE NamedFieldPuns #-}
module PostgREST.Admin module PostgREST.Admin
( runAdmin ( runAdmin
) where ) where
@@ -7,45 +9,37 @@ import qualified Network.HTTP.Types.Status as HTTP
import qualified Network.Wai as Wai import qualified Network.Wai as Wai
import qualified Network.Wai.Handler.Warp as Warp import qualified Network.Wai.Handler.Warp as Warp
import Control.Monad.Extra (whenJust) import Control.Monad.Extra (whenJust)
import Network.Socket hiding (addrFamily)
import Network.Socket
import Network.Socket.ByteString import Network.Socket.ByteString
import PostgREST.AppState (AppState, getConfig, getMainThreadId) import PostgREST.AppState (AppState)
import PostgREST.Config (AppConfig (..)) import PostgREST.Config (AppConfig (..))
import PostgREST.MediaType (MediaType (..), toContentType) import PostgREST.MediaType (MediaType (..), toContentType)
import PostgREST.Metrics (metricsToText) import PostgREST.Metrics (metricsToText)
import PostgREST.Network (resolveSocketToAddress) import PostgREST.Network (resolveHost)
import PostgREST.Observation (Observation (..)) import PostgREST.Observation (Observation (..))
import qualified PostgREST.AppState as AppState import qualified PostgREST.AppState as AppState
import qualified Network.Socket as NS import Protolude
import Protolude
runAdmin :: AppState -> Maybe NS.Socket -> NS.Socket -> Warp.Settings -> IO () runAdmin :: AppState -> Warp.Settings -> IO ()
runAdmin appState maybeAdminSocket socketREST settings = do runAdmin appState settings = do
conf <- getConfig appState AppConfig{configAdminServerPort} <- AppState.getConfig appState
whenJust maybeAdminSocket $ \adminSocket -> do whenJust (AppState.getSocketAdmin appState) $ \adminSocket -> do
address <- resolveSocketToAddress adminSocket host <- resolveHost adminSocket
void . forkIO $ handle onError $ observer $ AdminStartObs host configAdminServerPort
Warp.runSettingsSocket (adminServerSettings conf address) adminSocket adminApp void . forkIO $ Warp.runSettingsSocket settings adminSocket adminApp
where where
adminApp = admin appState socketREST adminApp = admin appState
observer = AppState.getObserver appState observer = AppState.getObserver appState
adminServerSettings config addr =
settings
& Warp.setBeforeMainLoop (observer $ AdminStartObs addr)
& maybe identity Warp.setPort (configAdminServerPort config)
onError ex = do
observer $ AdminServerCrashedObs ex
killThread (getMainThreadId appState) -- Admin server crash is deemed unrecoverable, so we kill postgrest
-- | PostgREST admin application -- | PostgREST admin application
admin :: AppState.AppState -> NS.Socket -> Wai.Application admin :: AppState.AppState -> Wai.Application
admin appState socketREST req respond = do admin appState req respond = do
isMainAppReachable <- isRight <$> reachMainApp socketREST isMainAppReachable <- isRight <$> reachMainApp (AppState.getSocketREST appState)
isLoaded <- AppState.isLoaded appState isLoaded <- AppState.isLoaded appState
isPending <- AppState.isPending appState isPending <- AppState.isPending appState
+162 -10
View File
@@ -6,30 +6,44 @@ Description : PostgREST functions to translate HTTP request to a domain type cal
{-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE NamedFieldPuns #-}
module PostgREST.ApiRequest module PostgREST.ApiRequest
( ApiRequest(..) ( ApiRequest(..)
, InvokeMethod(..)
, Mutation(..)
, MediaType(..)
, Action(..)
, DbAction(..)
, Payload(..)
, userApiRequest , userApiRequest
, userPreferences , userPreferences
) where ) where
import qualified Data.CaseInsensitive as CI import qualified Data.Aeson as JSON
import qualified Data.HashMap.Strict as HM import qualified Data.Aeson.Key as K
import qualified Data.List.NonEmpty as NonEmptyList import qualified Data.Aeson.KeyMap as KM
import qualified Data.Set as S import qualified Data.ByteString.Char8 as BS
import qualified Data.Text.Encoding as T import qualified Data.ByteString.Lazy as LBS
import qualified Data.CaseInsensitive as CI
import qualified Data.Csv as CSV
import qualified Data.HashMap.Strict as HM
import qualified Data.List.NonEmpty as NonEmptyList
import qualified Data.Map.Strict as M
import qualified Data.Set as S
import qualified Data.Text.Encoding as T
import qualified Data.Vector as V
import Data.Either.Combinators (mapBoth)
import Control.Arrow ((***))
import Data.Aeson.Types (emptyArray, emptyObject)
import Data.List (lookup) import Data.List (lookup)
import Data.Ranged.Ranges (emptyRange, rangeIntersection, import Data.Ranged.Ranges (emptyRange, rangeIntersection,
rangeIsEmpty) rangeIsEmpty)
import Network.HTTP.Types.Header (RequestHeaders, hCookie) import Network.HTTP.Types.Header (RequestHeaders, hCookie)
import Network.HTTP.Types.URI (parseSimpleQuery)
import Network.Wai (Request (..)) import Network.Wai (Request (..))
import Network.Wai.Parse (parseHttpAccept) import Network.Wai.Parse (parseHttpAccept)
import Web.Cookie (parseCookies) import Web.Cookie (parseCookies)
import PostgREST.ApiRequest.Payload (getPayload)
import PostgREST.ApiRequest.QueryParams (QueryParams (..)) import PostgREST.ApiRequest.QueryParams (QueryParams (..))
import PostgREST.ApiRequest.Types (Action (..), DbAction (..),
InvokeMethod (..),
Mutation (..), Payload (..),
RequestBody, Resource (..))
import PostgREST.Config (AppConfig (..), import PostgREST.Config (AppConfig (..),
OpenAPIMode (..)) OpenAPIMode (..))
import PostgREST.Config.Database (TimezoneNames) import PostgREST.Config.Database (TimezoneNames)
@@ -50,6 +64,44 @@ import qualified PostgREST.MediaType as MediaType
import Protolude import Protolude
type RequestBody = LBS.ByteString
data Payload
= ProcessedJSON -- ^ Cached attributes of a JSON payload
{ payRaw :: LBS.ByteString
-- ^ This is the raw ByteString that comes from the request body. We
-- cache this instead of an Aeson Value because it was detected that for
-- large payloads the encoding had high memory usage, see
-- https://github.com/PostgREST/postgrest/pull/1005 for more details
, payKeys :: S.Set Text
-- ^ Keys of the object or if it's an array these keys are guaranteed to
-- be the same across all its objects
}
| ProcessedUrlEncoded { payArray :: [(Text, Text)], payKeys :: S.Set Text }
| RawJSON { payRaw :: LBS.ByteString }
| RawPay { payRaw :: LBS.ByteString }
data InvokeMethod = Inv | InvRead Bool deriving Eq
data Mutation = MutationCreate | MutationDelete | MutationSingleUpsert | MutationUpdate deriving Eq
data Resource
= ResourceRelation Text
| ResourceRoutine Text
| ResourceSchema
data DbAction
= ActRelationRead {dbActQi :: QualifiedIdentifier, actHeadersOnly :: Bool}
| ActRelationMut {dbActQi :: QualifiedIdentifier, actMutation :: Mutation}
| ActRoutine {dbActQi :: QualifiedIdentifier, actInvMethod :: InvokeMethod}
| ActSchemaRead Schema Bool
data Action
= ActDb DbAction
| ActRelationInfo QualifiedIdentifier
| ActRoutineInfo QualifiedIdentifier InvokeMethod
| ActSchemaInfo
{-| {-|
Describes what the user wants to do. This data type is a Describes what the user wants to do. This data type is a
translation of the raw elements of an HTTP request into domain translation of the raw elements of an HTTP request into domain
@@ -188,3 +240,103 @@ getRanges method QueryParams{qsRanges} hdrs
-- The only emptyRange allowed is the limit zero range -- The only emptyRange allowed is the limit zero range
isInvalidRange = topLevelRange == emptyRange && not (hasLimitZero limitRange) isInvalidRange = topLevelRange == emptyRange && not (hasLimitZero limitRange)
topLevelRange = fromMaybe allRange $ HM.lookup "limit" ranges -- if no limit is specified, get all the request rows topLevelRange = fromMaybe allRange $ HM.lookup "limit" ranges -- if no limit is specified, get all the request rows
getPayload :: RequestBody -> MediaType -> QueryParams.QueryParams -> Action -> Either ApiRequestError (Maybe Payload, S.Set FieldName)
getPayload reqBody contentMediaType QueryParams{qsColumns} action = do
checkedPayload <- if shouldParsePayload then payload else Right Nothing
let cols = case (checkedPayload, columns) of
(Just ProcessedJSON{payKeys}, _) -> payKeys
(Just ProcessedUrlEncoded{payKeys}, _) -> payKeys
(Just RawJSON{}, Just cls) -> cls
_ -> S.empty
return (checkedPayload, cols)
where
payload :: Either ApiRequestError (Maybe Payload)
payload = mapBoth InvalidBody Just $ case (contentMediaType, isProc) of
(MTApplicationJSON, _) ->
if isJust columns
then Right $ RawJSON reqBody
else note "All object keys must match" . payloadAttributes reqBody
=<< if LBS.null reqBody && isProc
then Right emptyObject
else first BS.pack $
-- Drop parsing error message in favor of generic one (https://github.com/PostgREST/postgrest/issues/2344)
maybe (Left "Empty or invalid json") Right $ JSON.decode reqBody
(MTTextCSV, _) -> do
json <- csvToJson <$> first BS.pack (CSV.decodeByName reqBody)
note "All lines must have same number of fields" $ payloadAttributes (JSON.encode json) json
(MTUrlEncoded, True) ->
Right $ ProcessedUrlEncoded params (S.fromList $ fst <$> params)
(MTUrlEncoded, False) ->
let paramsMap = HM.fromList $ (identity *** JSON.String) <$> params in
Right $ ProcessedJSON (JSON.encode paramsMap) $ S.fromList (HM.keys paramsMap)
(MTTextPlain, True) -> Right $ RawPay reqBody
(MTTextXML, True) -> Right $ RawPay reqBody
(MTOctetStream, True) -> Right $ RawPay reqBody
(ct, _) -> Left $ "Content-Type not acceptable: " <> MediaType.toMime ct
shouldParsePayload = case action of
ActDb (ActRelationMut _ MutationDelete) -> False
ActDb (ActRelationMut _ _) -> True
ActDb (ActRoutine _ Inv) -> True
_ -> False
columns = case action of
ActDb (ActRelationMut _ MutationCreate) -> qsColumns
ActDb (ActRelationMut _ MutationUpdate) -> qsColumns
ActDb (ActRoutine _ Inv) -> qsColumns
_ -> Nothing
isProc = case action of
ActDb (ActRoutine _ _) -> True
_ -> False
params = (T.decodeUtf8 *** T.decodeUtf8) <$> parseSimpleQuery (LBS.toStrict reqBody)
type CsvData = V.Vector (M.Map Text LBS.ByteString)
{-|
Converts CSV like
a,b
1,hi
2,bye
into a JSON array like
[ {"a": "1", "b": "hi"}, {"a": 2, "b": "bye"} ]
The reason for its odd signature is so that it can compose
directly with CSV.decodeByName
-}
csvToJson :: (CSV.Header, CsvData) -> JSON.Value
csvToJson (_, vals) =
JSON.Array $ V.map rowToJsonObj vals
where
rowToJsonObj = JSON.Object . KM.fromMapText .
M.map (\str ->
if str == "NULL"
then JSON.Null
else JSON.String . T.decodeUtf8 $ LBS.toStrict str
)
payloadAttributes :: RequestBody -> JSON.Value -> Maybe Payload
payloadAttributes raw json =
-- Test that Array contains only Objects having the same keys
case json of
JSON.Array arr ->
case arr V.!? 0 of
Just (JSON.Object o) ->
let canonicalKeys = S.fromList $ K.toText <$> KM.keys o
areKeysUniform = all (\case
JSON.Object x -> S.fromList (K.toText <$> KM.keys x) == canonicalKeys
_ -> False) arr in
if areKeysUniform
then Just $ ProcessedJSON raw canonicalKeys
else Nothing
Just _ -> Nothing
Nothing -> Just emptyPJArray
JSON.Object o -> Just $ ProcessedJSON raw (S.fromList $ K.toText <$> KM.keys o)
-- truncate everything else to an empty array.
_ -> Just emptyPJArray
where
emptyPJArray = ProcessedJSON (JSON.encode emptyArray) S.empty
-138
View File
@@ -1,138 +0,0 @@
-- |
-- Module : PostgREST.ApiRequest.Payload
-- Description : Parser for PostgREST Request Body
--
-- This module is in charge of parsing the request body (payload)
--
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE NamedFieldPuns #-}
module PostgREST.ApiRequest.Payload
( getPayload
) where
import qualified Data.Aeson as JSON
import qualified Data.Aeson.Key as K
import qualified Data.Aeson.KeyMap as KM
import qualified Data.ByteString.Char8 as BS
import qualified Data.ByteString.Lazy as LBS
import qualified Data.Csv as CSV
import qualified Data.HashMap.Strict as HM
import qualified Data.Map.Strict as M
import qualified Data.Set as S
import qualified Data.Text.Encoding as T
import qualified Data.Vector as V
import Control.Arrow ((***))
import Data.Aeson.Types (emptyArray, emptyObject)
import Data.Either.Combinators (mapBoth)
import Network.HTTP.Types.URI (parseSimpleQuery)
import PostgREST.ApiRequest.QueryParams (QueryParams (..))
import PostgREST.ApiRequest.Types
import PostgREST.Error (ApiRequestError (..))
import PostgREST.MediaType (MediaType (..))
import PostgREST.SchemaCache.Identifiers (FieldName)
import qualified PostgREST.MediaType as MediaType
import Protolude
getPayload :: RequestBody -> MediaType -> QueryParams -> Action -> Either ApiRequestError (Maybe Payload, S.Set FieldName)
getPayload reqBody contentMediaType QueryParams{qsColumns} action = do
checkedPayload <- if shouldParsePayload then payload else Right Nothing
let cols = case (checkedPayload, columns) of
(Just ProcessedJSON{payKeys}, _) -> payKeys
(Just ProcessedUrlEncoded{payKeys}, _) -> payKeys
(Just RawJSON{}, Just cls) -> cls
_ -> S.empty
return (checkedPayload, cols)
where
payload :: Either ApiRequestError (Maybe Payload)
payload = mapBoth InvalidBody Just $ case (contentMediaType, isProc) of
(MTApplicationJSON, _) ->
if isJust columns
then Right $ RawJSON reqBody
else note "All object keys must match" . payloadAttributes reqBody
=<< if LBS.null reqBody && isProc
then Right emptyObject
else first BS.pack $
-- Drop parsing error message in favor of generic one (https://github.com/PostgREST/postgrest/issues/2344)
maybe (Left "Empty or invalid json") Right $ JSON.decode reqBody
(MTTextCSV, _) -> do
json <- csvToJson <$> first BS.pack (CSV.decodeByName reqBody)
note "All lines must have same number of fields" $ payloadAttributes (JSON.encode json) json
(MTUrlEncoded, True) ->
Right $ ProcessedUrlEncoded params (S.fromList $ fst <$> params)
(MTUrlEncoded, False) ->
let paramsMap = HM.fromList $ (identity *** JSON.String) <$> params in
Right $ ProcessedJSON (JSON.encode paramsMap) $ S.fromList (HM.keys paramsMap)
(MTTextPlain, True) -> Right $ RawPay reqBody
(MTTextXML, True) -> Right $ RawPay reqBody
(MTOctetStream, True) -> Right $ RawPay reqBody
(ct, _) -> Left $ "Content-Type not acceptable: " <> MediaType.toMime ct
shouldParsePayload = case action of
ActDb (ActRelationMut _ MutationDelete) -> False
ActDb (ActRelationMut _ _) -> True
ActDb (ActRoutine _ Inv) -> True
_ -> False
columns = case action of
ActDb (ActRelationMut _ MutationCreate) -> qsColumns
ActDb (ActRelationMut _ MutationUpdate) -> qsColumns
ActDb (ActRoutine _ Inv) -> qsColumns
_ -> Nothing
isProc = case action of
ActDb (ActRoutine _ _) -> True
_ -> False
params = (T.decodeUtf8 *** T.decodeUtf8) <$> parseSimpleQuery (LBS.toStrict reqBody)
type CsvData = V.Vector (M.Map Text LBS.ByteString)
{-|
Converts CSV like
a,b
1,hi
2,bye
into a JSON array like
[ {"a": "1", "b": "hi"}, {"a": 2, "b": "bye"} ]
The reason for its odd signature is so that it can compose
directly with CSV.decodeByName
-}
csvToJson :: (CSV.Header, CsvData) -> JSON.Value
csvToJson (_, vals) =
JSON.Array $ V.map rowToJsonObj vals
where
rowToJsonObj = JSON.Object . KM.fromMapText .
M.map (\str ->
if str == "NULL"
then JSON.Null
else JSON.String . T.decodeUtf8 $ LBS.toStrict str
)
payloadAttributes :: RequestBody -> JSON.Value -> Maybe Payload
payloadAttributes raw json =
-- Test that Array contains only Objects having the same keys
case json of
JSON.Array arr ->
case arr V.!? 0 of
Just (JSON.Object o) ->
let canonicalKeys = S.fromList $ K.toText <$> KM.keys o
areKeysUniform = all (\case
JSON.Object x -> S.fromList (K.toText <$> KM.keys x) == canonicalKeys
_ -> False) arr in
if areKeysUniform
then Just $ ProcessedJSON raw canonicalKeys
else Nothing
Just _ -> Nothing
Nothing -> Just emptyPJArray
JSON.Object o -> Just $ ProcessedJSON raw (S.fromList $ K.toText <$> KM.keys o)
-- truncate everything else to an empty array.
_ -> Just emptyPJArray
where
emptyPJArray = ProcessedJSON (JSON.encode emptyArray) S.empty
+1 -6
View File
@@ -19,7 +19,6 @@ module PostgREST.ApiRequest.Preferences
, PreferMaxAffected(..) , PreferMaxAffected(..)
, fromHeaders , fromHeaders
, shouldCount , shouldCount
, shouldExplainCount
, prefAppliedHeader , prefAppliedHeader
) where ) where
@@ -156,7 +155,7 @@ fromHeaders allowTxDbOverride acceptedTzNames headers =
listStripPrefix prefix prefList = listToMaybe $ mapMaybe (BS.stripPrefix prefix) prefList listStripPrefix prefix prefList = listToMaybe $ mapMaybe (BS.stripPrefix prefix) prefList
timezonePref = listStripPrefix "timezone=" prefs timezonePref = listStripPrefix "timezone=" prefs
isTimezonePrefAccepted = ((S.member . decodeUtf8 <$> timezonePref) <*> pure acceptedTzNames) == Just True isTimezonePrefAccepted = (S.member <$> (decodeUtf8 <$> timezonePref) <*> pure acceptedTzNames) == Just True
maxAffectedPref = listStripPrefix "max-affected=" prefs >>= readMaybe . BS.unpack maxAffectedPref = listStripPrefix "max-affected=" prefs >>= readMaybe . BS.unpack
@@ -239,10 +238,6 @@ shouldCount :: Maybe PreferCount -> Bool
shouldCount prefCount = shouldCount prefCount =
prefCount == Just ExactCount || prefCount == Just EstimatedCount prefCount == Just ExactCount || prefCount == Just EstimatedCount
shouldExplainCount :: Maybe PreferCount -> Bool
shouldExplainCount prefCount =
prefCount == Just PlannedCount || prefCount == Just EstimatedCount
-- | Whether to commit or roll back transactions. -- | Whether to commit or roll back transactions.
data PreferTransaction data PreferTransaction
= Commit -- ^ Commit transaction - the default. = Commit -- ^ Commit transaction - the default.
+1 -58
View File
@@ -30,69 +30,12 @@ module PostgREST.ApiRequest.Types
, QuantOperator(..) , QuantOperator(..)
, FtsOperator(..) , FtsOperator(..)
, SelectItem(..) , SelectItem(..)
, Payload (..)
, InvokeMethod (..)
, Mutation (..)
, Resource (..)
, DbAction (..)
, Action (..)
, RequestBody
) where ) where
import qualified Data.ByteString.Lazy as LBS import PostgREST.SchemaCache.Identifiers (FieldName)
import qualified Data.Set as S
import PostgREST.SchemaCache.Identifiers (FieldName,
QualifiedIdentifier (..),
Schema)
import Protolude import Protolude
data InvokeMethod = Inv | InvRead Bool
deriving Eq
data Mutation
= MutationCreate
| MutationDelete
| MutationSingleUpsert
| MutationUpdate
deriving Eq
data Resource
= ResourceRelation Text
| ResourceRoutine Text
| ResourceSchema
data DbAction
= ActRelationRead {dbActQi :: QualifiedIdentifier, actHeadersOnly :: Bool}
| ActRelationMut {dbActQi :: QualifiedIdentifier, actMutation :: Mutation}
| ActRoutine {dbActQi :: QualifiedIdentifier, actInvMethod :: InvokeMethod}
| ActSchemaRead Schema Bool
data Action
= ActDb DbAction
| ActRelationInfo QualifiedIdentifier
| ActRoutineInfo QualifiedIdentifier InvokeMethod
| ActSchemaInfo
type RequestBody = LBS.ByteString
data Payload
= ProcessedJSON -- ^ Cached attributes of a JSON payload
{ payRaw :: LBS.ByteString
-- ^ This is the raw ByteString that comes from the request body. We
-- cache this instead of an Aeson Value because it was detected that for
-- large payloads the encoding had high memory usage, see
-- https://github.com/PostgREST/postgrest/pull/1005 for more details
, payKeys :: S.Set Text
-- ^ Keys of the object or if it's an array these keys are guaranteed to
-- be the same across all its objects
}
| ProcessedUrlEncoded { payArray :: [(Text, Text)], payKeys :: S.Set Text }
| RawJSON { payRaw :: LBS.ByteString }
| RawPay { payRaw :: LBS.ByteString }
-- | The value in `/tbl?select=alias:field.aggregateFunction()::cast` -- | The value in `/tbl?select=alias:field.aggregateFunction()::cast`
data SelectItem data SelectItem
= SelectField = SelectField
+43 -116
View File
@@ -9,24 +9,18 @@ Some of its functionality includes:
- Producing HTTP Headers according to RFCs. - Producing HTTP Headers according to RFCs.
- Content Negotiation - Content Negotiation
-} -}
{-# LANGUAGE RecordWildCards #-} {-# LANGUAGE RecordWildCards #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE ViewPatterns #-}
module PostgREST.App module PostgREST.App
( postgrest ( postgrest
, run , run
) where ) where
import GHC.IO.Exception (IOErrorType (..))
import System.IO.Error (ioeGetErrorType)
import Control.Monad.Except (liftEither) import Control.Monad.Except (liftEither)
import Data.Either.Combinators (mapLeft, whenLeft) import Data.Either.Combinators (mapLeft, whenLeft)
import Data.Maybe (fromJust) import Data.Maybe (fromJust)
import Data.String (IsString (..)) import Data.String (IsString (..))
import Network.Wai.Handler.Warp (defaultSettings, setHost, import Network.Wai.Handler.Warp (defaultSettings, setHost, setPort,
setOnException, setPort,
setServerName) setServerName)
import qualified Data.Text.Encoding as T import qualified Data.Text.Encoding as T
@@ -41,7 +35,6 @@ import qualified PostgREST.Cors as Cors
import qualified PostgREST.Error as Error import qualified PostgREST.Error as Error
import qualified PostgREST.Listener as Listener import qualified PostgREST.Listener as Listener
import qualified PostgREST.Logger as Logger import qualified PostgREST.Logger as Logger
import qualified PostgREST.MainTx as MainTx
import qualified PostgREST.Plan as Plan import qualified PostgREST.Plan as Plan
import qualified PostgREST.Query as Query import qualified PostgREST.Query as Query
import qualified PostgREST.Response as Response import qualified PostgREST.Response as Response
@@ -50,65 +43,49 @@ import qualified PostgREST.Unix as Unix (installSignalHandlers)
import PostgREST.ApiRequest (ApiRequest (..)) import PostgREST.ApiRequest (ApiRequest (..))
import PostgREST.AppState (AppState) import PostgREST.AppState (AppState)
import PostgREST.Auth.Types (AuthResult (..)) import PostgREST.Auth.Types (AuthResult (..))
import PostgREST.Config (AppConfig (..), LogLevel (..)) import PostgREST.Config (AppConfig (..), LogLevel (..),
LogQuery (..))
import PostgREST.Config.PgVersion (PgVersion (..))
import PostgREST.Error (Error) import PostgREST.Error (Error)
import PostgREST.Network (resolveSocketToAddress) import PostgREST.Network (resolveHost)
import PostgREST.Observation (Observation (..)) import PostgREST.Observation (Observation (..))
import PostgREST.Response.Performance (ServerTiming (..), import PostgREST.Response.Performance (ServerTiming (..),
serverTimingHeader) serverTimingHeader)
import PostgREST.SchemaCache (SchemaCache (..)) import PostgREST.SchemaCache (SchemaCache (..))
import PostgREST.TimeIt (timeItT)
import PostgREST.Version (docsVersion, prettyVersion) import PostgREST.Version (docsVersion, prettyVersion)
import qualified Data.ByteString.Char8 as BS import qualified Data.ByteString.Char8 as BS
import qualified Data.List as L import qualified Data.List as L
import Data.Streaming.Network (bindPortTCP, import qualified Network.HTTP.Types as HTTP
bindRandomPortTCP) import qualified Network.Socket as NS
import qualified Data.Text as T import Protolude hiding (Handler)
import qualified Network.HTTP.Types as HTTP import System.TimeIt (timeItT)
import qualified Network.Socket as NS
import PostgREST.Unix (createAndBindDomainSocket)
import Protolude hiding (Handler)
type Handler = ExceptT Error type Handler = ExceptT Error
run :: AppState -> IO () run :: AppState -> IO ()
run appState = do run appState = do
let observer = AppState.getObserver appState
conf@AppConfig{..} <- AppState.getConfig appState conf@AppConfig{..} <- AppState.getConfig appState
AppState.schemaCacheLoader appState -- Loads the initial SchemaCache AppState.schemaCacheLoader appState -- Loads the initial SchemaCache
(mainSocket, adminSocket) <- initSockets conf Unix.installSignalHandlers (AppState.getMainThreadId appState) (AppState.schemaCacheLoader appState) (AppState.readInDbConfig False appState)
Unix.installSignalHandlers observer (AppState.getMainThreadId appState) (AppState.schemaCacheLoader appState) (AppState.readInDbConfig False appState)
Listener.runListener appState Listener.runListener appState
Admin.runAdmin appState adminSocket mainSocket (serverSettings conf) Admin.runAdmin appState (serverSettings conf)
let app = postgrest configLogLevel appState (AppState.schemaCacheLoader appState) let app = postgrest configLogLevel appState (AppState.schemaCacheLoader appState)
do case configServerUnixSocket of
address <- resolveSocketToAddress mainSocket Just path -> do
observer $ AppServerAddressObs address observer $ AppServerUnixObs path
Nothing -> do
port <- NS.socketPort $ AppState.getSocketREST appState
host <- resolveHost $ AppState.getSocketREST appState
observer $ AppServerPortObs (fromJust host) port
Warp.runSettingsSocket (serverSettings conf & setOnException onWarpException) mainSocket app Warp.runSettingsSocket (serverSettings conf) (AppState.getSocketREST appState) app
where
observer = AppState.getObserver appState
onWarpException :: Maybe Wai.Request -> SomeException -> IO ()
onWarpException _ ex =
when (shouldDisplayException ex) $
observer $ WarpServerObs $ show ex
-- Similar to wai defaultShouldDisplayException in
-- https://github.com/yesodweb/wai//blob/8c3882c60f6abe043889fc20c7efd3fa9747fa4a/warp/Network/Wai/Handler/Warp/Settings.hs#L251-L258
-- but without omitting AsyncException since it's important to log for ThreadKilled, StackOverflow and other cases.
-- We want to reuse this to avoid flooding the logs for some transient failure cases.
shouldDisplayException :: SomeException -> Bool
shouldDisplayException se
| Just (_ :: Warp.InvalidRequest) <- fromException se = False
| Just (ioeGetErrorType -> et) <- fromException se, et == ResourceVanished || et == InvalidArgument = False
| otherwise = True
serverSettings :: AppConfig -> Warp.Settings serverSettings :: AppConfig -> Warp.Settings
serverSettings AppConfig{..} = serverSettings AppConfig{..} =
@@ -131,21 +108,18 @@ postgrest logLevel appState connWorker =
Right authResult -> do Right authResult -> do
appConf <- AppState.getConfig appState -- the config must be read again because it can reload appConf <- AppState.getConfig appState -- the config must be read again because it can reload
maybeSchemaCache <- AppState.getSchemaCache appState maybeSchemaCache <- AppState.getSchemaCache appState
pgVer <- AppState.getPgVersion appState
let let
eitherResponse :: IO (Either Error Wai.Response) eitherResponse :: IO (Either Error Wai.Response)
eitherResponse = eitherResponse =
runExceptT $ postgrestResponse appState appConf maybeSchemaCache authResult req runExceptT $ postgrestResponse appState appConf maybeSchemaCache pgVer authResult req
response <- either Error.errorResponseFor identity <$> eitherResponse response <- either Error.errorResponseFor identity <$> eitherResponse
-- Launch the connWorker when the connection is down. The postgrest -- Launch the connWorker when the connection is down. The postgrest
-- function can respond successfully (with a stale schema cache) before -- function can respond successfully (with a stale schema cache) before
-- the connWorker is done. However, when there's an empty schema cache -- the connWorker is done.
-- postgrest responds with the error `PGRST002`; this means that the schema when (isServiceUnavailable response) connWorker
-- cache is still loading, so we don't launch the connWorker here because
-- it would duplicate the loading process, e.g. https://github.com/PostgREST/postgrest/issues/3704
-- TODO: this process may be unnecessary when the Listener is enabled. Revisit once https://github.com/PostgREST/postgrest/issues/1766 is done
when (isServiceUnavailable response && isJust maybeSchemaCache) connWorker
resp <- do resp <- do
delay <- AppState.getNextDelay appState delay <- AppState.getNextDelay appState
return $ addRetryHint delay response return $ addRetryHint delay response
@@ -155,18 +129,16 @@ postgrestResponse
:: AppState.AppState :: AppState.AppState
-> AppConfig -> AppConfig
-> Maybe SchemaCache -> Maybe SchemaCache
-> PgVersion
-> AuthResult -> AuthResult
-> Wai.Request -> Wai.Request
-> Handler IO Wai.Response -> Handler IO Wai.Response
postgrestResponse appState conf@AppConfig{..} maybeSchemaCache authResult@AuthResult{..} req = do postgrestResponse appState conf@AppConfig{..} maybeSchemaCache pgVer authResult@AuthResult{..} req = do
let observer = AppState.getObserver appState
sCache <- sCache <-
case maybeSchemaCache of case maybeSchemaCache of
Just sCache -> Just sCache ->
return sCache return sCache
Nothing -> do Nothing ->
lift $ observer SchemaCacheEmptyObs
throwError Error.NoSchemaCacheError throwError Error.NoSchemaCacheError
body <- lift $ Wai.strictRequestBody req body <- lift $ Wai.strictRequestBody req
@@ -178,32 +150,24 @@ postgrestResponse appState conf@AppConfig{..} maybeSchemaCache authResult@AuthRe
(parseTime, apiReq@ApiRequest{..}) <- withTiming $ liftEither . mapLeft Error.ApiRequestError $ ApiRequest.userApiRequest conf prefs req body (parseTime, apiReq@ApiRequest{..}) <- withTiming $ liftEither . mapLeft Error.ApiRequestError $ ApiRequest.userApiRequest conf prefs req body
(planTime, plan) <- withTiming $ liftEither $ Plan.actionPlan iAction conf apiReq sCache (planTime, plan) <- withTiming $ liftEither $ Plan.actionPlan iAction conf apiReq sCache
let mainQ = Query.mainQuery plan conf apiReq authResult configDbPreRequest let query = Query.query conf authResult apiReq plan sCache pgVer
tx = MainTx.mainTx mainQ conf authResult apiReq plan sCache logSQL = lift . AppState.getObserver appState . DBQuery (Query.getSQLQuery query)
obsQuery s = when configLogQuery $ observer $ QueryObs mainQ s
(txTime, txResult) <- withTiming $ do (queryTime, queryResult) <- withTiming $ do
case tx of case query of
MainTx.NoDbTx r -> pure r Query.NoDbQuery r -> pure r
MainTx.DbTx{..} -> do Query.DbQuery{..} -> do
dbRes <- lift $ AppState.usePool appState (dqTransaction dqIsoLevel dqTxMode $ runExceptT dqDbHandler) dbRes <- lift $ AppState.usePool appState (dqTransaction dqIsoLevel dqTxMode $ runExceptT dqDbHandler)
let eitherResp = join $ mapLeft (Error.PgErr . Error.PgError (Just authRole /= configDbAnonRole)) dbRes let eitherResp = mapLeft Error.PgErr . mapLeft (Error.PgError (Just authRole /= configDbAnonRole)) $ dbRes
when (configLogQuery /= LogQueryDisabled) $ whenLeft eitherResp $ logSQL . Error.status
-- TODO: we use obsQuery twice, one here and one below because in case of an error with the usePool above, the request will finish here and return an error message. liftEither eitherResp >>= liftEither
-- This is because of a combination of ExceptT + our Error module which has Wai.responseLBS.
-- This needs refactoring so only the below obsQuery is used.
lift $ whenLeft eitherResp $ obsQuery . Error.status
liftEither eitherResp
(respTime, resp) <- withTiming $ do (respTime, resp) <- withTiming $ do
let response = Response.actionResponse txResult apiReq (T.decodeUtf8 prettyVersion, docsVersion) conf sCache iSchema iNegotiatedByProfile let response = Response.actionResponse queryResult apiReq (T.decodeUtf8 prettyVersion, docsVersion) conf sCache iSchema iNegotiatedByProfile
status' = either Error.status Response.pgrstStatus response when (configLogQuery /= LogQueryDisabled) $ logSQL $ either Error.status Response.pgrstStatus response
-- TODO: see above obsQuery, only this obsQuery should remain after refactoring (because the QueryObs depends on the status)
lift $ obsQuery status'
liftEither response liftEither response
return $ toWaiResponse (ServerTiming jwtTime parseTime planTime txTime respTime) resp return $ toWaiResponse (ServerTiming jwtTime parseTime planTime queryTime respTime) resp
where where
toWaiResponse :: ServerTiming -> Response.PgrstResponse -> Wai.Response toWaiResponse :: ServerTiming -> Response.PgrstResponse -> Wai.Response
@@ -235,40 +199,3 @@ addRetryHint delay response = do
isServiceUnavailable :: Wai.Response -> Bool isServiceUnavailable :: Wai.Response -> Bool
isServiceUnavailable response = Wai.responseStatus response == HTTP.status503 isServiceUnavailable response = Wai.responseStatus response == HTTP.status503
type AppSockets = (NS.Socket, Maybe NS.Socket)
initSockets :: AppConfig -> IO AppSockets
initSockets AppConfig{..} = do
let
cfg'usp = configServerUnixSocket
cfg'uspm = configServerUnixSocketMode
cfg'host = configServerHost
cfg'port = configServerPort
cfg'adminHost = configAdminServerHost
cfg'adminPort = configAdminServerPort
sock <- case cfg'usp of
-- I'm not using `streaming-commons`' bindPath function here because it's not defined for Windows,
-- but we need to have runtime error if we try to use it in Windows, not compile time error
Just path -> createAndBindDomainSocket path cfg'uspm
Nothing -> do
(_, sock) <-
if cfg'port /= 0
then do
sock <- bindPortTCP cfg'port (fromString $ T.unpack cfg'host)
pure (cfg'port, sock)
else do
-- explicitly bind to a random port, returning bound port number
(num, sock) <- bindRandomPortTCP (fromString $ T.unpack cfg'host)
pure (num, sock)
pure sock
adminSock <- case cfg'adminPort of
Just adminPort -> do
adminSock <- bindPortTCP adminPort (fromString $ T.unpack cfg'adminHost)
pure $ Just adminSock
Nothing -> pure Nothing
pure (sock, adminSock)
+80 -27
View File
@@ -13,9 +13,11 @@ module PostgREST.AppState
, getNextListenerDelay , getNextListenerDelay
, getTime , getTime
, getJwtCacheState , getJwtCacheState
, getSocketREST
, getSocketAdmin
, init , init
, initSockets
, initWithPool , initWithPool
, putConfig -- For tests TODO refactoring
, putNextListenerDelay , putNextListenerDelay
, putSchemaCache , putSchemaCache
, putPgVersion , putPgVersion
@@ -30,18 +32,20 @@ module PostgREST.AppState
import qualified Data.ByteString.Char8 as BS import qualified Data.ByteString.Char8 as BS
import Data.Either.Combinators (whenLeft) import Data.Either.Combinators (whenLeft)
import qualified Data.Text as T (unpack)
import qualified Hasql.Pool as SQL import qualified Hasql.Pool as SQL
import qualified Hasql.Pool.Config as SQL import qualified Hasql.Pool.Config as SQL
import qualified Hasql.Session as SQL import qualified Hasql.Session as SQL
import qualified Hasql.Transaction.Sessions as SQL import qualified Hasql.Transaction.Sessions as SQL
import qualified Network.HTTP.Types.Status as HTTP import qualified Network.HTTP.Types.Status as HTTP
import qualified Network.Socket as NS
import qualified PostgREST.Auth.JwtCache as JwtCache import qualified PostgREST.Auth.JwtCache as JwtCache
import qualified PostgREST.Error as Error import qualified PostgREST.Error as Error
import qualified PostgREST.Logger as Logger import qualified PostgREST.Logger as Logger
import qualified PostgREST.Metrics as Metrics import qualified PostgREST.Metrics as Metrics
import PostgREST.Observation import PostgREST.Observation
import PostgREST.TimeIt (timeItT)
import PostgREST.Version (prettyVersion) import PostgREST.Version (prettyVersion)
import System.TimeIt (timeItT)
import Control.AutoUpdate (defaultUpdateSettings, mkAutoUpdate, import Control.AutoUpdate (defaultUpdateSettings, mkAutoUpdate,
updateAction) updateAction)
@@ -53,7 +57,7 @@ import Data.IORef (IORef, atomicWriteIORef, newIORef,
readIORef) readIORef)
import Data.Time.Clock (UTCTime, getCurrentTime) import Data.Time.Clock (UTCTime, getCurrentTime)
import PostgREST.Auth.JwtCache (JwtCacheState, update) import PostgREST.Auth.JwtCache (JwtCacheState)
import PostgREST.Config (AppConfig (..), import PostgREST.Config (AppConfig (..),
addFallbackAppName, addFallbackAppName,
readAppConfig) readAppConfig)
@@ -65,8 +69,11 @@ import PostgREST.Config.PgVersion (PgVersion (..),
import PostgREST.SchemaCache (SchemaCache (..), import PostgREST.SchemaCache (SchemaCache (..),
querySchemaCache, querySchemaCache,
showSummary) showSummary)
import PostgREST.SchemaCache.Identifiers (quoteQi) import PostgREST.SchemaCache.Identifiers (dumpQi)
import PostgREST.Unix (createAndBindDomainSocket)
import Data.Streaming.Network (bindPortTCP, bindRandomPortTCP)
import Data.String (IsString (..))
import Protolude import Protolude
data AppState = AppState data AppState = AppState
@@ -92,6 +99,10 @@ data AppState = AppState
, stateNextDelay :: IORef Int , stateNextDelay :: IORef Int
-- | Keeps track of the next delay for the listener -- | Keeps track of the next delay for the listener
, stateNextListenerDelay :: IORef Int , stateNextListenerDelay :: IORef Int
-- | Network socket for REST API
, stateSocketREST :: NS.Socket
-- | Network socket for the admin UI
, stateSocketAdmin :: Maybe NS.Socket
-- | Observation handler -- | Observation handler
, stateObserver :: ObservationHandler , stateObserver :: ObservationHandler
-- | JWT Cache -- | JWT Cache
@@ -106,6 +117,8 @@ data SchemaCacheStatus
| SCPending | SCPending
deriving Eq deriving Eq
type AppSockets = (NS.Socket, Maybe NS.Socket)
init :: AppConfig -> IO AppState init :: AppConfig -> IO AppState
init conf@AppConfig{configLogLevel, configDbPoolSize} = do init conf@AppConfig{configLogLevel, configDbPoolSize} = do
loggerState <- Logger.init loggerState <- Logger.init
@@ -114,11 +127,14 @@ init conf@AppConfig{configLogLevel, configDbPoolSize} = do
observer $ AppStartObs prettyVersion observer $ AppStartObs prettyVersion
jwtCacheState <- JwtCache.init
pool <- initPool conf observer pool <- initPool conf observer
initWithPool pool conf loggerState metricsState observer --{ stateSocketREST = sock, stateSocketAdmin = adminSock} (sock, adminSock) <- initSockets conf
state' <- initWithPool (sock, adminSock) pool conf jwtCacheState loggerState metricsState observer
pure state' { stateSocketREST = sock, stateSocketAdmin = adminSock}
initWithPool :: SQL.Pool -> AppConfig -> Logger.LoggerState -> Metrics.MetricsState -> ObservationHandler -> IO AppState initWithPool :: AppSockets -> SQL.Pool -> AppConfig -> JwtCache.JwtCacheState -> Logger.LoggerState -> Metrics.MetricsState -> ObservationHandler -> IO AppState
initWithPool pool conf loggerState metricsState observer = do initWithPool (sock, adminSock) pool conf jwtCacheState loggerState metricsState observer = do
appState <- AppState pool appState <- AppState pool
<$> newIORef minimumPgVersion -- assume we're in a supported version when starting, this will be corrected on a later step <$> newIORef minimumPgVersion -- assume we're in a supported version when starting, this will be corrected on a later step
@@ -131,8 +147,10 @@ initWithPool pool conf loggerState metricsState observer = do
<*> myThreadId <*> myThreadId
<*> newIORef 0 <*> newIORef 0
<*> newIORef 1 <*> newIORef 1
<*> pure sock
<*> pure adminSock
<*> pure observer <*> pure observer
<*> JwtCache.init conf observer <*> pure jwtCacheState
<*> pure loggerState <*> pure loggerState
<*> pure metricsState <*> pure metricsState
@@ -149,6 +167,40 @@ initWithPool pool conf loggerState metricsState observer = do
destroy :: AppState -> IO () destroy :: AppState -> IO ()
destroy = destroyPool destroy = destroyPool
initSockets :: AppConfig -> IO AppSockets
initSockets AppConfig{..} = do
let
cfg'usp = configServerUnixSocket
cfg'uspm = configServerUnixSocketMode
cfg'host = configServerHost
cfg'port = configServerPort
cfg'adminHost = configAdminServerHost
cfg'adminPort = configAdminServerPort
sock <- case cfg'usp of
-- I'm not using `streaming-commons`' bindPath function here because it's not defined for Windows,
-- but we need to have runtime error if we try to use it in Windows, not compile time error
Just path -> createAndBindDomainSocket path cfg'uspm
Nothing -> do
(_, sock) <-
if cfg'port /= 0
then do
sock <- bindPortTCP cfg'port (fromString $ T.unpack cfg'host)
pure (cfg'port, sock)
else do
-- explicitly bind to a random port, returning bound port number
(num, sock) <- bindRandomPortTCP (fromString $ T.unpack cfg'host)
pure (num, sock)
pure sock
adminSock <- case cfg'adminPort of
Just adminPort -> do
adminSock <- bindPortTCP adminPort (fromString $ T.unpack cfg'adminHost)
pure $ Just adminSock
Nothing -> pure Nothing
pure (sock, adminSock)
initPool :: AppConfig -> ObservationHandler -> IO SQL.Pool initPool :: AppConfig -> ObservationHandler -> IO SQL.Pool
initPool AppConfig{..} observer = do initPool AppConfig{..} observer = do
SQL.acquire $ SQL.settings SQL.acquire $ SQL.settings
@@ -219,14 +271,10 @@ usePool AppState{stateObserver=observer, stateMainThreadId=mainThreadId, ..} ses
-- | Flush the connection pool so that any future use of the pool will -- | Flush the connection pool so that any future use of the pool will
-- use connections freshly established after this call. -- use connections freshly established after this call.
-- | Emits PoolFlushed observation
flushPool :: AppState -> IO () flushPool :: AppState -> IO ()
flushPool AppState{..} = do flushPool AppState{..} = SQL.release statePool
SQL.release statePool
stateObserver PoolFlushed
-- | Destroy the pool on shutdown. -- | Destroy the pool on shutdown.
-- | Differs from flushPool in not emiting PoolFlushed observation.
destroyPool :: AppState -> IO () destroyPool :: AppState -> IO ()
destroyPool AppState{..} = SQL.release statePool destroyPool AppState{..} = SQL.release statePool
@@ -266,6 +314,12 @@ getTime = stateGetTime
getJwtCacheState :: AppState -> JwtCacheState getJwtCacheState :: AppState -> JwtCacheState
getJwtCacheState = stateJwtCache getJwtCacheState = stateJwtCache
getSocketREST :: AppState -> NS.Socket
getSocketREST = stateSocketREST
getSocketAdmin :: AppState -> Maybe NS.Socket
getSocketAdmin = stateSocketAdmin
getMainThreadId :: AppState -> ThreadId getMainThreadId :: AppState -> ThreadId
getMainThreadId = stateMainThreadId getMainThreadId = stateMainThreadId
@@ -313,6 +367,8 @@ retryingSchemaCacheLoad appState@AppState{stateObserver=observer, stateMainThrea
observer $ ConnectionRetryObs delay observer $ ConnectionRetryObs delay
putNextListenerDelay appState delay putNextListenerDelay appState delay
flushPool appState
(,) <$> qPgVersion <*> (qInDbConfig *> qSchemaCache) (,) <$> qPgVersion <*> (qInDbConfig *> qSchemaCache)
) )
where where
@@ -327,16 +383,14 @@ retryingSchemaCacheLoad appState@AppState{stateObserver=observer, stateMainThrea
observer ExitDBNoRecoveryObs observer ExitDBNoRecoveryObs
killThread mainThreadId killThread mainThreadId
return Nothing return Nothing
Right actualPgVersion -> Right actualPgVersion -> do
if actualPgVersion < minimumPgVersion then do when (actualPgVersion < minimumPgVersion) $ do
observer $ ExitUnsupportedPgVersion actualPgVersion minimumPgVersion observer $ ExitUnsupportedPgVersion actualPgVersion minimumPgVersion
killThread mainThreadId killThread mainThreadId
return Nothing observer $ DBConnectedObs $ pgvFullName actualPgVersion
else do observer $ PoolInit configDbPoolSize
observer $ DBConnectedObs $ pgvFullName actualPgVersion putPgVersion appState actualPgVersion
observer $ PoolInit configDbPoolSize return $ Just actualPgVersion
putPgVersion appState actualPgVersion
return $ Just actualPgVersion
qInDbConfig :: IO () qInDbConfig :: IO ()
qInDbConfig = do qInDbConfig = do
@@ -361,10 +415,6 @@ retryingSchemaCacheLoad appState@AppState{stateObserver=observer, stateMainThrea
-- IORef on putSchemaCache. This is why SCacheStatus is put at SCPending here to signal the Admin server (using isPending) that we're on a recovery state. -- IORef on putSchemaCache. This is why SCacheStatus is put at SCPending here to signal the Admin server (using isPending) that we're on a recovery state.
putSCacheStatus appState SCPending putSCacheStatus appState SCPending
putSchemaCache appState $ Just sCache putSchemaCache appState $ Just sCache
-- Flush the pool after loading the schema cache to reset any stale session cache entries
-- We do it after successfully querying the schema cache (because this can fail and during retries we would flush the pool repeatedly unnecessarily)
-- and after marking sCacheStatus as pending,
flushPool appState
observer $ SchemaCacheQueriedObs resultTime observer $ SchemaCacheQueriedObs resultTime
(t, _) <- timeItT $ observer $ SchemaCacheSummaryObs $ showSummary sCache (t, _) <- timeItT $ observer $ SchemaCacheSummaryObs $ showSummary sCache
observer $ SchemaCacheLoadedObs t observer $ SchemaCacheLoadedObs t
@@ -392,7 +442,7 @@ readInDbConfig startingUp appState@AppState{stateObserver=observer} = do
pgVer <- getPgVersion appState pgVer <- getPgVersion appState
dbSettings <- dbSettings <-
if configDbConfig conf then do if configDbConfig conf then do
qDbSettings <- usePool appState (queryDbSettings (quoteQi <$> configDbPreConfig conf) (configDbPreparedStatements conf)) qDbSettings <- usePool appState (queryDbSettings (dumpQi <$> configDbPreConfig conf) (configDbPreparedStatements conf))
case qDbSettings of case qDbSettings of
Left e -> do Left e -> do
observer $ ConfigReadErrorObs e observer $ ConfigReadErrorObs e
@@ -421,7 +471,10 @@ readInDbConfig startingUp appState@AppState{stateObserver=observer} = do
-- After the config has reloaded, jwt-secret might have changed, so -- After the config has reloaded, jwt-secret might have changed, so
-- if it has changed, it is important to invalidate the jwt cache -- if it has changed, it is important to invalidate the jwt cache
-- entries, because they were cached using the old secret -- entries, because they were cached using the old secret
update (getJwtCacheState appState) newConf if configJwtSecret conf == configJwtSecret newConf then
pass
else
JwtCache.emptyCache (getJwtCacheState appState) -- atomic O(1) operation
if startingUp then if startingUp then
pass pass
+152 -17
View File
@@ -1,4 +1,3 @@
{-# LANGUAGE RecordWildCards #-}
{-| {-|
Module : PostgREST.Auth Module : PostgREST.Auth
Description : PostgREST authentication functions. Description : PostgREST authentication functions.
@@ -11,6 +10,8 @@ Authentication should always be implemented in an external service.
In the test suite there is an example of simple login function that can be used for a In the test suite there is an example of simple login function that can be used for a
very simple authentication system inside the PostgreSQL database. very simple authentication system inside the PostgreSQL database.
-} -}
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE RecordWildCards #-}
module PostgREST.Auth module PostgREST.Auth
( getResult ( getResult
, getJwtDur , getJwtDur
@@ -18,44 +19,178 @@ module PostgREST.Auth
, middleware , middleware
) where ) where
import qualified Data.Aeson as JSON
import qualified Data.Aeson.Key as K
import qualified Data.Aeson.KeyMap as KM
import qualified Data.Aeson.Types as JSON
import qualified Data.ByteString as BS import qualified Data.ByteString as BS
import qualified Data.ByteString.Internal as BS
import qualified Data.ByteString.Lazy.Char8 as LBS
import qualified Data.Scientific as Sci
import qualified Data.Text as T
import qualified Data.Vault.Lazy as Vault import qualified Data.Vault.Lazy as Vault
import qualified Data.Vector as V
import qualified Jose.Jwk as JWT
import qualified Jose.Jwt as JWT
import qualified Network.HTTP.Types.Header as HTTP import qualified Network.HTTP.Types.Header as HTTP
import qualified Network.Wai as Wai import qualified Network.Wai as Wai
import qualified Network.Wai.Middleware.HttpAuth as Wai import qualified Network.Wai.Middleware.HttpAuth as Wai
import Data.List (lookup) import Control.Monad.Except (liftEither)
import PostgREST.TimeIt (timeItT) import Data.Either.Combinators (mapLeft)
import System.IO.Unsafe (unsafePerformIO) import Data.List (lookup)
import Data.Time.Clock (UTCTime, nominalDiffTimeToSeconds)
import Data.Time.Clock.POSIX (utcTimeToPOSIXSeconds)
import System.IO.Unsafe (unsafePerformIO)
import System.TimeIt (timeItT)
import PostgREST.AppState (AppState, getConfig, getJwtCacheState, import PostgREST.AppState (AppState, getConfig, getJwtCacheState,
getTime) getTime)
import PostgREST.Auth.Jwt (parseClaims)
import PostgREST.Auth.JwtCache (lookupJwtCache) import PostgREST.Auth.JwtCache (lookupJwtCache)
import PostgREST.Auth.Types (AuthResult (..)) import PostgREST.Auth.Types (AuthResult (..))
import PostgREST.Config (AppConfig (..)) import PostgREST.Config (AppConfig (..), FilterExp (..),
import PostgREST.Error (Error (..)) JSPath, JSPathExp (..))
import PostgREST.Error (Error (..), JwtClaimsError (..),
JwtDecodeError (..), JwtError (..))
import Protolude import Protolude
-- | Validate authorization header -- | Receives the JWT secret and audience (from config) and a JWT and returns a
-- JSON object of JWT claims.
parseToken :: AppConfig -> Maybe ByteString -> UTCTime -> ExceptT Error IO JSON.Value
parseToken _ Nothing _ = return JSON.emptyObject
parseToken _ (Just "") _ = throwE . JwtErr $ JwtDecodeErr EmptyAuthHeader
parseToken AppConfig{..} (Just tkn) time = do
secret <- liftEither . maybeToRight (JwtErr JwtSecretMissing) $ configJWKS
tknWith3Parts <- liftEither $ hasThreeParts tkn
eitherContent <- liftIO $ JWT.decode (JWT.keys secret) Nothing tknWith3Parts
content <- liftEither . mapLeft (JwtErr . jwtDecodeError) $ eitherContent
liftEither $ mapLeft JwtErr $ verifyClaims content
where
hasThreeParts :: ByteString -> Either Error ByteString
hasThreeParts token = case length $ BS.split (BS.c2w '.') token of
3 -> Right token
n -> Left $ JwtErr $ JwtDecodeErr $ UnexpectedParts n
jwtDecodeError :: JWT.JwtError -> JwtError
-- The only errors we can get from JWT.decode function are:
-- BadAlgorithm
-- KeyError
-- BadCrypto
jwtDecodeError (JWT.KeyError m) = JwtDecodeErr $ KeyError m
jwtDecodeError (JWT.BadAlgorithm m) = JwtDecodeErr $ BadAlgorithm m
jwtDecodeError JWT.BadCrypto = JwtDecodeErr BadCrypto
-- Control never reaches here, the decode function only returns the above three
jwtDecodeError _ = JwtDecodeErr UnreachableDecodeError
verifyClaims :: JWT.JwtContent -> Either JwtError JSON.Value
verifyClaims (JWT.Jws (_, claims)) = case JSON.decodeStrict claims of
Just jclaims@(JSON.Object mclaims) ->
verifyClaim mclaims "exp" isValidExpClaim JWTExpired >>
verifyClaim mclaims "nbf" isValidNbfClaim JWTNotYetValid >>
verifyClaim mclaims "iat" isValidIatClaim JWTIssuedAtFuture >>
verifyClaim mclaims "aud" isValidAudClaim JWTNotInAudience >>
return jclaims
_ -> Left $ JwtClaimsErr ParsingClaimsFailed
-- TODO: We could enable JWE support here (encrypted tokens)
verifyClaims _ = Left $ JwtDecodeErr UnsupportedTokenType
verifyClaim mclaims claim func err = do
isValid <- maybe (Right True) func (KM.lookup claim mclaims)
unless isValid $ Left $ JwtClaimsErr err
allowedSkewSeconds = 30 :: Int64
now = floor . nominalDiffTimeToSeconds $ utcTimeToPOSIXSeconds time
sciToInt = fromMaybe 0 . Sci.toBoundedInteger
allStrings = all (\case (JSON.String _) -> True; _ -> False)
isValidExpClaim :: JSON.Value -> Either JwtError Bool
isValidExpClaim (JSON.Number secs) = Right $ now <= (sciToInt secs + allowedSkewSeconds)
isValidExpClaim _ = Left $ JwtClaimsErr ExpClaimNotNumber
isValidNbfClaim :: JSON.Value -> Either JwtError Bool
isValidNbfClaim (JSON.Number secs) = Right $ now >= (sciToInt secs - allowedSkewSeconds)
isValidNbfClaim _ = Left $ JwtClaimsErr NbfClaimNotNumber
isValidIatClaim :: JSON.Value -> Either JwtError Bool
isValidIatClaim (JSON.Number secs) = Right $ now >= (sciToInt secs - allowedSkewSeconds)
isValidIatClaim _ = Left $ JwtClaimsErr IatClaimNotNumber
isValidAudClaim :: JSON.Value -> Either JwtError Bool
isValidAudClaim JSON.Null = Right True -- {"aud": null} is valid for all audiences
isValidAudClaim (JSON.String str) = Right $ maybe (const True) (==) configJwtAudience str
isValidAudClaim (JSON.Array arr)
| null arr = Right True -- {"aud": []} is valid for all audiences
| allStrings arr = Right $ maybe True (\a -> JSON.String a `elem` arr) configJwtAudience
isValidAudClaim _ = Left $ JwtClaimsErr AudClaimNotStringOrArray
parseClaims :: Monad m =>
AppConfig -> JSON.Value -> ExceptT Error m AuthResult
parseClaims AppConfig{..} jclaims@(JSON.Object mclaims) = do
-- role defaults to anon if not specified in jwt
role <- liftEither . maybeToRight (JwtErr JwtTokenRequired) $
unquoted <$> walkJSPath (Just jclaims) configJwtRoleClaimKey <|> configDbAnonRole
return AuthResult
{ authClaims = mclaims & KM.insert "role" (JSON.toJSON $ decodeUtf8 role)
, authRole = role
}
where
walkJSPath :: Maybe JSON.Value -> JSPath -> Maybe JSON.Value
walkJSPath x [] = x
walkJSPath (Just (JSON.Object o)) (JSPKey key:rest) = walkJSPath (KM.lookup (K.fromText key) o) rest
walkJSPath (Just (JSON.Array ar)) (JSPIdx idx:rest) = walkJSPath (ar V.!? idx) rest
walkJSPath (Just (JSON.Array ar)) [JSPFilter (EqualsCond txt)] = findFirstMatch (==) txt ar
walkJSPath (Just (JSON.Array ar)) [JSPFilter (NotEqualsCond txt)] = findFirstMatch (/=) txt ar
walkJSPath (Just (JSON.Array ar)) [JSPFilter (StartsWithCond txt)] = findFirstMatch T.isPrefixOf txt ar
walkJSPath (Just (JSON.Array ar)) [JSPFilter (EndsWithCond txt)] = findFirstMatch T.isSuffixOf txt ar
walkJSPath (Just (JSON.Array ar)) [JSPFilter (ContainsCond txt)] = findFirstMatch T.isInfixOf txt ar
walkJSPath _ _ = Nothing
findFirstMatch matchWith pattern = foldr checkMatch Nothing
where
checkMatch (JSON.String txt) acc
| pattern `matchWith` txt = Just $ JSON.String txt
| otherwise = acc
checkMatch _ acc = acc
unquoted :: JSON.Value -> BS.ByteString
unquoted (JSON.String t) = encodeUtf8 t
unquoted v = LBS.toStrict $ JSON.encode v
-- impossible case - just added to please -Wincomplete-patterns
parseClaims _ _ = return AuthResult { authClaims = KM.empty, authRole = mempty }
-- | Validate authorization header.
-- Parse and store JWT claims for future use in the request. -- Parse and store JWT claims for future use in the request.
middleware :: AppState -> Wai.Middleware middleware :: AppState -> Wai.Middleware
middleware appState app req respond = do middleware appState app req respond = do
conf@AppConfig{..} <- getConfig appState conf <- getConfig appState
time <- getTime appState time <- getTime appState
let token = Wai.extractBearerAuth =<< lookup HTTP.hAuthorization (Wai.requestHeaders req) let token = Wai.extractBearerAuth =<< lookup HTTP.hAuthorization (Wai.requestHeaders req)
parseJwt = runExceptT $ lookupJwtCache jwtCacheState token >>= parseClaims conf time parseJwt = runExceptT $ parseToken conf token time >>= parseClaims conf
jwtCacheState = getJwtCacheState appState jwtCacheState = getJwtCacheState appState
-- If ServerTimingEnabled -> calculate JWT validation time -- If ServerTimingEnabled -> calculate JWT validation time
req' <- if configServerTimingEnabled then do -- If JwtCacheMaxLifetime -> cache JWT validation result
(dur, authResult) <- timeItT parseJwt req' <- case (configServerTimingEnabled conf, configJwtCacheMaxLifetime conf) of
pure $ req { Wai.vault = Wai.vault req & Vault.insert authResultKey authResult & Vault.insert jwtDurKey dur } (True, 0) -> do
else do (dur, authResult) <- timeItT parseJwt
authResult <- parseJwt return $ req { Wai.vault = Wai.vault req & Vault.insert authResultKey authResult & Vault.insert jwtDurKey dur }
pure $ req { Wai.vault = Wai.vault req & Vault.insert authResultKey authResult }
(True, maxLifetime) -> do
(dur, authResult) <- timeItT $ case token of
Just tkn -> lookupJwtCache jwtCacheState tkn maxLifetime parseJwt time
Nothing -> parseJwt
return $ req { Wai.vault = Wai.vault req & Vault.insert authResultKey authResult & Vault.insert jwtDurKey dur }
(False, 0) -> do
authResult <- parseJwt
return $ req { Wai.vault = Wai.vault req & Vault.insert authResultKey authResult }
(False, maxLifetime) -> do
authResult <- case token of
Just tkn -> lookupJwtCache jwtCacheState tkn maxLifetime parseJwt time
Nothing -> parseJwt
return $ req { Wai.vault = Wai.vault req & Vault.insert authResultKey authResult }
app req' respond app req' respond
-151
View File
@@ -1,151 +0,0 @@
{-|
Module : PostgREST.Auth.Jwt
Description : PostgREST JWT support functions.
This module provides functions to deal with JWT parsing and validation (http://jwt.io).
-}
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE ImpredicativeTypes #-}
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE QuantifiedConstraints #-}
module PostgREST.Auth.Jwt
( parseAndDecodeClaims
, parseClaims) where
import qualified Data.Aeson as JSON
import qualified Data.Aeson.Key as K
import qualified Data.Aeson.KeyMap as KM
import qualified Data.ByteString as BS
import qualified Data.ByteString.Internal as BS
import qualified Data.ByteString.Lazy.Char8 as LBS
import qualified Data.Scientific as Sci
import qualified Data.Text as T
import qualified Data.Vector as V
import qualified Jose.Jwk as JWT
import qualified Jose.Jwt as JWT
import Control.Monad.Except (liftEither)
import Data.Either.Combinators (mapLeft)
import Data.Text ()
import Data.Time.Clock (UTCTime, nominalDiffTimeToSeconds)
import Data.Time.Clock.POSIX (utcTimeToPOSIXSeconds)
import PostgREST.Auth.Types (AuthResult (..))
import PostgREST.Config (AppConfig (..), FilterExp (..), JSPath,
JSPathExp (..), audMatchesCfg)
import PostgREST.Error (Error (..),
JwtClaimsError (AudClaimNotStringOrArray, ExpClaimNotNumber, IatClaimNotNumber, JWTExpired, JWTIssuedAtFuture, JWTNotInAudience, JWTNotYetValid, NbfClaimNotNumber, ParsingClaimsFailed),
JwtDecodeError (..), JwtError (..))
import Data.Aeson ((.:?))
import Data.Aeson.Types (parseMaybe)
import Jose.Jwk (JwkSet)
import Protolude hiding (first)
parseAndDecodeClaims :: (MonadError Error m, MonadIO m) => JwkSet -> ByteString -> m JSON.Object
parseAndDecodeClaims jwkSet token = parseToken jwkSet token >>= decodeClaims
decodeClaims :: MonadError Error m => JWT.JwtContent -> m JSON.Object
decodeClaims (JWT.Jws (_, claims)) = maybe (throwError (JwtErr $ JwtClaimsErr ParsingClaimsFailed)) pure (JSON.decodeStrict claims)
decodeClaims _ = throwError $ JwtErr $ JwtDecodeErr UnsupportedTokenType
validateClaims :: MonadError Error m => UTCTime -> (Text -> Bool) -> JSON.Object -> m ()
validateClaims time audMatches claims = liftEither $ maybeToLeft () (fmap JwtErr . getAlt $ JwtClaimsErr <$> checkForErrors time audMatches claims)
data ValidAud = VAString Text | VAArray [Text] deriving Generic
instance JSON.FromJSON ValidAud where
parseJSON = JSON.genericParseJSON JSON.defaultOptions { JSON.sumEncoding = JSON.UntaggedValue }
checkForErrors :: (Applicative m, Monoid (m JwtClaimsError)) => UTCTime -> (Text -> Bool) -> JSON.Object -> m JwtClaimsError
checkForErrors time audMatches = mconcat
[
claim "exp" ExpClaimNotNumber $ inThePast JWTExpired
, claim "nbf" NbfClaimNotNumber $ inTheFuture JWTNotYetValid
, claim "iat" IatClaimNotNumber $ inTheFuture JWTIssuedAtFuture
, claim "aud" AudClaimNotStringOrArray $ checkValue (not . validAud) JWTNotInAudience
]
where
allowedSkewSeconds = 30 :: Int64
sciToInt = fromMaybe 0 . Sci.toBoundedInteger
toSec = floor . nominalDiffTimeToSeconds . utcTimeToPOSIXSeconds
now = toSec time
inTheFuture = checkTime ((now + allowedSkewSeconds) <)
inThePast = checkTime ((now - allowedSkewSeconds) >)
checkTime cond = checkValue (cond. sciToInt)
validAud = \case
(VAString aud) -> audMatches aud
(VAArray auds) -> null auds || any audMatches auds
checkValue invalid msg val =
if invalid val then
pure msg
else
mempty
claim key parseError checkParsed = maybe (pure parseError) (maybe mempty checkParsed) . parseMaybe (.:? key)
-- | Receives the JWT secret and audience (from config) and a JWT and returns a
-- JSON object of JWT claims.
parseToken :: (MonadError Error m, MonadIO m) => JwkSet -> ByteString -> m JWT.JwtContent
parseToken _ "" = throwError $ JwtErr $ JwtDecodeErr EmptyAuthHeader
parseToken secret tkn = do
-- secret <- liftEither . maybeToRight (JwtErr JwtSecretMissing) $ configJWKS
tknWith3Parts <- hasThreeParts tkn
eitherContent <- liftIO $ JWT.decode (JWT.keys secret) Nothing tknWith3Parts
liftEither . mapLeft (JwtErr . jwtDecodeError) $ eitherContent
--liftEither $ mapLeft JwtErr $ verifyClaims content
where
--hasThreeParts :: ByteString -> Either Error ByteString
hasThreeParts token = case length $ BS.split (BS.c2w '.') token of
3 -> pure token
n -> throwError $ JwtErr $ JwtDecodeErr $ UnexpectedParts n
jwtDecodeError :: JWT.JwtError -> JwtError
-- The only errors we can get from JWT.decode function are:
-- BadAlgorithm
-- KeyError
-- BadCrypto
jwtDecodeError (JWT.KeyError m) = JwtDecodeErr $ KeyError m
jwtDecodeError (JWT.BadAlgorithm m) = JwtDecodeErr $ BadAlgorithm m
jwtDecodeError JWT.BadCrypto = JwtDecodeErr BadCrypto
-- Control never reaches here, the decode function only returns the above three
jwtDecodeError _ = JwtDecodeErr UnreachableDecodeError
parseClaims :: (MonadError Error m, MonadIO m) => AppConfig -> UTCTime -> JSON.Object -> m AuthResult
parseClaims cfg@AppConfig{configJwtRoleClaimKey, configDbAnonRole} time mclaims = do
validateClaims time (audMatchesCfg cfg) mclaims
-- role defaults to anon if not specified in jwt
role <- liftEither . maybeToRight (JwtErr JwtTokenRequired) $
unquoted <$> walkJSPath (Just $ JSON.Object mclaims) configJwtRoleClaimKey <|> configDbAnonRole
pure AuthResult
{ authClaims = mclaims & KM.insert "role" (JSON.toJSON $ decodeUtf8 role)
, authRole = role
}
where
walkJSPath :: Maybe JSON.Value -> JSPath -> Maybe JSON.Value
walkJSPath x [] = x
walkJSPath (Just (JSON.Object o)) (JSPKey key:rest) = walkJSPath (KM.lookup (K.fromText key) o) rest
walkJSPath (Just (JSON.Array ar)) (JSPIdx idx:rest) = walkJSPath (ar V.!? idx) rest
walkJSPath (Just (JSON.Array ar)) [JSPFilter (EqualsCond txt)] = findFirstMatch (==) txt ar
walkJSPath (Just (JSON.Array ar)) [JSPFilter (NotEqualsCond txt)] = findFirstMatch (/=) txt ar
walkJSPath (Just (JSON.Array ar)) [JSPFilter (StartsWithCond txt)] = findFirstMatch T.isPrefixOf txt ar
walkJSPath (Just (JSON.Array ar)) [JSPFilter (EndsWithCond txt)] = findFirstMatch T.isSuffixOf txt ar
walkJSPath (Just (JSON.Array ar)) [JSPFilter (ContainsCond txt)] = findFirstMatch T.isInfixOf txt ar
walkJSPath _ _ = Nothing
findFirstMatch matchWith pattern = foldr checkMatch Nothing
where
checkMatch (JSON.String txt) acc
| pattern `matchWith` txt = Just $ JSON.String txt
| otherwise = acc
checkMatch _ acc = acc
unquoted :: JSON.Value -> BS.ByteString
unquoted (JSON.String t) = encodeUtf8 t
unquoted v = LBS.toStrict $ JSON.encode v
+77 -92
View File
@@ -1,114 +1,99 @@
{-| {-|
Module : PostgREST.Auth.JwtCache Module : PostgREST.Auth.JwtCache
Description : PostgREST JWT validation results Cache. Description : PostgREST Jwt Authentication Result Cache.
This module provides functions to deal with the JWT cache. This module provides functions to deal with the JWT cache
-} -}
{-# LANGUAGE ExistentialQuantification #-} {-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE StrictData #-}
module PostgREST.Auth.JwtCache module PostgREST.Auth.JwtCache
( init ( init
, update
, JwtCacheState , JwtCacheState
, lookupJwtCache , lookupJwtCache
, emptyCache
) where ) where
import qualified Data.Aeson as JSON import qualified Data.Aeson as JSON
import qualified Data.Aeson.KeyMap as KM import qualified Data.Aeson.KeyMap as KM
import qualified Data.Cache as C
import qualified Data.Scientific as Sci
import PostgREST.Error (Error (..), JwtError (JwtSecretMissing)) import Control.Debounce
import Control.Concurrent.STM (newTVarIO, readTVar, import Data.Time.Clock (UTCTime, nominalDiffTimeToSeconds)
writeTVar) import Data.Time.Clock.POSIX (utcTimeToPOSIXSeconds)
import Control.Concurrent.STM.TVar (TVar) import System.Clock (TimeSpec (..))
import Control.Monad.Error.Class (liftEither)
import Data.ByteString hiding (all, init)
import Data.IORef (IORef, newIORef,
readIORef, writeIORef)
import Jose.Jwk (JwkSet)
import PostgREST.Auth.Jwt (parseAndDecodeClaims)
import PostgREST.Cache.Sieve (alwaysValid)
import qualified PostgREST.Cache.Sieve as SC
import PostgREST.Config (AppConfig (..))
import PostgREST.Observation (Observation (JwtCacheEviction, JwtCacheLookup),
ObservationHandler)
import Protolude
data JwtCacheState = JwtCacheState ObservationHandler (IORef JwtCache) import PostgREST.Auth.Types (AuthResult (..))
import PostgREST.Error (Error (..))
class CacheVariant m v where import Protolude
cached :: SC.Cache m ByteString v -> ByteString -> ExceptT Error IO JSON.Object
{-| -- | JWT Cache and IO action that triggers purging old entries from the cache
Jwt caching can have three different configurations: data JwtCacheState = JwtCacheState
* missing JWT Key (no caching and throw error when JWT token present in the request) { jwtCache :: C.Cache ByteString AuthResult
* JWT cache turned off , purgeCache :: IO ()
* JWT cache turned on }
All three options are represented by JwtCache data type.
Handling of reconfiguration is centralized in this module.
-}
data JwtCache =
JwtNoJwks |
JwtNoCache JwkSet |
forall m v. CacheVariant m v => JwtCache JwkSet (TVar Int) (SC.Cache m ByteString v)
instance CacheVariant IO (Either Error JSON.Object) where
cached c = lift . SC.cached c >=> liftEither
instance CacheVariant (ExceptT Error IO) JSON.Object where
cached = SC.cached
decode :: JwtCache -> ByteString -> ExceptT Error IO JSON.Object
decode JwtNoJwks = const $ throwError (JwtErr JwtSecretMissing)
decode (JwtNoCache key) = parseAndDecodeClaims key
decode (JwtCache _ _ c) = cached c
-- | Reconfigure JWT caching and update JwtCacheState accordingly
update :: JwtCacheState -> AppConfig -> IO ()
update (JwtCacheState observationHandler jwtCacheState) config@AppConfig{configJWKS, configJwtCacheMaxEntries} =
let reinitialize =
newJwtCache config observationHandler
>>= writeIORef jwtCacheState
in
readIORef jwtCacheState >>= \case
(JwtCache decodingKey maxSize _) ->
if configJWKS /= Just decodingKey || configJwtCacheMaxEntries <= 0 then
-- reinitialize if key changed or cache disabled
reinitialize
else
-- max size changed - set it and let the cache shrink itself if necessary
atomically $ writeTVar maxSize configJwtCacheMaxEntries
_ -> reinitialize
init :: AppConfig -> ObservationHandler -> IO JwtCacheState
init config = fmap (<$>) JwtCacheState <*> (newJwtCache config >=> newIORef)
-- | Initialize JwtCacheState -- | Initialize JwtCacheState
newJwtCache :: AppConfig -> ObservationHandler -> IO JwtCache init :: IO JwtCacheState
newJwtCache AppConfig{configJWKS, configJwtCacheMaxEntries} observationHandler = do init = do
maybe (pure JwtNoJwks) initCache configJWKS cache <- C.newCache Nothing -- no default expiration
where -- purgeExpired has O(n^2) complexity
initCache key = if configJwtCacheMaxEntries <= 0 then pure (JwtNoCache key) else createCache key configJwtCacheMaxEntries -- so we wrap it in debounce to make sure it:
-- 1) is executed asynchronously
-- 2) only a single purge operation is running at a time
debounce <- mkDebounce defaultDebounceSettings
-- debounceFreq is set to default 1 second
{ debounceAction = C.purgeExpired cache
, debounceEdge = leadingEdge
}
pure $ JwtCacheState cache debounce
createCache key maxSize = do -- | Used to retrieve and insert JWT to JWT Cache
maxSizeTVar <- newTVarIO maxSize lookupJwtCache :: JwtCacheState -> ByteString -> Int -> IO (Either Error AuthResult) -> UTCTime -> IO (Either Error AuthResult)
JwtCache key maxSizeTVar <$> lookupJwtCache JwtCacheState{jwtCache, purgeCache} token maxLifetime parseJwt utc = do
notCachingErrors (readTVar maxSizeTVar) key checkCache <- C.lookup jwtCache token
authResult <- maybe parseJwt (pure . Right) checkCache
notCachingErrors :: STM Int -> JwkSet -> IO (SC.Cache (ExceptT Error IO) ByteString JSON.Object) case (authResult,checkCache) of
notCachingErrors maxSize key = SC.cacheIO (SC.CacheConfig maxSize -- From comment:
(parseAndDecodeClaims key) -- https://github.com/PostgREST/postgrest/pull/3801#discussion_r1857987914
(lift . observationHandler . JwtCacheLookup) -- lookup metrics --
(const . const $ lift $ observationHandler JwtCacheEviction) -- evictions metrics -- We purge expired cache entries on a cache miss
alwaysValid) -- no invalidation for now -- The reasoning is that:
--
-- 1. We expect it to be rare (otherwise there is no point of the cache)
-- 2. It makes sure the cache is not growing (as inserting new entries
-- does garbage collection)
-- 3. Since this is time expiration based cache there is no real risk of
-- starvation - sooner or later we are going to have a cache miss.
lookupJwtCache :: JwtCacheState -> Maybe ByteString -> ExceptT Error IO JSON.Object (Right res, Nothing) -> do -- cache miss
lookupJwtCache (JwtCacheState _ cacheState) k = liftIO (readIORef cacheState) >>= flip (maybe (pure KM.empty)) k . decode
let timeSpec = getTimeSpec res maxLifetime utc
-- insert new cache entry
C.insert' jwtCache (Just timeSpec) token res
-- Execute IO action to purge the cache
-- It is assumed this action returns immidiately
-- so that request processing is not blocked.
purgeCache
_ -> pure ()
return authResult
-- Used to extract JWT exp claim and add to JWT Cache
getTimeSpec :: AuthResult -> Int -> UTCTime -> TimeSpec
getTimeSpec res maxLifetime utc = do
let expireJSON = KM.lookup "exp" (authClaims res)
utcToSecs = floor . nominalDiffTimeToSeconds . utcTimeToPOSIXSeconds
sciToInt = fromMaybe 0 . Sci.toBoundedInteger
case expireJSON of
Just (JSON.Number seconds) -> TimeSpec (sciToInt seconds - utcToSecs utc) 0
_ -> TimeSpec (fromIntegral maxLifetime :: Int64) 0
-- | Empty the cache (done when the config is reloaded)
emptyCache :: JwtCacheState -> IO ()
emptyCache JwtCacheState{jwtCache} = C.purge jwtCache
+122 -28
View File
@@ -1,4 +1,5 @@
{-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE QuasiQuotes #-}
{-# LANGUAGE RecordWildCards #-} {-# LANGUAGE RecordWildCards #-}
module PostgREST.CLI module PostgREST.CLI
( main ( main
@@ -13,6 +14,8 @@ import qualified Data.ByteString.Lazy as LBS
import qualified Hasql.Transaction.Sessions as SQL import qualified Hasql.Transaction.Sessions as SQL
import qualified Options.Applicative as O import qualified Options.Applicative as O
import Text.Heredoc (str)
import PostgREST.AppState (AppState) import PostgREST.AppState (AppState)
import PostgREST.Config (AppConfig (..)) import PostgREST.Config (AppConfig (..))
import PostgREST.Observation (Observation (..)) import PostgREST.Observation (Observation (..))
@@ -21,7 +24,6 @@ import PostgREST.Version (prettyVersion)
import qualified PostgREST.App as App import qualified PostgREST.App as App
import qualified PostgREST.AppState as AppState import qualified PostgREST.AppState as AppState
import qualified PostgREST.Client as Client
import qualified PostgREST.Config as Config import qualified PostgREST.Config as Config
import Protolude import Protolude
@@ -29,26 +31,16 @@ import Protolude
main :: CLI -> IO () main :: CLI -> IO ()
main CLI{cliCommand, cliPath} = do main CLI{cliCommand, cliPath} = do
conf <- conf@AppConfig{..} <-
either panic identity <$> Config.readAppConfig mempty cliPath Nothing mempty mempty either panic identity <$> Config.readAppConfig mempty cliPath Nothing mempty mempty
case cliCommand of
Client adminCmd -> runClientCommand conf adminCmd
Run runCmd -> runAppCommand conf runCmd
-- | Run command using http-client to communicate with an already running postgrest
runClientCommand :: AppConfig -> ClientCommand -> IO ()
runClientCommand conf CmdReady = Client.ready conf
-- | Run postgrest with command
runAppCommand :: AppConfig -> RunCommand -> IO ()
runAppCommand conf@AppConfig{..} runCmd = do
-- Per https://github.com/PostgREST/postgrest/issues/268, we want to -- Per https://github.com/PostgREST/postgrest/issues/268, we want to
-- explicitly close the connections to PostgreSQL on shutdown. -- explicitly close the connections to PostgreSQL on shutdown.
-- 'AppState.destroy' takes care of that. -- 'AppState.destroy' takes care of that.
bracket bracket
(AppState.init conf) (AppState.init conf)
AppState.destroy AppState.destroy
(\appState -> case runCmd of (\appState -> case cliCommand of
CmdDumpConfig -> do CmdDumpConfig -> do
when configDbConfig $ AppState.readInDbConfig True appState when configDbConfig $ AppState.readInDbConfig True appState
putStr . Config.toText =<< AppState.getConfig appState putStr . Config.toText =<< AppState.getConfig appState
@@ -79,13 +71,6 @@ data CLI = CLI
} }
data Command data Command
= Client ClientCommand
| Run RunCommand
data ClientCommand
= CmdReady
data RunCommand
= CmdRun = CmdRun
| CmdDumpConfig | CmdDumpConfig
| CmdDumpSchema | CmdDumpSchema
@@ -112,7 +97,7 @@ readCLIShowHelp =
<> O.help "Show the version information" <> O.help "Show the version information"
exampleParser = exampleParser =
O.infoOption Config.exampleConfigFile $ O.infoOption exampleConfigFile $
O.long "example" O.long "example"
<> O.short 'e' <> O.short 'e'
<> O.help "Show an example configuration file" <> O.help "Show an example configuration file"
@@ -120,7 +105,7 @@ readCLIShowHelp =
cliParser :: O.Parser CLI cliParser :: O.Parser CLI
cliParser = cliParser =
CLI CLI
<$> (dumpConfigFlag <|> dumpSchemaFlag <|> readyFlag) <$> (dumpConfigFlag <|> dumpSchemaFlag)
<*> O.optional configFileOption <*> O.optional configFileOption
configFileOption = configFileOption =
@@ -129,16 +114,125 @@ readCLIShowHelp =
<> O.help "Path to configuration file" <> O.help "Path to configuration file"
dumpConfigFlag = dumpConfigFlag =
O.flag (Run CmdRun) (Run CmdDumpConfig) $ O.flag CmdRun CmdDumpConfig $
O.long "dump-config" O.long "dump-config"
<> O.help "Dump loaded configuration and exit" <> O.help "Dump loaded configuration and exit"
dumpSchemaFlag = dumpSchemaFlag =
O.flag (Run CmdRun) (Run CmdDumpSchema) $ O.flag CmdRun CmdDumpSchema $
O.long "dump-schema" O.long "dump-schema"
<> O.help "Dump loaded schema as JSON and exit (for debugging, output structure is unstable)" <> O.help "Dump loaded schema as JSON and exit (for debugging, output structure is unstable)"
readyFlag = exampleConfigFile :: [Char]
O.flag (Run CmdRun) (Client CmdReady) $ exampleConfigFile =
O.long "ready" [str|## Admin server used for checks. It's disabled by default unless a port is specified.
<> O.help "Checks the health of PostgREST by doing a request on the admin server /ready endpoint" |# admin-server-port = 3001
|
|## The database role to use when no client authentication is provided
|# db-anon-role = "anon"
|
|## Notification channel for reloading the schema cache
|db-channel = "pgrst"
|
|## Enable or disable the notification channel
|db-channel-enabled = true
|
|## Enable in-database configuration
|db-config = true
|
|## Function for in-database configuration
|## db-pre-config = "postgrest.pre_config"
|
|## Extra schemas to add to the search_path of every request
|db-extra-search-path = "public"
|
|## Limit rows in response
|# db-max-rows = 1000
|
|## Allow getting the EXPLAIN plan through the `Accept: application/vnd.pgrst.plan` header
|# db-plan-enabled = false
|
|## Number of open connections in the pool
|db-pool = 10
|
|## Time in seconds to wait to acquire a slot from the connection pool
|# db-pool-acquisition-timeout = 10
|
|## Time in seconds after which to recycle pool connections
|# db-pool-max-lifetime = 1800
|
|## Time in seconds after which to recycle unused pool connections
|# db-pool-max-idletime = 30
|
|## Allow automatic database connection retrying
|# db-pool-automatic-recovery = true
|
|## Stored proc to exec immediately after auth
|# db-pre-request = "stored_proc_name"
|
|## Enable or disable prepared statements. disabling is only necessary when behind a connection pooler.
|## When disabled, statements will be parametrized but won't be prepared.
|db-prepared-statements = true
|
|## The name of which database schema to expose to REST clients
|db-schemas = "public"
|
|## How to terminate database transactions
|## Possible values are:
|## commit (default)
|## Transaction is always committed, this can not be overriden
|## commit-allow-override
|## Transaction is committed, but can be overriden with Prefer tx=rollback header
|## rollback
|## Transaction is always rolled back, this can not be overriden
|## rollback-allow-override
|## Transaction is rolled back, but can be overriden with Prefer tx=commit header
|db-tx-end = "commit"
|
|## The standard connection URI format, documented at
|## https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING
|db-uri = "postgresql://"
|
|# jwt-aud = "your_audience_claim"
|
|## Jspath to the role claim key
|jwt-role-claim-key = ".role"
|
|## Choose a secret, JSON Web Key (or set) to enable JWT auth
|## (use "@filename" to load from separate file)
|# jwt-secret = "secret_with_at_least_32_characters"
|jwt-secret-is-base64 = false
|
|## Enables and set JWT Cache max lifetime, disables caching with 0
|# jwt-cache-max-lifetime = 0
|
|## Logging level, the admitted values are: crit, error, warn, info and debug.
|log-level = "error"
|
|## Log the requested SQL query at the current log-level.
|log-query = "disabled"
|
|## Determine if the OpenAPI output should follow or ignore role privileges or be disabled entirely.
|## Admitted values: follow-privileges, ignore-privileges, disabled
|openapi-mode = "follow-privileges"
|
|## Base url for the OpenAPI output
|openapi-server-proxy-uri = ""
|
|## Configurable CORS origins
|# server-cors-allowed-origins = ""
|
|server-host = "!4"
|server-port = 3000
|
|## Allow getting the request-response timing information through the `Server-Timing` header
|server-timing-enabled = false
|
|## Unix socket location
|## if specified it takes precedence over server-port
|# server-unix-socket = "/tmp/pgrst.sock"
|
|## Unix socket file mode
|## When none is provided, 660 is applied by default
|# server-unix-socket-mode = "660"
|]
-218
View File
@@ -1,218 +0,0 @@
{-|
Module : PostgREST.Cache.Sieve
Description : PostgREST cache implementation based on Sieve algorithm.
This module provides implementation of a mutable cache on Sieve algorithm.
-}
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE GADTs #-}
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE PolyKinds #-}
{-# LANGUAGE RecordWildCards #-}
{-# LANGUAGE RecursiveDo #-}
{-# LANGUAGE StrictData #-}
{-# LANGUAGE TupleSections #-}
module PostgREST.Cache.Sieve (
Cache
, CacheConfig (..)
, Discard (..)
, alwaysValid
, cache
, cacheIO
, cached
)
where
import Control.Concurrent.STM
import Control.Monad.Extra (whileM)
import Data.Some
import qualified Focus as F
import Protolude hiding (elem, head)
import qualified StmHamt.SizedHamt as SH
data ListNode k v (b :: Bool) = ListNode {
nextPtr :: NodePtr k v,
prevNextPtrPtr :: NodePtrPtr k v,
elem :: NodeElem k v b
}
data NodeElem :: Type -> Type -> Bool -> Type where
Head :: {
entries :: SH.SizedHamt (HamtEntry k v),
finger :: NodePtrPtr k v
} -> NodeElem k v False
Entry :: Hashable k => {
visited :: TVar Bool,
ekey :: k,
entryValue :: v
} -> NodeElem k v True
type HamtEntry k v = ListNode k v True
type AnyNode k v = Some (ListNode k v)
type NodePtr k v = TVar (AnyNode k v)
type NodePtrPtr k v = TVar (NodePtr k v)
data Discard m v = Refresh (m ()) | Invalid (m v)
data Cache m k v = (MonadIO m, Hashable k) => Cache (ListNode k v False) (CacheConfig m k v)
data CacheConfig m k v = CacheConfig {
maxSize :: STM Int,
load :: k -> m v,
requestListener :: Bool -> m (),
evictionListener :: k -> v -> m (),
validator :: m (k -> v -> Maybe (Discard m v))
}
alwaysValid :: Applicative m => m (k -> v -> Maybe (Discard m v))
alwaysValid = pure (const . const Nothing)
cacheIO :: (MonadIO m, Hashable k) => CacheConfig m k v -> IO (Cache m k v)
cacheIO = atomically . cache
cache :: (MonadIO m, Hashable k) => CacheConfig m k v -> STM (Cache m k v)
cache cacheConfig = mdo
tail <- newTVar (Some head)
entries <- SH.new
finger <- newTVar tail
head <- ListNode tail <$> newTVar tail <*> pure Head {..}
pure $ Cache head cacheConfig
cached :: Cache m k v -> k -> m v
cached (Cache head@ListNode{prevNextPtrPtr=neck, elem=Head{..}} CacheConfig{..}) k = do
checkValid <- validator
tryMaybe
-- Fast path: lookup value, update stats and return the value if found and valid
((liftIO . atomically) (lookup checkValid) >>= notify (requestListener . isJust) >>= validate)
-- Slow path: load/calculate value and insert it (if still not found)
(do
value <- load k
whileM (not <$> tryInsert value)
pure value)
where
tryMaybe f notFound = f >>= maybe notFound pure
notify = ((<$) <*>)
validate = fmap join . traverse (\case
-- valid value
(Right v) -> pure $ Just v
-- refresh value
(Left (Refresh act)) -> act $> Nothing
-- discard value and return alt result
(Left (Invalid res)) -> Just <$> res)
lookup checkValid = SH.focus focus (ekey . elem) k entries
where
focus = F.Focus
-- not found
(pure (Nothing, F.Leave))
-- found
-- check entry validity
(\e@ListNode{elem=Entry{visited, entryValue}} ->
maybe
-- entry valid
(mark visited True $> (Just $ Right entryValue, F.Leave))
-- entry invalid
-- remove it
((removeEntry e $>) . (, F.Remove) . Just . Left)
(checkValid k entryValue)
)
mark t b = whenM ((/= b) <$> readTVar t) (writeTVar t b)
-- perform a single entry eviction and possibly insertion atomically
-- returning False if could not insert
-- (either because entry currently pointed by the finger was visited
-- or because after this entry eviction the cache is still full)
-- so that other threads don't have to wait when visiting entries.
-- First check if entry is still not in the cache - this time inside transaction.
--
-- Execute evictionListener if an entry was evicted
tryInsert value = do
(result, evicted) <- liftIO . atomically $ do
-- Use SH.focus to performa a single lookup instead of 2
-- we cannot modify Hamt from inside focus
-- so if there is any entry to remove
-- we need to delete it after
(res, evictedKey) <- SH.focus focus (ekey . elem) k entries
case evictedKey of
(Just Entry{ekey=entryKey, entryValue}) -> do
SH.focus F.delete (ekey . elem) entryKey entries
pure (res, evictionListener entryKey entryValue)
Nothing -> pure (res, pure ())
evicted $> result
where
focus = F.Focus (do
(hasSpace, evictedKey) <- evictionStep
if hasSpace then do
entry <- newLinkedEntry value
-- done, maybe evicted, insert entry
pure ((True, evictedKey), F.Set entry)
else
-- not done, maybe evicted, don't modify entries
pure ((False, evictedKey), F.Leave))
-- Entry found case
(\ListNode{elem=Entry{visited}} -> do
-- mark as visited
mark visited True
-- done, no evictions, don't modify entries
pure ((True, Nothing), F.Leave))
-- if the cache is full precoesses a single node
-- removing it if it is marked as unvisited
-- or clearing visited mark
-- returns True if there is space in the cache
-- puts evictionListener in state if an entry was evicted
evictionStep = do
currDiff <- liftA2 (-) (SH.size entries) (max 1 <$> maxSize)
if currDiff >= 0 then do
-- no space in the cache
-- need to evict an entry
(nextFinger, evictedKey) <- readTVar finger >>= evict
writeTVar finger nextFinger
-- return if enough space and evicted key if any
pure (isJust evictedKey && currDiff == 0, evictedKey)
else
-- there is space in the cache
pure (True, Nothing)
evict :: TVar (Some (ListNode k v)) -> STM (NodePtr k v, Maybe (NodeElem k v True))
evict = readTVar >=> \case
(Some e@ListNode{nextPtr, prevNextPtrPtr, elem=elem@Entry{visited}}) -> do
ifM (readTVar visited)
(writeTVar visited False $> (nextPtr, Nothing))
(unlinkEntry e *> fmap (, Just elem) (readTVar prevNextPtrPtr))
-- skip head
(Some ListNode{nextPtr, elem=Head{}}) -> evict nextPtr
unlinkEntry :: HamtEntry k v -> STM ()
unlinkEntry (ListNode{nextPtr, prevNextPtrPtr=currPrev}) = do
nextEntry <- readTVar nextPtr
withSome nextEntry $ \e -> do
prevNextPtr <- readTVar currPrev
writeTVar (prevNextPtrPtr e) prevNextPtr
writeTVar prevNextPtr nextEntry
newLinkedEntry v = do
oldNeckNextPtr <- readTVar neck
newNeckNextPtr <- newTVar (Some head)
newNeck <- ListNode newNeckNextPtr <$>
newTVar oldNeckNextPtr <*>
(Entry <$> newTVar False <*> pure k <*> pure v)
-- update pointers
writeTVar oldNeckNextPtr (Some newNeck)
writeTVar neck newNeckNextPtr
-- return HAMT entry
pure newNeck
removeEntry = fmap (*>) unlinkEntry <*> adjustFinger
adjustFinger ListNode{nextPtr, prevNextPtrPtr} =
whenM ((nextPtr ==) <$> readTVar finger) $
readTVar prevNextPtrPtr >>= writeTVar finger
-100
View File
@@ -1,100 +0,0 @@
{-|
Module : PostgREST.Client
Description : PostgREST HTTP client
-}
{-# LANGUAGE NamedFieldPuns #-}
module PostgREST.Client
( ready
) where
import qualified Data.Text as T
import qualified Network.HTTP.Client as HC
import qualified Network.HTTP.Types.Status as HTTP
import Network.HTTP.Client (HttpException (..))
import System.IO (hFlush)
import PostgREST.Config (AppConfig (..))
import PostgREST.Network (isSpecialHostName)
import Protolude
data PgrstClientError
= NoAdminServer
| NoSpecialHostNamesAllowed Text
| PostgRESTNotReady Text
| HTTPConnectionRefused Text
| HTTPExceptionInvalidURL Text
-- | This is invoked by the CLI "--ready" flag.
-- The http-client sends and a request to /ready endpoint
-- and exits with success or failure.
ready :: AppConfig -> IO ()
ready AppConfig{configAdminServerHost, configAdminServerPort} = do
client <- HC.newManager HC.defaultManagerSettings
readyURL <- getURL
req <- HC.parseRequest (T.unpack readyURL) `catch` handleHttpException
resp <- HC.httpLbs req client `catch` handleHttpException
let status = HC.responseStatus resp
if status >= HTTP.status200 && status < HTTP.status300
then printAndExitWithSuccess $ "OK: " <> readyURL
else printAndExitWithFailure $ clientErrorMsg (PostgRESTNotReady readyURL)
where
getURL :: IO Text
getURL =
-- Here, we have three cases:
-- 1. If the admin port config is not defined, we exit
-- with "no admin server error"
-- 2. Otherwise, if admin server is running, then we check if
-- postgrest server-host is configured with special hostname like "*4",
-- if it is, we fail with "no special hostname allowed with "--ready".
-- The reason for this is that we can't know the actual address.
-- 3. Finally, if we know the "actual" hostname and the port, then we
-- construct the URL and return it.
case configAdminServerPort of
Nothing -> printAndExitWithFailure $ clientErrorMsg NoAdminServer
Just port ->
if isSpecialHostName configAdminServerHost
then printAndExitWithFailure $ clientErrorMsg (NoSpecialHostNamesAllowed configAdminServerHost)
else return $ makeReadyUrl port
-- NOTE: http-client automatically resolves hostnames
makeReadyUrl :: Int -> Text
makeReadyUrl p = "http://" <> wrapIfIpv6 configAdminServerHost <> ":" <> (T.pack . show) p <> "/ready"
where
-- IPv6 needs to wrapped in [], it has ':' as separator
wrapIfIpv6 :: Text -> Text
wrapIfIpv6 s
| T.any (== ':') s = "[" <> s <> "]"
| otherwise = s
-- | Handle HTTP exception for "http-client" requests
handleHttpException :: HttpException -> IO a
handleHttpException (HttpExceptionRequest req _) = do
let url = show (HC.getUri req)
printAndExitWithFailure $ clientErrorMsg (HTTPConnectionRefused $ T.pack url)
handleHttpException (InvalidUrlException url _) = do
printAndExitWithFailure $ clientErrorMsg (HTTPExceptionInvalidURL $ T.pack url)
-- | Print the message on stdout and exit with success
printAndExitWithSuccess :: Text -> IO a
printAndExitWithSuccess msg = putStrLn (T.unpack msg) >> hFlush stdout >> exitSuccess
-- | Print the message on stderr and exit with failure
printAndExitWithFailure :: Text -> IO a
printAndExitWithFailure msg = hPutStrLn stderr (T.unpack msg) >> hFlush stderr >> exitWith (ExitFailure 1)
-- | Pgrst client error to error message
clientErrorMsg :: PgrstClientError -> Text
clientErrorMsg err = "ERROR: " <>
case err of
NoAdminServer -> "Admin server is not running. Please check admin-server-port config."
NoSpecialHostNamesAllowed host ->
"The `--ready` flag cannot be used when server-host is configured as \"" <> host <> "\". "
<> "Please update your server-host config to \"localhost\"."
PostgRESTNotReady url -> url
HTTPConnectionRefused url -> "connection refused to " <> url
HTTPExceptionInvalidURL url -> "invalid url - " <> url
+25 -128
View File
@@ -17,6 +17,7 @@ module PostgREST.Config
, JSPathExp(..) , JSPathExp(..)
, FilterExp(..) , FilterExp(..)
, LogLevel(..) , LogLevel(..)
, LogQuery(..)
, OpenAPIMode(..) , OpenAPIMode(..)
, Proxy(..) , Proxy(..)
, toText , toText
@@ -27,8 +28,6 @@ module PostgREST.Config
, parseSecret , parseSecret
, addFallbackAppName , addFallbackAppName
, addTargetSessionAttrs , addTargetSessionAttrs
, exampleConfigFile
, audMatchesCfg
) where ) where
import qualified Data.Aeson as JSON import qualified Data.Aeson as JSON
@@ -37,7 +36,6 @@ import qualified Data.ByteString.Base64 as B64
import qualified Data.CaseInsensitive as CI import qualified Data.CaseInsensitive as CI
import qualified Data.Configurator as C import qualified Data.Configurator as C
import qualified Data.Map.Strict as M import qualified Data.Map.Strict as M
import qualified Data.String as S
import qualified Data.Text as T import qualified Data.Text as T
import qualified Data.Text.Encoding as T import qualified Data.Text.Encoding as T
import qualified Jose.Jwa as JWT import qualified Jose.Jwa as JWT
@@ -68,8 +66,6 @@ import PostgREST.SchemaCache.Identifiers (QualifiedIdentifier, dumpQi,
import Protolude hiding (Proxy, toList) import Protolude hiding (Proxy, toList)
audMatchesCfg :: AppConfig -> Text -> Bool
audMatchesCfg = maybe (const True) (==) . configJwtAudience
data AppConfig = AppConfig data AppConfig = AppConfig
{ configAppSettings :: [(Text, Text)] { configAppSettings :: [(Text, Text)]
@@ -101,9 +97,9 @@ data AppConfig = AppConfig
, configJwtRoleClaimKey :: JSPath , configJwtRoleClaimKey :: JSPath
, configJwtSecret :: Maybe BS.ByteString , configJwtSecret :: Maybe BS.ByteString
, configJwtSecretIsBase64 :: Bool , configJwtSecretIsBase64 :: Bool
, configJwtCacheMaxEntries :: Int , configJwtCacheMaxLifetime :: Int
, configLogLevel :: LogLevel , configLogLevel :: LogLevel
, configLogQuery :: Bool , configLogQuery :: LogQuery
, configOpenApiMode :: OpenAPIMode , configOpenApiMode :: OpenAPIMode
, configOpenApiSecurityActive :: Bool , configOpenApiSecurityActive :: Bool
, configOpenApiServerProxyUri :: Maybe Text , configOpenApiServerProxyUri :: Maybe Text
@@ -118,7 +114,7 @@ data AppConfig = AppConfig
, configAdminServerPort :: Maybe Int , configAdminServerPort :: Maybe Int
, configRoleSettings :: RoleSettings , configRoleSettings :: RoleSettings
, configRoleIsoLvl :: RoleIsolationLvl , configRoleIsoLvl :: RoleIsolationLvl
, configInternalSCQuerySleep :: Maybe Int32 , configInternalSCSleep :: Maybe Int32
} }
data LogLevel = LogCrit | LogError | LogWarn | LogInfo | LogDebug data LogLevel = LogCrit | LogError | LogWarn | LogInfo | LogDebug
@@ -132,6 +128,14 @@ dumpLogLevel = \case
LogInfo -> "info" LogInfo -> "info"
LogDebug -> "debug" LogDebug -> "debug"
data LogQuery = LogQueryMain | LogQueryDisabled
deriving (Eq)
dumpLogQuery :: LogQuery -> Text
dumpLogQuery = \case
LogQueryMain -> "main-query"
LogQueryDisabled -> "disabled"
data OpenAPIMode = OAFollowPriv | OAIgnorePriv | OADisabled data OpenAPIMode = OAFollowPriv | OAIgnorePriv | OADisabled
deriving Eq deriving Eq
@@ -173,9 +177,9 @@ toText conf =
,("jwt-role-claim-key", q . T.intercalate mempty . fmap dumpJSPath . configJwtRoleClaimKey) ,("jwt-role-claim-key", q . T.intercalate mempty . fmap dumpJSPath . configJwtRoleClaimKey)
,("jwt-secret", q . T.decodeUtf8 . showJwtSecret) ,("jwt-secret", q . T.decodeUtf8 . showJwtSecret)
,("jwt-secret-is-base64", T.toLower . show . configJwtSecretIsBase64) ,("jwt-secret-is-base64", T.toLower . show . configJwtSecretIsBase64)
,("jwt-cache-max-entries", show . configJwtCacheMaxEntries) ,("jwt-cache-max-lifetime", show . configJwtCacheMaxLifetime)
,("log-level", q . dumpLogLevel . configLogLevel) ,("log-level", q . dumpLogLevel . configLogLevel)
,("log-query", T.toLower . show . configLogQuery) ,("log-query", q . dumpLogQuery . configLogQuery)
,("openapi-mode", q . dumpOpenApiMode . configOpenApiMode) ,("openapi-mode", q . dumpOpenApiMode . configOpenApiMode)
,("openapi-security-active", T.toLower . show . configOpenApiSecurityActive) ,("openapi-security-active", T.toLower . show . configOpenApiSecurityActive)
,("openapi-server-proxy-uri", q . fromMaybe mempty . configOpenApiServerProxyUri) ,("openapi-server-proxy-uri", q . fromMaybe mempty . configOpenApiServerProxyUri)
@@ -283,9 +287,9 @@ parser optPath env dbSettings roleSettings roleIsolationLvl =
<*> (fromMaybe False <$> optWithAlias <*> (fromMaybe False <$> optWithAlias
(optBool "jwt-secret-is-base64") (optBool "jwt-secret-is-base64")
(optBool "secret-is-base64")) (optBool "secret-is-base64"))
<*> (fromMaybe 1000 <$> optInt "jwt-cache-max-entries") <*> (fromMaybe 0 <$> optInt "jwt-cache-max-lifetime")
<*> parseLogLevel "log-level" <*> parseLogLevel "log-level"
<*> (fromMaybe False <$> optBool "log-query") <*> parseLogQuery "log-query"
<*> parseOpenAPIMode "openapi-mode" <*> parseOpenAPIMode "openapi-mode"
<*> (fromMaybe False <$> optBool "openapi-security-active") <*> (fromMaybe False <$> optBool "openapi-security-active")
<*> parseOpenAPIServerProxyURI "openapi-server-proxy-uri" <*> parseOpenAPIServerProxyURI "openapi-server-proxy-uri"
@@ -301,7 +305,7 @@ parser optPath env dbSettings roleSettings roleIsolationLvl =
<*> parseAdminServerPort "admin-server-port" <*> parseAdminServerPort "admin-server-port"
<*> pure roleSettings <*> pure roleSettings
<*> pure roleIsolationLvl <*> pure roleIsolationLvl
<*> optInt "internal-schema-cache-query-sleep" <*> optInt "internal-schema-cache-sleep"
where where
parseAppSettings :: C.Key -> C.Parser C.Config [(Text, Text)] parseAppSettings :: C.Key -> C.Parser C.Config [(Text, Text)]
parseAppSettings key = addFromEnv . fmap (fmap coerceText) <$> C.subassocs key C.value parseAppSettings key = addFromEnv . fmap (fmap coerceText) <$> C.subassocs key C.value
@@ -361,6 +365,14 @@ parser optPath env dbSettings roleSettings roleIsolationLvl =
Just "debug" -> pure LogDebug Just "debug" -> pure LogDebug
Just _ -> fail "Invalid logging level. Check your configuration." Just _ -> fail "Invalid logging level. Check your configuration."
parseLogQuery :: C.Key -> C.Parser C.Config LogQuery
parseLogQuery k =
optString k >>= \case
Nothing -> pure LogQueryDisabled
Just "disabled" -> pure LogQueryDisabled
Just "main-query" -> pure LogQueryMain
Just _ -> fail "Invalid SQL logging value. Check your configuration."
parseTxEnd :: C.Key -> ((Bool, Bool) -> Bool) -> C.Parser C.Config Bool parseTxEnd :: C.Key -> ((Bool, Bool) -> Bool) -> C.Parser C.Config Bool
parseTxEnd k f = parseTxEnd k f =
optString k >>= \case optString k >>= \case
@@ -616,118 +628,3 @@ addConnStringOption dbUri key val = dbUri <>
uriFmt = key <> "=" <> toS (escapeURIString isUnescapedInURIComponent $ toS val) uriFmt = key <> "=" <> toS (escapeURIString isUnescapedInURIComponent $ toS val)
keyValFmt = key <> "=" <> "'" <> T.replace "'" "\\'" val <> "'" keyValFmt = key <> "=" <> "'" <> T.replace "'" "\\'" val <> "'"
lookAtOptions x = T.breakOn "?" . snd $ T.breakOnEnd "@" x -- start from after `@` to not mess passwords that include `?`, see https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS lookAtOptions x = T.breakOn "?" . snd $ T.breakOnEnd "@" x -- start from after `@` to not mess passwords that include `?`, see https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS
-- | Example config file displayed on postgrest "--example" flag
exampleConfigFile :: [Char]
exampleConfigFile = S.unlines
[ "## Admin server used for checks. It's disabled by default unless a port is specified."
, "# admin-server-port = 3001"
, ""
, "## The database role to use when no client authentication is provided"
, "# db-anon-role = \"anon\""
, ""
, "## Notification channel for reloading the schema cache"
, "db-channel = \"pgrst\""
, ""
, "## Enable or disable the notification channel"
, "db-channel-enabled = true"
, ""
, "## Enable in-database configuration"
, "db-config = true"
, ""
, "## Function for in-database configuration"
, "## db-pre-config = \"postgrest.pre_config\""
, ""
, "## Extra schemas to add to the search_path of every request"
, "db-extra-search-path = \"public\""
, ""
, "## Limit rows in response"
, "# db-max-rows = 1000"
, ""
, "## Allow getting the EXPLAIN plan through the `Accept: application/vnd.pgrst.plan` header"
, "# db-plan-enabled = false"
, ""
, "## Number of open connections in the pool"
, "db-pool = 10"
, ""
, "## Time in seconds to wait to acquire a slot from the connection pool"
, "# db-pool-acquisition-timeout = 10"
, ""
, "## Time in seconds after which to recycle pool connections"
, "# db-pool-max-lifetime = 1800"
, ""
, "## Time in seconds after which to recycle unused pool connections"
, "# db-pool-max-idletime = 30"
, ""
, "## Allow automatic database connection retrying"
, "# db-pool-automatic-recovery = true"
, ""
, "## Stored proc to exec immediately after auth"
, "# db-pre-request = \"stored_proc_name\""
, ""
, "## Enable or disable prepared statements. disabling is only necessary when behind a connection pooler."
, "## When disabled, statements will be parametrized but won't be prepared."
, "db-prepared-statements = true"
, ""
, "## The name of which database schema to expose to REST clients"
, "db-schemas = \"public\""
, ""
, "## How to terminate database transactions"
, "## Possible values are:"
, "## commit (default)"
, "## Transaction is always committed, this can not be overriden"
, "## commit-allow-override"
, "## Transaction is committed, but can be overriden with Prefer tx=rollback header"
, "## rollback"
, "## Transaction is always rolled back, this can not be overriden"
, "## rollback-allow-override"
, "## Transaction is rolled back, but can be overriden with Prefer tx=commit header"
, "db-tx-end = \"commit\""
, ""
, "## The standard connection URI format, documented at"
, "## https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING"
, "db-uri = \"postgresql://\""
, ""
, "# jwt-aud = \"your_audience_claim\""
, ""
, "## Jspath to the role claim key"
, "jwt-role-claim-key = \".role\""
, ""
, "## Choose a secret, JSON Web Key (or set) to enable JWT auth"
, "## (use \"@filename\" to load from separate file)"
, "# jwt-secret = \"secret_with_at_least_32_characters\""
, "jwt-secret-is-base64 = false"
, ""
, "## Enables JWT Cache and sets its max size, disables caching with 0"
, "# jwt-cache-max-entries = 0"
, ""
, "## Logging level, the admitted values are: crit, error, warn, info and debug."
, "log-level = \"error\""
, ""
, "## Log the SQL query at the current log-level."
, "log-query = false"
, ""
, "## Determine if the OpenAPI output should follow or ignore role privileges or be disabled entirely."
, "## Admitted values: follow-privileges, ignore-privileges, disabled"
, "openapi-mode = \"follow-privileges\""
, ""
, "## Base url for the OpenAPI output"
, "openapi-server-proxy-uri = \"\""
, ""
, "## Configurable CORS origins"
, "# server-cors-allowed-origins = \"\""
, ""
, "server-host = \"!4\""
, "server-port = 3000"
, ""
, "## Allow getting the request-response timing information through the `Server-Timing` header"
, "server-timing-enabled = false"
, ""
, "## Unix socket location"
, "## if specified it takes precedence over server-port"
, "# server-unix-socket = \"/tmp/pgrst.sock\""
, ""
, "## Unix socket file mode"
, "## When none is provided, 660 is applied by default"
, "# server-unix-socket-mode = \"660\""
]
+6 -7
View File
@@ -16,7 +16,6 @@ import Control.Arrow ((***))
import PostgREST.Config.PgVersion (PgVersion (..), pgVersion150) import PostgREST.Config.PgVersion (PgVersion (..), pgVersion150)
import qualified Data.HashMap.Strict as HM import qualified Data.HashMap.Strict as HM
import qualified Data.Text as T
import qualified Hasql.Decoders as HD import qualified Hasql.Decoders as HD
import qualified Hasql.Encoders as HE import qualified Hasql.Encoders as HE
@@ -33,8 +32,8 @@ type RoleSettings = (HM.HashMap ByteString (HM.HashMap ByteString ByteString
type RoleIsolationLvl = HM.HashMap ByteString SQL.IsolationLevel type RoleIsolationLvl = HM.HashMap ByteString SQL.IsolationLevel
type TimezoneNames = Set Text -- cache timezone names for prefer timezone= type TimezoneNames = Set Text -- cache timezone names for prefer timezone=
toIsolationLevel :: Text -> SQL.IsolationLevel toIsolationLevel :: (Eq a, IsString a) => a -> SQL.IsolationLevel
toIsolationLevel a = case T.toLower a of toIsolationLevel a = case a of
"repeatable read" -> SQL.RepeatableRead "repeatable read" -> SQL.RepeatableRead
"serializable" -> SQL.Serializable "serializable" -> SQL.Serializable
_ -> SQL.ReadCommitted _ -> SQL.ReadCommitted
@@ -102,7 +101,7 @@ queryDbSettings preConfFunc prepared =
SELECT setdatabase as database, SELECT setdatabase as database,
unnest(setconfig) as setting unnest(setconfig) as setting
FROM pg_catalog.pg_db_role_setting FROM pg_catalog.pg_db_role_setting
WHERE setrole = quote_ident(CURRENT_USER)::regrole::oid WHERE setrole = CURRENT_USER::regrole::oid
AND setdatabase IN (0, (SELECT oid FROM pg_catalog.pg_database WHERE datname = CURRENT_CATALOG)) AND setdatabase IN (0, (SELECT oid FROM pg_catalog.pg_database WHERE datname = CURRENT_CATALOG))
), ),
kv_settings AS ( kv_settings AS (
@@ -143,13 +142,13 @@ queryRoleSettings pgVer prepared =
select r.rolname, unnest(r.rolconfig) as setting select r.rolname, unnest(r.rolconfig) as setting
from pg_auth_members m from pg_auth_members m
join pg_roles r on r.oid = m.roleid join pg_roles r on r.oid = m.roleid
where member = quote_ident(current_user)::regrole::oid where member = current_user::regrole::oid
), ),
kv_settings AS ( kv_settings AS (
SELECT SELECT
rolname, rolname,
substr(setting, 1, strpos(setting, '=') - 1) as key, substr(setting, 1, strpos(setting, '=') - 1) as key,
substr(setting, strpos(setting, '=') + 1) as value lower(substr(setting, strpos(setting, '=') + 1)) as value
FROM role_setting FROM role_setting
), ),
iso_setting AS ( iso_setting AS (
@@ -168,7 +167,7 @@ queryRoleSettings pgVer prepared =
|] |]
hasParameterPrivilege hasParameterPrivilege
| pgVer >= pgVersion150 = "or has_parameter_privilege(quote_ident(current_user)::regrole::oid, ps.name, 'set')" | pgVer >= pgVersion150 = "or has_parameter_privilege(current_user::regrole::oid, ps.name, 'set')"
| otherwise = "" | otherwise = ""
processRows :: [(Text, Maybe Text, [(Text, Text)])] -> (RoleSettings, RoleIsolationLvl) processRows :: [(Text, Maybe Text, [(Text, Text)])] -> (RoleSettings, RoleIsolationLvl)
+5 -1
View File
@@ -3,6 +3,7 @@
module PostgREST.Config.PgVersion module PostgREST.Config.PgVersion
( PgVersion(..) ( PgVersion(..)
, minimumPgVersion , minimumPgVersion
, pgVersion130
, pgVersion140 , pgVersion140
, pgVersion150 , pgVersion150
, pgVersion170 , pgVersion170
@@ -25,7 +26,10 @@ instance Ord PgVersion where
-- | Tells the minimum PostgreSQL version required by this version of PostgREST -- | Tells the minimum PostgreSQL version required by this version of PostgREST
minimumPgVersion :: PgVersion minimumPgVersion :: PgVersion
minimumPgVersion = pgVersion130 minimumPgVersion = pgVersion121
pgVersion121 :: PgVersion
pgVersion121 = PgVersion 120001 "12.1" "12.1"
pgVersion130 :: PgVersion pgVersion130 :: PgVersion
pgVersion130 = PgVersion 130000 "13.0" "13.0" pgVersion130 = PgVersion 130000 "13.0" "13.0"
+12 -24
View File
@@ -3,7 +3,6 @@ Module : PostgREST.Error
Description : PostgREST error HTTP responses Description : PostgREST error HTTP responses
-} -}
{-# OPTIONS_GHC -fno-warn-orphans #-} {-# OPTIONS_GHC -fno-warn-orphans #-}
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE RecordWildCards #-} {-# LANGUAGE RecordWildCards #-}
module PostgREST.Error module PostgREST.Error
@@ -42,7 +41,6 @@ import Network.HTTP.Types.Header (Header)
import PostgREST.MediaType (MediaType (..)) import PostgREST.MediaType (MediaType (..))
import qualified PostgREST.MediaType as MediaType import qualified PostgREST.MediaType as MediaType
import PostgREST.SchemaCache (SchemaCache (SchemaCache, dbTablesFuzzyIndex))
import PostgREST.SchemaCache.Identifiers (QualifiedIdentifier (..), import PostgREST.SchemaCache.Identifiers (QualifiedIdentifier (..),
Schema) Schema)
import PostgREST.SchemaCache.Relationship (Cardinality (..), import PostgREST.SchemaCache.Relationship (Cardinality (..),
@@ -51,8 +49,10 @@ import PostgREST.SchemaCache.Relationship (Cardinality (..),
RelationshipsMap) RelationshipsMap)
import PostgREST.SchemaCache.Routine (Routine (..), import PostgREST.SchemaCache.Routine (Routine (..),
RoutineParam (..)) RoutineParam (..))
import PostgREST.SchemaCache.Table (Table (..))
import Protolude import Protolude
class (ErrorBody a, JSON.ToJSON a) => PgrstError a where class (ErrorBody a, JSON.ToJSON a) => PgrstError a where
status :: a -> HTTP.Status status :: a -> HTTP.Status
headers :: a -> [Header] headers :: a -> [Header]
@@ -250,7 +250,7 @@ data SchemaCacheError
| NoRelBetween Text Text (Maybe Text) Text RelationshipsMap | NoRelBetween Text Text (Maybe Text) Text RelationshipsMap
| NoRpc Text Text [Text] MediaType Bool [QualifiedIdentifier] [Routine] | NoRpc Text Text [Text] MediaType Bool [QualifiedIdentifier] [Routine]
| ColumnNotFound Text Text | ColumnNotFound Text Text
| TableNotFound Text Text SchemaCache | TableNotFound Text Text [Table]
deriving Show deriving Show
instance PgrstError SchemaCacheError where instance PgrstError SchemaCacheError where
@@ -313,7 +313,7 @@ instance ErrorBody SchemaCacheError where
where where
onlySingleParams = isInvPost && contentType `elem` [MTTextPlain, MTTextXML, MTOctetStream] onlySingleParams = isInvPost && contentType `elem` [MTTextPlain, MTTextXML, MTOctetStream]
hint (AmbiguousRpc _) = Just "Try renaming the parameters or the function itself in the database so function overloading can be resolved" hint (AmbiguousRpc _) = Just "Try renaming the parameters or the function itself in the database so function overloading can be resolved"
hint (TableNotFound schemaName relName schemaCache) = JSON.String <$> tableNotFoundHint schemaName relName schemaCache hint (TableNotFound schemaName relName tbls) = JSON.String <$> tableNotFoundHint schemaName relName tbls
hint _ = Nothing hint _ = Nothing
@@ -386,7 +386,7 @@ noRelBetweenHint parent child schema allRels = ("Perhaps you meant '" <>) <$>
-- Just "Perhaps you meant to call the function api.test" -- Just "Perhaps you meant to call the function api.test"
-- --
-- >>> noRpcHint "api" "other" [] procs [] -- >>> noRpcHint "api" "other" [] procs []
-- Nothing -- Just "Perhaps you meant to call the function api.another"
-- --
-- >>> noRpcHint "api" "noclosealternative" [] procs [] -- >>> noRpcHint "api" "noclosealternative" [] procs []
-- Nothing -- Nothing
@@ -423,30 +423,18 @@ noRpcHint schema procName params allProcs overloadedProcs =
-- E.g. ["val", "param", "name"] into "(name, param, val)" -- E.g. ["val", "param", "name"] into "(name, param, val)"
listToText = ("(" <>) . (<> ")") . T.intercalate ", " . sort listToText = ("(" <>) . (<> ")") . T.intercalate ", " . sort
possibleProcs possibleProcs
| null overloadedProcs = getFuzzyHint HintProcedure fuzzySetOfProcs procName | null overloadedProcs = Fuzzy.getOne fuzzySetOfProcs procName
| otherwise = (procName <>) <$> getFuzzyHint HintParams fuzzySetOfParams (listToText params) | otherwise = (procName <>) <$> Fuzzy.getOne fuzzySetOfParams (listToText params)
-- | -- |
-- Do a fuzzy search in all tables in the same schema and return closest result -- Do a fuzzy search in all tables in the same schema and return closest result
tableNotFoundHint :: Text -> Text -> SchemaCache -> Maybe Text tableNotFoundHint :: Text -> Text -> [Table] -> Maybe Text
tableNotFoundHint schema tblName SchemaCache{dbTablesFuzzyIndex} tableNotFoundHint schema tblName tblList
= fmap (\tbl -> "Perhaps you meant the table '" <> schema <> "." <> tbl <> "'") perhapsTable = fmap (\tbl -> "Perhaps you meant the table '" <> schema <> "." <> tbl <> "'") perhapsTable
where where
perhapsTable = (\fuzzySet -> getFuzzyHint HintTable fuzzySet tblName) =<< HM.lookup schema dbTablesFuzzyIndex perhapsTable = Fuzzy.getOne fuzzyTableSet tblName
fuzzyTableSet = Fuzzy.fromList [ tableName tbl | tbl <- tblList, tableSchema tbl == schema]
data HintType
= HintTable
| HintProcedure
| HintParams
-- | Get hint using Fuzzy Search with at least 0.75 similarity score
getFuzzyHint :: HintType -> Fuzzy.FuzzySet -> Text -> Maybe Text
getFuzzyHint hintType =
let minScore = 0.75 :: Double -- used for table and procedure name hints
in case hintType of
HintTable -> Fuzzy.getOneWithMinScore minScore
HintProcedure -> Fuzzy.getOneWithMinScore minScore
HintParams -> Fuzzy.getOne -- For params, we stick to `getOne` which defaults to 0.33 min score, not a security risk to reveal params
compressedRel :: Relationship -> JSON.Value compressedRel :: Relationship -> JSON.Value
-- An ambiguousness error cannot happen for computed relationships TODO refactor so this mempty is not needed -- An ambiguousness error cannot happen for computed relationships TODO refactor so this mempty is not needed
@@ -541,7 +529,7 @@ instance ErrorBody SQL.UsageError where
code (SQL.SessionUsageError (SQL.QueryError _ _ e)) = code e code (SQL.SessionUsageError (SQL.QueryError _ _ e)) = code e
code SQL.AcquisitionTimeoutUsageError = "PGRST003" code SQL.AcquisitionTimeoutUsageError = "PGRST003"
message (SQL.ConnectionUsageError _) = "Database connection error." message (SQL.ConnectionUsageError _) = "Database connection error. Retrying the connection."
message (SQL.SessionUsageError (SQL.QueryError _ _ e)) = message e message (SQL.SessionUsageError (SQL.QueryError _ _ e)) = message e
message SQL.AcquisitionTimeoutUsageError = "Timed out acquiring connection from connection pool." message SQL.AcquisitionTimeoutUsageError = "Timed out acquiring connection from connection pool."
+24 -59
View File
@@ -1,4 +1,3 @@
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE MultiWayIf #-} {-# LANGUAGE MultiWayIf #-}
{-# LANGUAGE RecordWildCards #-} {-# LANGUAGE RecordWildCards #-}
@@ -16,35 +15,24 @@ import PostgREST.Version (prettyVersion)
import qualified PostgREST.AppState as AppState import qualified PostgREST.AppState as AppState
import qualified PostgREST.Config as Config import qualified PostgREST.Config as Config
import Control.Arrow ((&&&)) import Protolude
import Data.Bitraversable (bisequence)
import Data.Either.Combinators (whenRight)
import qualified Data.Text as T
import qualified Database.PostgreSQL.LibPQ as LibPQ
import qualified Hasql.Session as SQL
import PostgREST.Config.Database (queryPgVersion)
import PostgREST.Config.PgVersion (pgvFullName)
import Protolude
-- | Starts the Listener in a thread -- | Starts the Listener in a thread
runListener :: AppState -> IO () runListener :: AppState -> IO ()
runListener appState = do runListener appState = do
AppConfig{..} <- getConfig appState AppConfig{..} <- getConfig appState
when configDbChannelEnabled $ when configDbChannelEnabled $
void . forkIO . void $ retryingListen appState False void . forkIO $ retryingListen appState
-- | Starts a LISTEN connection and handles notifications. It recovers with exponential backoff with a cap of 32 seconds, if the LISTEN connection is lost. -- | Starts a LISTEN connection and handles notifications. It recovers with exponential backoff with a cap of 32 seconds, if the LISTEN connection is lost.
-- | This function never returns (but can throw) and return type enforces that. retryingListen :: AppState -> IO ()
retryingListen :: AppState -> Bool -> IO Void retryingListen appState = do
retryingListen appState hasDbListenerBug = do
AppConfig{..} <- AppState.getConfig appState AppConfig{..} <- AppState.getConfig appState
let let
dbChannel = toS configDbChannel dbChannel = toS configDbChannel
onError err = do handleFinally err = do
AppState.putIsListenerOn appState False AppState.putIsListenerOn appState False
observer $ DBListenFail dbChannel (Right err) observer $ DBListenFail dbChannel (Right err)
when (isDbListenerBug err) $
observer DBListenBugCallQueryFix
unless configDbPoolAutomaticRecovery $ unless configDbPoolAutomaticRecovery $
killThread mainThreadId killThread mainThreadId
@@ -54,43 +42,29 @@ retryingListen appState hasDbListenerBug = do
threadDelay (delay * oneSecondInMicro) threadDelay (delay * oneSecondInMicro)
unless (delay == maxDelay) $ unless (delay == maxDelay) $
AppState.putNextListenerDelay appState (delay * 2) AppState.putNextListenerDelay appState (delay * 2)
-- loop running the listener retryingListen appState
retryingListen appState (isDbListenerBug err)
-- Execute the listener with with error handling -- forkFinally allows to detect if the thread dies
handle onError $ do void . flip forkFinally handleFinally $ do
-- Make sure we don't leak connections on errors dbOrError <- SQL.acquire $ toUtf8 (Config.addTargetSessionAttrs $ Config.addFallbackAppName prettyVersion configDbUri)
bracket case dbOrError of
-- acquire connection Right db -> do
(SQL.acquire $ toUtf8 (Config.addTargetSessionAttrs $ Config.addFallbackAppName prettyVersion configDbUri)) SQL.listen db $ SQL.toPgIdentifier dbChannel
-- release connection AppState.putIsListenerOn appState True
(`whenRight` releaseConnection) $
-- use connection
\case
Right db -> do
(pqHost, pqPort) <- SQL.withLibPQConnection db $ bisequence . (LibPQ.host &&& LibPQ.port)
pgFullName <- SQL.run (queryPgVersion False) db >>= either throwIO (pure . pgvFullName)
when hasDbListenerBug $ SQL.run callNotifQueryUsage db >>= either throwIO pure
SQL.listen db $ SQL.toPgIdentifier dbChannel
AppState.putIsListenerOn appState True delay <- AppState.getNextListenerDelay appState
when (delay > 1) $ do -- if we did a retry
-- assume we lost notifications, refresh the schema cache
AppState.schemaCacheLoader appState
-- reset the delay
AppState.putNextListenerDelay appState 1
delay <- AppState.getNextListenerDelay appState observer $ DBListenStart dbChannel
when (delay > 1) $ do -- if we did a retry SQL.waitForNotifications handleNotification db
-- assume we lost notifications, refresh the schema cache
AppState.schemaCacheLoader appState
-- reset the delay
AppState.putNextListenerDelay appState 1
observer $ DBListenStart pqHost pqPort pgFullName dbChannel Left err -> do
observer $ DBListenFail dbChannel (Left err)
-- wait for notifications exitFailure
-- this will never return, in case of an error it will throw and be caught by onError
forever $ SQL.waitForNotifications handleNotification db
Left err -> do
observer $ DBListenFail dbChannel (Left err)
exitFailure
where where
observer = AppState.getObserver appState observer = AppState.getObserver appState
mainThreadId = AppState.getMainThreadId appState mainThreadId = AppState.getMainThreadId appState
@@ -105,12 +79,3 @@ retryingListen appState hasDbListenerBug = do
cacheReloader = cacheReloader =
AppState.schemaCacheLoader appState AppState.schemaCacheLoader appState
releaseConnection = void . forkIO . handle (observer . DBListenerConnectionCleanupFail) . SQL.release
isDbListenerBug e = "could not access status of transaction" `T.isInfixOf` show e
-- Used to fix a Postgres bug in the listener, see: https://github.com/PostgREST/postgrest/issues/3147#issuecomment-3494591361
-- This query advances the async notification query tail, which solves this issue.
callNotifQueryUsage :: SQL.Session ()
callNotifQueryUsage = SQL.sql "SELECT pg_notification_queue_usage();"
+8 -171
View File
@@ -1,5 +1,3 @@
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE RecordWildCards #-}
{-| {-|
Module : PostgREST.Logger Module : PostgREST.Logger
Description : Logging based on the Observation.hs module. Access logs get sent to stdout and server diagnostic get sent to stderr. Description : Logging based on the Observation.hs module. Access logs get sent to stdout and server diagnostic get sent to stderr.
@@ -12,16 +10,10 @@ module PostgREST.Logger
, LoggerState , LoggerState
) where ) where
import Control.AutoUpdate (defaultUpdateSettings, import Control.AutoUpdate (defaultUpdateSettings,
mkAutoUpdate, mkAutoUpdate, updateAction)
updateAction)
import Control.Debounce import Control.Debounce
import qualified Data.ByteString.Char8 as BS import qualified Data.ByteString.Char8 as BS
import qualified Data.Text.Encoding as T
import qualified Hasql.Decoders as HD
import qualified Hasql.DynamicStatements.Snippet as SQL hiding (sql)
import qualified Hasql.DynamicStatements.Statement as SQL
import qualified Hasql.Statement as SQL
import Data.Time (ZonedTime, defaultTimeLocale, formatTime, import Data.Time (ZonedTime, defaultTimeLocale, formatTime,
getZonedTime) getZonedTime)
@@ -34,16 +26,8 @@ import System.IO.Unsafe (unsafePerformIO)
import PostgREST.Config (LogLevel (..)) import PostgREST.Config (LogLevel (..))
import PostgREST.Observation import PostgREST.Observation
import PostgREST.Query (MainQuery (..))
import qualified Data.ByteString.Lazy as LBS import Protolude
import qualified Data.Text as T
import qualified Hasql.Connection as SQL
import qualified Hasql.Pool.Observation as SQL
import Numeric (showFFloat)
import PostgREST.Config.PgVersion (pgvName)
import qualified PostgREST.Error as Error
import Protolude
data LoggerState = LoggerState data LoggerState = LoggerState
{ stateGetZTime :: IO ZonedTime -- ^ Time with time zone used for logs { stateGetZTime :: IO ZonedTime -- ^ Time with time zone used for logs
@@ -103,33 +87,19 @@ observationLogger loggerState logLevel obs = case obs of
o@(QueryErrorCodeHighObs _) -> do o@(QueryErrorCodeHighObs _) -> do
when (logLevel >= LogError) $ do when (logLevel >= LogError) $ do
logWithZTime loggerState $ observationMessage o logWithZTime loggerState $ observationMessage o
o@SchemaCacheEmptyObs ->
when (logLevel >= LogError) $ do
logWithZTime loggerState $ observationMessage o
o@(HasqlPoolObs _) -> do o@(HasqlPoolObs _) -> do
when (logLevel >= LogDebug) $ do when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o logWithZTime loggerState $ observationMessage o
QueryObs gq status -> do o@(DBQuery sql status) -> do
when (shouldLogResponse logLevel status) $ -- Does not log SQL when it's empty (for OPTIONS requests or for the default OpenAPI output)
logMainQ loggerState gq when (sql /= mempty && shouldLogResponse logLevel status) $ do
logWithZTime loggerState $ observationMessage o
o@PoolRequest -> o@PoolRequest ->
when (logLevel >= LogDebug) $ do when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o logWithZTime loggerState $ observationMessage o
o@PoolRequestFullfilled -> o@PoolRequestFullfilled ->
when (logLevel >= LogDebug) $ do when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o logWithZTime loggerState $ observationMessage o
o@PoolFlushed ->
when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o
o@JwtCacheEviction ->
when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o
o@(JwtCacheLookup _) ->
when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o
o@(WarpServerObs _) ->
when (logLevel >= LogDebug) $ do
logWithZTime loggerState $ observationMessage o
o -> o ->
logWithZTime loggerState $ observationMessage o logWithZTime loggerState $ observationMessage o
@@ -137,136 +107,3 @@ logWithZTime :: LoggerState -> Text -> IO ()
logWithZTime loggerState txt = do logWithZTime loggerState txt = do
zTime <- stateGetZTime loggerState zTime <- stateGetZTime loggerState
hPutStrLn stderr $ toS (formatTime defaultTimeLocale "%d/%b/%Y:%T %z: " zTime) <> txt hPutStrLn stderr $ toS (formatTime defaultTimeLocale "%d/%b/%Y:%T %z: " zTime) <> txt
logMainQ :: LoggerState -> MainQuery -> IO ()
logMainQ loggerState MainQuery{mqOpenAPI=(x, y, z),..} =
let snipts = renderSnippet <$> [mqTxVars, fromMaybe mempty mqPreReq, mqMain, x, y, z, fromMaybe mempty mqExplain]
-- Does not log SQL when it's empty (happens on OPTIONS requests and when the openapi queries are not generated)
logQ q = when (q /= mempty) $ logWithZTime loggerState $ showOnSingleLine '\n' $ T.decodeUtf8 q in
mapM_ logQ snipts
-- TODO: maybe patch upstream hasql-dynamic-statements so we have a less hackish way to convert
-- the SQL.Snippet or maybe don't use hasql-dynamic-statements and resort to plain strings for the queries and use regular hasql
renderSnippet :: SQL.Snippet -> ByteString
renderSnippet snippet =
let SQL.Statement sql _ _ _ = SQL.dynamicallyParameterized snippet decoder prepared
decoder = HD.noResult -- unused
prepared = False -- unused
in
sql
observationMessage :: Observation -> Text
observationMessage = \case
AdminStartObs address ->
"Admin server listening on " <> address
AdminServerCrashedObs ex ->
"Admin server crashed unexpectedly: " <> (showOnSingleLine '\t' . show) ex
AppStartObs ver ->
"Starting PostgREST " <> T.decodeUtf8 ver <> "..."
AppServerAddressObs address ->
"API server listening on " <> address
DBConnectedObs ver ->
"Successfully connected to " <> ver
ExitUnsupportedPgVersion pgVer minPgVer ->
"Cannot run in this PostgreSQL version (" <> pgvName pgVer <> "), PostgREST needs at least " <> pgvName minPgVer
ExitDBNoRecoveryObs ->
"Automatic recovery disabled, exiting."
ExitDBFatalError ServerAuthError usageErr ->
"Failed to establish a connection. " <> jsonMessage usageErr
ExitDBFatalError ServerPgrstBug usageErr ->
"This is probably a bug in PostgREST, please report it at https://github.com/PostgREST/postgrest/issues. " <> jsonMessage usageErr
ExitDBFatalError ServerError42P05 usageErr ->
"If you are using connection poolers in transaction mode, try setting db-prepared-statements to false. " <> jsonMessage usageErr
ExitDBFatalError ServerError08P01 usageErr ->
"Connection poolers in statement mode are not supported." <> jsonMessage usageErr
SchemaCacheEmptyObs ->
T.decodeUtf8 . LBS.toStrict . Error.errorPayload $ Error.NoSchemaCacheError
SchemaCacheErrorObs dbSchemas extraPaths usageErr ->
"Failed to load the schema cache using "
<> "db-schemas=" <> T.intercalate "," (toList dbSchemas)
<> " and "
<> "db-extra-search-path=" <> T.intercalate "," extraPaths <> ". " <> jsonMessage usageErr
SchemaCacheQueriedObs resultTime ->
"Schema cache queried in " <> showMillis resultTime <> " milliseconds"
SchemaCacheSummaryObs summary ->
"Schema cache loaded " <> summary
SchemaCacheLoadedObs resultTime ->
"Schema cache loaded in " <> showMillis resultTime <> " milliseconds"
ConnectionRetryObs delay ->
"Attempting to reconnect to the database in " <> (show delay::Text) <> " seconds..."
QueryPgVersionError usageErr ->
"Failed to query the PostgreSQL version. " <> jsonMessage usageErr
DBListenStart host port fullName channel -> do
"Listener connected to " <> fullName <> " on " <> show (fold $ host <> fmap (":" <>) port) <> " and listening for database notifications on the " <> show channel <> " channel"
DBListenFail channel listenErr ->
"Failed listening for database notifications on the " <> show channel <> " channel. " <>
either showListenerConnError showListenerException listenErr
DBListenRetry delay ->
"Retrying listening for database notifications in " <> (show delay::Text) <> " seconds..."
DBListenBugCallQueryFix ->
"This is likely a PostgreSQL bug in the notification queue, executing the following to try to solve it: SELECT pg_notification_queue_usage();"
DBListenerGotSCacheMsg channel ->
"Received a schema cache reload message on the " <> show channel <> " channel"
DBListenerGotConfigMsg channel ->
"Received a config reload message on the " <> show channel <> " channel"
DBListenerConnectionCleanupFail ex ->
"Failed during listener connection cleanup: " <> showOnSingleLine '\t' (show ex)
QueryObs{} ->
mempty -- TODO pending refactor: The logic for printing the query cannot be done here. Join the observationMessage function into observationLogger to avoid this mempty.
ConfigReadErrorObs usageErr ->
"Failed to query database settings for the config parameters." <> jsonMessage usageErr
QueryRoleSettingsErrorObs usageErr ->
"Failed to query the role settings. " <> jsonMessage usageErr
QueryErrorCodeHighObs usageErr ->
jsonMessage usageErr
ConfigInvalidObs err ->
"Failed reloading config: " <> err
ConfigSucceededObs ->
"Config reloaded"
PoolInit poolSize ->
"Connection Pool initialized with a maximum size of " <> show poolSize <> " connections"
PoolAcqTimeoutObs usageErr ->
jsonMessage usageErr
HasqlPoolObs (SQL.ConnectionObservation uuid status) ->
"Connection " <> show uuid <> (
case status of
SQL.ConnectingConnectionStatus -> " is being established"
SQL.ReadyForUseConnectionStatus -> " is available"
SQL.InUseConnectionStatus -> " is used"
SQL.TerminatedConnectionStatus reason -> " is terminated due to " <> case reason of
SQL.AgingConnectionTerminationReason -> "max lifetime"
SQL.IdlenessConnectionTerminationReason -> "max idletime"
SQL.ReleaseConnectionTerminationReason -> "release"
SQL.NetworkErrorConnectionTerminationReason _ -> "network error" -- usage error is already logged, no need to repeat the same message.
)
PoolRequest ->
"Trying to borrow a connection from pool"
PoolRequestFullfilled ->
"Borrowed a connection from the pool"
PoolFlushed ->
"Database connection pool flushed"
JwtCacheLookup _ ->
"Looked up a JWT in JWT cache"
JwtCacheEviction ->
"Evicted entry from JWT cache"
TerminationUnixSignalObs signal ->
"Received termination unix signal " <> signal
WarpServerObs txt ->
"Warp server: " <> txt
where
showMillis :: Double -> Text
showMillis x = toS $ showFFloat (Just 1) x ""
jsonMessage err = T.decodeUtf8 . LBS.toStrict . Error.errorPayload $ Error.PgError False err
showListenerConnError :: SQL.ConnectionError -> Text
showListenerConnError = maybe "Connection error" (showOnSingleLine '\t' . T.decodeUtf8)
showListenerException :: SomeException -> Text
showListenerException = showOnSingleLine '\t' . show
showOnSingleLine :: Char -> Text -> Text
showOnSingleLine split txt = T.intercalate " " $ T.filter (/= split) <$> T.lines txt -- the errors from hasql-notifications come intercalated with "\t\n"
-279
View File
@@ -1,279 +0,0 @@
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE RecordWildCards #-}
{-|
Module : PostgREST.MainTx
Description : PostgREST transaction executor
This module parametrizes, prepares, executes SQL queries and decodes their results.
-}
module PostgREST.MainTx
( MainTx (..)
, DbResult (..)
, ResultSet (..)
, mainTx
) where
import Control.Lens ((^?))
import Control.Monad.Extra (whenJust)
import qualified Data.Aeson.Lens as L
import qualified Data.ByteString as BS hiding
(break)
import qualified Data.ByteString.Char8 as BS
import qualified Data.HashMap.Strict as HM
import qualified Data.Set as S
import qualified Hasql.Decoders as HD
import qualified Hasql.DynamicStatements.Statement as SQL
import qualified Hasql.Session as SQL (Session)
import qualified Hasql.Transaction as SQL
import qualified Hasql.Transaction.Sessions as SQL
import qualified PostgREST.Error as Error
import qualified PostgREST.SchemaCache as SchemaCache
import PostgREST.ApiRequest (ApiRequest (..))
import PostgREST.ApiRequest.Preferences (PreferCount (..),
PreferHandling (..),
PreferMaxAffected (..),
PreferTransaction (..),
Preferences (..))
import PostgREST.ApiRequest.Types (Mutation (..))
import PostgREST.Auth.Types (AuthResult (..))
import PostgREST.Config (AppConfig (..),
OpenAPIMode (..))
import PostgREST.Error (Error)
import PostgREST.MediaType (MediaType (..))
import PostgREST.Plan (ActionPlan (..),
CrudPlan (..),
DbActionPlan (..),
InfoPlan (..),
InspectPlan (..))
import PostgREST.Query (MainQuery (..))
import PostgREST.SchemaCache (SchemaCache (..))
import PostgREST.SchemaCache.Identifiers (QualifiedIdentifier (..))
import PostgREST.SchemaCache.Routine (Routine (..), RoutineMap)
import PostgREST.SchemaCache.Table (TablesMap)
import Protolude hiding (Handler)
type DbHandler = ExceptT Error SQL.Transaction
data MainTx
= DbTx {
dqIsoLevel :: SQL.IsolationLevel
, dqTxMode :: SQL.Mode
, dqDbHandler :: DbHandler DbResult
, dqTransaction :: SQL.IsolationLevel -> SQL.Mode -> SQL.Transaction (Either Error DbResult) -> SQL.Session (Either Error DbResult)
}
| NoDbTx DbResult
data DbResult
= DbCrudResult CrudPlan ResultSet
| DbPlanResult MediaType BS.ByteString
| MaybeDbResult InspectPlan (Maybe (TablesMap, RoutineMap, Maybe Text))
| NoDbResult InfoPlan
-- | Standard result set format used for the mqMain query
data ResultSet
= RSStandard
{ rsTableTotal :: Maybe Int64
-- ^ count of all the table rows
, rsQueryTotal :: Int64
-- ^ count of the query rows
, rsLocation :: [(BS.ByteString, BS.ByteString)]
-- ^ The Location header(only used for inserts) is represented as a list of strings containing
-- variable bindings like @"k1=eq.42"@, or the empty list if there is no location header.
, rsBody :: BS.ByteString
-- ^ the aggregated body of the query
, rsGucHeaders :: Maybe BS.ByteString
-- ^ the HTTP headers to be added to the response
, rsGucStatus :: Maybe Text
-- ^ the HTTP status to be added to the response
, rsInserted :: Maybe Int64
-- ^ the number of rows inserted (Only used for upserts)
}
mainTx :: MainQuery -> AppConfig -> AuthResult -> ApiRequest -> ActionPlan -> SchemaCache -> MainTx
mainTx _ _ _ _ (NoDb x) _ = NoDbTx $ NoDbResult x
mainTx genQ@MainQuery{..} conf@AppConfig{..} AuthResult{..} apiReq (Db plan) sCache =
DbTx isoLvl txMode dbHandler transaction
where
transaction = if configDbPreparedStatements then SQL.transaction else SQL.unpreparedTransaction
isoLvl = planIsoLvl conf authRole plan
txMode = planTxMode plan
dbHandler = do
lift $ SQL.statement mempty $ SQL.dynamicallyParameterized mqTxVars
HD.noResult configDbPreparedStatements
lift $ whenJust mqPreReq $ \q ->
SQL.statement mempty $ SQL.dynamicallyParameterized q
HD.noResult configDbPreparedStatements
actionResult genQ plan conf apiReq sCache
planTxMode :: DbActionPlan -> SQL.Mode
planTxMode (DbCrud _ x) = pTxMode x
planTxMode (MayUseDb x) = ipTxmode x
planIsoLvl :: AppConfig -> ByteString -> DbActionPlan -> SQL.IsolationLevel
planIsoLvl AppConfig{configRoleIsoLvl} role actPlan = case actPlan of
DbCrud _ CallReadPlan{crProc} -> fromMaybe roleIsoLvl $ pdIsoLvl crProc
_ -> roleIsoLvl
where
roleIsoLvl = HM.findWithDefault SQL.ReadCommitted role configRoleIsoLvl
actionResult :: MainQuery -> DbActionPlan -> AppConfig -> ApiRequest -> SchemaCache -> ExceptT Error SQL.Transaction DbResult
actionResult MainQuery{..} (DbCrud True plan) conf@AppConfig{..} apiReq _ = do
explRes <- lift $ SQL.statement mempty $ SQL.dynamicallyParameterized mqMain planRow configDbPreparedStatements
optionalRollback conf apiReq
pure $ DbPlanResult (pMedia plan) explRes
actionResult MainQuery{..} (DbCrud _ plan@WrappedReadPlan{..}) conf@AppConfig{..} apiReq@ApiRequest{iPreferences=Preferences{..}} _ = do
resultSet@RSStandard{rsTableTotal=tableTotal} <- lift $ SQL.statement mempty $ dynStmt (HD.singleRow $ standardRow True)
failNotSingular pMedia resultSet
optionalRollback conf apiReq
explainTotal <- lift . fmap join $ traverse (\snip ->
SQL.statement mempty $ SQL.dynamicallyParameterized snip decodeExplain configDbPreparedStatements)
mqExplain
pure $ DbCrudResult plan
resultSet{rsTableTotal=case preferCount of
Just PlannedCount -> explainTotal
Just EstimatedCount -> if tableTotal > (fromIntegral <$> configDbMaxRows)
then max <$> tableTotal <*> explainTotal
else tableTotal
_ -> tableTotal}
where
dynStmt decod = SQL.dynamicallyParameterized mqMain decod configDbPreparedStatements
decodeExplain :: HD.Result (Maybe Int64)
decodeExplain =
let row = HD.singleRow $ column HD.bytea in
(^? L.nth 0 . L.key "Plan" . L.key "Plan Rows" . L._Integral) <$> row
actionResult MainQuery{..} (DbCrud _ plan@MutateReadPlan{..}) conf@AppConfig{..} apiReq@ApiRequest{iPreferences=Preferences{..}} _ = do
resultSet <- lift $ SQL.statement mempty $ dynStmt decodeRow
failMutation resultSet
optionalRollback conf apiReq
pure $ DbCrudResult plan resultSet
where
dynStmt decod = SQL.dynamicallyParameterized mqMain decod configDbPreparedStatements
failMutation resultSet = case mrMutation of
MutationCreate -> do
failNotSingular pMedia resultSet
MutationUpdate -> do
failNotSingular pMedia resultSet
failExceedsMaxAffectedPref (preferMaxAffected,preferHandling) resultSet
MutationSingleUpsert -> do
failPut resultSet
MutationDelete -> do
failNotSingular pMedia resultSet
failExceedsMaxAffectedPref (preferMaxAffected,preferHandling) resultSet
decodeRow = fromMaybe (RSStandard Nothing 0 mempty mempty Nothing Nothing Nothing) <$> HD.rowMaybe (standardRow False)
actionResult MainQuery{..} (DbCrud _ plan@CallReadPlan{..}) conf@AppConfig{..} apiReq@ApiRequest{iPreferences=Preferences{..}} _ = do
resultSet <- lift $ SQL.statement mempty $ dynStmt decodeRow
optionalRollback conf apiReq
failNotSingular pMedia resultSet
failExceedsMaxAffectedPref (preferMaxAffected,preferHandling) resultSet
pure $ DbCrudResult plan resultSet
where
dynStmt decod = SQL.dynamicallyParameterized mqMain decod configDbPreparedStatements
decodeRow = fromMaybe (RSStandard (Just 0) 0 mempty mempty Nothing Nothing Nothing) <$> HD.rowMaybe (standardRow True)
actionResult MainQuery{mqOpenAPI=(tblsQ, funcsQ, schQ)} (MayUseDb plan@InspectPlan{ipSchema=tSchema}) AppConfig{..} _ sCache =
mainActionQuery
where
mainActionQuery = lift $
case configOpenApiMode of
OAFollowPriv -> do
tableAccess <- SQL.statement mempty $ SQL.dynamicallyParameterized tblsQ decodeAccessibleIdentifiers configDbPreparedStatements
accFuncs <- SQL.statement mempty $ SQL.dynamicallyParameterized funcsQ SchemaCache.decodeFuncs configDbPreparedStatements
schDesc <- SQL.statement mempty $ SQL.dynamicallyParameterized schQ decodeSchemaDesc configDbPreparedStatements
let tbls = HM.filterWithKey (\qi _ -> S.member qi tableAccess) $ SchemaCache.dbTables sCache
pure $ MaybeDbResult plan (Just (tbls, accFuncs, schDesc))
OAIgnorePriv -> do
schDesc <- SQL.statement mempty (SQL.dynamicallyParameterized schQ decodeSchemaDesc configDbPreparedStatements)
let tbls = HM.filterWithKey (\(QualifiedIdentifier sch _) _ -> sch == tSchema) (SchemaCache.dbTables sCache)
routs = HM.filterWithKey (\(QualifiedIdentifier sch _) _ -> sch == tSchema) (SchemaCache.dbRoutines sCache)
pure $ MaybeDbResult plan (Just (tbls, routs, schDesc))
OADisabled ->
pure $ MaybeDbResult plan Nothing
decodeSchemaDesc :: HD.Result (Maybe Text)
decodeSchemaDesc = join <$> HD.rowMaybe (nullableColumn HD.text)
decodeAccessibleIdentifiers :: HD.Result (S.Set QualifiedIdentifier)
decodeAccessibleIdentifiers =
let
row = QualifiedIdentifier
<$> column HD.text
<*> column HD.text
in
S.fromList <$> HD.rowList row
-- Makes sure the querystring pk matches the payload pk
-- e.g. PUT /items?id=eq.1 { "id" : 1, .. } is accepted,
-- PUT /items?id=eq.14 { "id" : 2, .. } is rejected.
-- If this condition is not satisfied then nothing is inserted,
-- check the WHERE for INSERT in QueryBuilder.hs to see how it's done
failPut :: ResultSet -> DbHandler ()
failPut RSStandard{rsQueryTotal=queryTotal} =
when (queryTotal /= 1) $ do
lift SQL.condemn
throwError $ Error.ApiRequestError Error.PutMatchingPkError
-- |
-- Fail a response if a single JSON object was requested and not exactly one
-- was found.
failNotSingular :: MediaType -> ResultSet -> DbHandler ()
failNotSingular mediaType RSStandard{rsQueryTotal=queryTotal} =
when (elem mediaType [MTVndSingularJSON True, MTVndSingularJSON False] && queryTotal /= 1) $ do
lift SQL.condemn
throwError $ Error.ApiRequestError . Error.SingularityError $ toInteger queryTotal
failExceedsMaxAffectedPref :: (Maybe PreferMaxAffected, Maybe PreferHandling) -> ResultSet -> DbHandler ()
failExceedsMaxAffectedPref (Nothing,_) _ = pure ()
failExceedsMaxAffectedPref (Just (PreferMaxAffected n), handling) RSStandard{rsQueryTotal=queryTotal} = when ((queryTotal > n) && (handling == Just Strict)) $ do
lift SQL.condemn
throwError $ Error.ApiRequestError . Error.MaxAffectedViolationError $ toInteger queryTotal
-- | Set a transaction to roll back if requested
optionalRollback :: AppConfig -> ApiRequest -> DbHandler ()
optionalRollback AppConfig{..} ApiRequest{iPreferences=Preferences{..}} = do
lift $ when (shouldRollback || (configDbTxRollbackAll && not shouldCommit)) $ do
SQL.sql "SET CONSTRAINTS ALL IMMEDIATE"
SQL.condemn
where
shouldCommit =
preferTransaction == Just Commit
shouldRollback =
preferTransaction == Just Rollback
-- | We use rowList because when doing EXPLAIN (FORMAT TEXT), the result comes as many rows. FORMAT JSON comes as one.
planRow :: HD.Result BS.ByteString
planRow = BS.unlines <$> HD.rowList (column HD.bytea)
column :: HD.Value a -> HD.Row a
column = HD.column . HD.nonNullable
nullableColumn :: HD.Value a -> HD.Row (Maybe a)
nullableColumn = HD.column . HD.nullable
arrayColumn :: HD.Value a -> HD.Row [a]
arrayColumn = column . HD.listArray . HD.nonNullable
standardRow :: Bool -> HD.Row ResultSet
standardRow noLocation =
RSStandard <$> nullableColumn HD.int8 <*> column HD.int8
<*> (if noLocation then pure mempty else fmap splitKeyValue <$> arrayColumn HD.bytea)
<*> (fromMaybe mempty <$> nullableColumn HD.bytea)
<*> nullableColumn HD.bytea
<*> nullableColumn HD.text
<*> nullableColumn HD.int8
where
splitKeyValue :: ByteString -> (ByteString, ByteString)
splitKeyValue kv =
let (k, v) = BS.break (== '=') kv in
(k, BS.tail v)
+19 -79
View File
@@ -1,14 +1,10 @@
{-# LANGUAGE RecordWildCards #-}
{-| {-|
Module : PostgREST.Logger Module : PostgREST.Logger
Description : Metrics based on the Observation module. See Observation.hs. Description : Metrics based on the Observation module. See Observation.hs.
-} -}
module PostgREST.Metrics module PostgREST.Metrics
( init ( init
, ConnTrack
, ConnStats (..)
, MetricsState (..) , MetricsState (..)
, connectionCounts
, observationMetrics , observationMetrics
, metricsToText , metricsToText
) where ) where
@@ -20,63 +16,35 @@ import Prometheus
import PostgREST.Observation import PostgREST.Observation
import Control.Arrow ((&&&)) import Protolude
import Data.Bitraversable (bisequenceA)
import Data.Tuple.Extra (both)
import Data.UUID (UUID)
import qualified Focus
import Protolude
import qualified StmHamt.SizedHamt as SH
data MetricsState = data MetricsState =
MetricsState { MetricsState Counter Gauge Gauge Gauge (Vector Label1 Counter) Gauge
poolTimeouts :: Counter,
connTrack :: ConnTrack,
poolWaiting :: Gauge,
poolMaxSize :: Gauge,
schemaCacheLoads :: Vector Label1 Counter,
schemaCacheQueryTime :: Gauge,
jwtCacheRequests :: Counter,
jwtCacheHits :: Counter,
jwtCacheEvictions :: Counter
}
init :: Int -> IO MetricsState init :: Int -> IO MetricsState
init configDbPoolSize = do init configDbPoolSize = do
metricState <- MetricsState <$> poolTimeouts <- register $ counter (Info "pgrst_db_pool_timeouts_total" "The total number of pool connection timeouts")
register (counter (Info "pgrst_db_pool_timeouts_total" "The total number of pool connection timeouts")) <*> poolAvailable <- register $ gauge (Info "pgrst_db_pool_available" "Available connections in the pool")
register (Metric ((identity &&& dbPoolAvailable) <$> connectionTracker)) <*> poolWaiting <- register $ gauge (Info "pgrst_db_pool_waiting" "Requests waiting to acquire a pool connection")
register (gauge (Info "pgrst_db_pool_waiting" "Requests waiting to acquire a pool connection")) <*> poolMaxSize <- register $ gauge (Info "pgrst_db_pool_max" "Max pool connections")
register (gauge (Info "pgrst_db_pool_max" "Max pool connections")) <*> schemaCacheLoads <- register $ vector "status" $ counter (Info "pgrst_schema_cache_loads_total" "The total number of times the schema cache was loaded")
register (vector "status" $ counter (Info "pgrst_schema_cache_loads_total" "The total number of times the schema cache was loaded")) <*> schemaCacheQueryTime <- register $ gauge (Info "pgrst_schema_cache_query_time_seconds" "The query time in seconds of the last schema cache load")
register (gauge (Info "pgrst_schema_cache_query_time_seconds" "The query time in seconds of the last schema cache load")) <*> setGauge poolMaxSize (fromIntegral configDbPoolSize)
register (counter (Info "pgrst_jwt_cache_requests_total" "The total number of JWT cache lookups")) <*> pure $ MetricsState poolTimeouts poolAvailable poolWaiting poolMaxSize schemaCacheLoads schemaCacheQueryTime
register (counter (Info "pgrst_jwt_cache_hits_total" "The total number of JWT cache hits")) <*>
register (counter (Info "pgrst_jwt_cache_evictions_total" "The total number of JWT cache evictions"))
setGauge (poolMaxSize metricState) (fromIntegral configDbPoolSize)
pure metricState
where
dbPoolAvailable = (pure . noLabelsGroup (Info "pgrst_db_pool_available" "Available connections in the pool") GaugeType . calcAvailable <$>) . connectionCounts
where
calcAvailable = liftA2 (-) connected inUse
toSample name labels = Sample name labels . encodeUtf8 . show
noLabelsGroup info sampleType = SampleGroup info sampleType . pure . toSample (metricName info) mempty
-- Only some observations are used as metrics -- Only some observations are used as metrics
observationMetrics :: MetricsState -> ObservationHandler observationMetrics :: MetricsState -> ObservationHandler
observationMetrics MetricsState{..} obs = case obs of observationMetrics (MetricsState poolTimeouts poolAvailable poolWaiting _ schemaCacheLoads schemaCacheQueryTime) obs = case obs of
(PoolAcqTimeoutObs _) -> do (PoolAcqTimeoutObs _) -> do
incCounter poolTimeouts incCounter poolTimeouts
-- Handle pool observations with connection tracking (HasqlPoolObs (SQL.ConnectionObservation _ status)) -> case status of
-- this is necessary because it is not possible SQL.ReadyForUseConnectionStatus -> do
-- to accurately maintain open/in use conneciton counts incGauge poolAvailable
-- statelessly based only on pool observation events. SQL.InUseConnectionStatus -> do
-- The reason is that hasql-pool emits TerminatedConnectionStatus decGauge poolAvailable
-- both for connections successfully established and failed when connecting. SQL.TerminatedConnectionStatus _ -> do
-- When receiving TerminatedConnectionStatus we have to find out decGauge poolAvailable
-- if we can decrement established connection count. To do that we have to track SQL.ConnectingConnectionStatus -> pure ()
-- established connections.
(HasqlPoolObs sqlObs) -> trackConnections connTrack sqlObs
PoolRequest -> PoolRequest ->
incGauge poolWaiting incGauge poolWaiting
PoolRequestFullfilled -> PoolRequestFullfilled ->
@@ -86,36 +54,8 @@ observationMetrics MetricsState{..} obs = case obs of
setGauge schemaCacheQueryTime resTime setGauge schemaCacheQueryTime resTime
SchemaCacheErrorObs{} -> do SchemaCacheErrorObs{} -> do
withLabel schemaCacheLoads "FAIL" incCounter withLabel schemaCacheLoads "FAIL" incCounter
JwtCacheLookup True -> incCounter jwtCacheRequests *> incCounter jwtCacheHits
JwtCacheLookup False -> incCounter jwtCacheRequests
JwtCacheEviction -> incCounter jwtCacheEvictions
_ -> _ ->
pure () pure ()
metricsToText :: IO LBS.ByteString metricsToText :: IO LBS.ByteString
metricsToText = exportMetricsAsText metricsToText = exportMetricsAsText
data ConnStats = ConnStats {
connected :: Int,
inUse :: Int
} deriving (Eq, Show)
data ConnTrack = ConnTrack { connTrackConnected :: SH.SizedHamt UUID, connTrackInUse :: SH.SizedHamt UUID }
connectionTracker :: IO ConnTrack
connectionTracker = ConnTrack <$> SH.newIO <*> SH.newIO
trackConnections :: ConnTrack -> SQL.Observation -> IO ()
trackConnections ConnTrack{..} (SQL.ConnectionObservation uuid status) = case status of
SQL.ReadyForUseConnectionStatus -> atomically $
SH.insert identity uuid connTrackConnected *>
SH.focus Focus.delete identity uuid connTrackInUse
SQL.TerminatedConnectionStatus _ -> atomically $
SH.focus Focus.delete identity uuid connTrackConnected *>
SH.focus Focus.delete identity uuid connTrackInUse
SQL.InUseConnectionStatus -> atomically $
SH.insert identity uuid connTrackInUse
_ -> mempty
connectionCounts :: ConnTrack -> IO ConnStats
connectionCounts = atomically . fmap (uncurry ConnStats) . bisequenceA . both SH.size . (connTrackConnected &&& connTrackInUse)
+12 -52
View File
@@ -1,61 +1,21 @@
module PostgREST.Network module PostgREST.Network
( resolveSocketToAddress ( resolveHost
, escapeHostName
, isSpecialHostName
) where ) where
import Data.IP (fromHostAddress, fromHostAddress6)
import Data.String (IsString (..)) import Data.String (IsString (..))
import qualified Network.Socket as NS import qualified Network.Socket as NS
import Protolude import Protolude
-- | Resolves the socket to an address depending on the socket type. The Show resolveHost :: NS.Socket -> IO (Maybe Text)
-- instance of the socket types automatically resolves it to the correct resolveHost sock = do
-- address. Example resolution:
-- -----------------------------------------------------
-- | IPv4 | IPv6 | Unix |
-- -----------------------------------------------------
-- | 127.0.0.1:80 | [2001:db8::1]:80 | /tmp/pgrst.sock |
-- -----------------------------------------------------
resolveSocketToAddress :: NS.Socket -> IO Text
resolveSocketToAddress sock = do
sn <- NS.getSocketName sock sn <- NS.getSocketName sock
return $ showSocketAddr sn case sn of
NS.SockAddrInet _ hostAddr -> pure $ Just $ fromString $ show $ fromHostAddress hostAddr
-- | -- The IPv6 addresses are wrapped in [] brackets. This is done in accordance
-- >>> let addr_ipv4 = NS.SockAddrInet 80 (NS.tupleToHostAddress (127,0,0,1)) -- to RFC 3986 (https://datatracker.ietf.org/doc/html/rfc3986#section-3.2.2).
-- >>> let addr_ipv6 = NS.SockAddrInet6 80 0 (0,0,0,1) 0 -- In short, we did this to have a clear separation between the port and host
-- >>> let addr_unix = NS.SockAddrUnix "/tmp/pgrst.sock" -- because the components of an IPv6 are separated with the ':' character.
-- NS.SockAddrInet6 _ _ hostAddr6 _ -> pure $ Just $ fromString $ "[" ++ show (fromHostAddress6 hostAddr6) ++ "]"
-- >>> showSocketAddr addr_ipv4 _ -> pure Nothing
-- "127.0.0.1:80"
-- >>> showSocketAddr addr_ipv6
-- "[::1]:80"
--
-- >>> showSocketAddr addr_unix
-- "/tmp/pgrst.sock"
showSocketAddr :: NS.SockAddr -> Text
showSocketAddr = fromString . show
-- | When printing special addresses like !4 or *6, we use the following mapping.
-- These special addresses come from:
-- https://hackage.haskell.org/package/streaming-commons-0.2.3.0/docs/\
-- Data-Streaming-Network.html#t:HostPreference
-- TODO: "!6" should not be printed as "0.0.0.0" address.
escapeHostName :: Text -> Text
escapeHostName "*" = "0.0.0.0"
escapeHostName "*4" = "0.0.0.0"
escapeHostName "!4" = "0.0.0.0"
escapeHostName "*6" = "0.0.0.0"
escapeHostName "!6" = "0.0.0.0"
escapeHostName h = h
-- | Check if a hostname is special
isSpecialHostName :: Text -> Bool
isSpecialHostName "*" = True
isSpecialHostName "*4" = True
isSpecialHostName "!4" = True
isSpecialHostName "*6" = True
isSpecialHostName "!6" = True
isSpecialHostName _ = False
+115 -18
View File
@@ -1,4 +1,4 @@
{-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE LambdaCase #-}
{-| {-|
Module : PostgREST.Observation Module : PostgREST.Observation
Description : This module holds an Observation type which is the core of Observability for PostgREST. Description : This module holds an Observation type which is the core of Observability for PostgREST.
@@ -9,41 +9,46 @@ Description : This module holds an Observation type which is the core of Observa
module PostgREST.Observation module PostgREST.Observation
( Observation(..) ( Observation(..)
, ObsFatalError(..) , ObsFatalError(..)
, observationMessage
, ObservationHandler , ObservationHandler
) where ) where
import qualified Data.ByteString.Lazy as LBS
import Data.List.NonEmpty (toList)
import qualified Data.Text as T
import qualified Data.Text.Encoding as T
import qualified Hasql.Connection as SQL import qualified Hasql.Connection as SQL
import qualified Hasql.Pool as SQL import qualified Hasql.Pool as SQL
import qualified Hasql.Pool.Observation as SQL import qualified Hasql.Pool.Observation as SQL
import Network.HTTP.Types.Status (Status) import Network.HTTP.Types.Status (Status)
import qualified Network.Socket as NS
import Numeric (showFFloat)
import PostgREST.Config.PgVersion import PostgREST.Config.PgVersion
import PostgREST.Query (MainQuery) import qualified PostgREST.Error as Error
import Protolude hiding (toList) import Protolude hiding (toList)
import Protolude.Partial (fromJust)
data Observation data Observation
= AdminStartObs Text = AdminStartObs (Maybe Text) (Maybe Int)
| AdminServerCrashedObs SomeException
| AppStartObs ByteString | AppStartObs ByteString
| AppServerAddressObs Text | AppServerPortObs Text NS.PortNumber
| AppServerUnixObs FilePath
| ExitUnsupportedPgVersion PgVersion PgVersion | ExitUnsupportedPgVersion PgVersion PgVersion
| ExitDBNoRecoveryObs | ExitDBNoRecoveryObs
| ExitDBFatalError ObsFatalError SQL.UsageError | ExitDBFatalError ObsFatalError SQL.UsageError
| DBConnectedObs Text | DBConnectedObs Text
| SchemaCacheEmptyObs
| SchemaCacheErrorObs (NonEmpty Text) [Text] SQL.UsageError | SchemaCacheErrorObs (NonEmpty Text) [Text] SQL.UsageError
| SchemaCacheQueriedObs Double | SchemaCacheQueriedObs Double
| SchemaCacheSummaryObs Text | SchemaCacheSummaryObs Text
| SchemaCacheLoadedObs Double | SchemaCacheLoadedObs Double
| ConnectionRetryObs Int | ConnectionRetryObs Int
| DBListenStart (Maybe ByteString) (Maybe ByteString) Text Text -- host, port, version string, channel | DBListenStart Text
| DBListenFail Text (Either SQL.ConnectionError SomeException) | DBListenFail Text (Either SQL.ConnectionError (Either SomeException ()))
| DBListenRetry Int | DBListenRetry Int
| DBListenBugCallQueryFix
| DBListenerGotSCacheMsg ByteString | DBListenerGotSCacheMsg ByteString
| DBListenerGotConfigMsg ByteString | DBListenerGotConfigMsg ByteString
| DBListenerConnectionCleanupFail SomeException | DBQuery ByteString Status
| QueryObs MainQuery Status
| ConfigReadErrorObs SQL.UsageError | ConfigReadErrorObs SQL.UsageError
| ConfigInvalidObs Text | ConfigInvalidObs Text
| ConfigSucceededObs | ConfigSucceededObs
@@ -55,13 +60,105 @@ data Observation
| HasqlPoolObs SQL.Observation | HasqlPoolObs SQL.Observation
| PoolRequest | PoolRequest
| PoolRequestFullfilled | PoolRequestFullfilled
| PoolFlushed
| JwtCacheLookup Bool
| JwtCacheEviction
| TerminationUnixSignalObs Text
| WarpServerObs Text
deriving (Generic)
data ObsFatalError = ServerAuthError | ServerPgrstBug | ServerError42P05 | ServerError08P01 data ObsFatalError = ServerAuthError | ServerPgrstBug | ServerError42P05 | ServerError08P01
type ObservationHandler = Observation -> IO () type ObservationHandler = Observation -> IO ()
observationMessage :: Observation -> Text
observationMessage = \case
AdminStartObs host port ->
"Admin server listening on " <> fromJust host <> ":" <> show (fromIntegral (fromJust port) :: Integer)
AppStartObs ver ->
"Starting PostgREST " <> T.decodeUtf8 ver <> "..."
AppServerPortObs host port ->
"API server listening on " <> host <> ":" <> show port
AppServerUnixObs sock ->
"API server listening on unix socket " <> show sock
DBConnectedObs ver ->
"Successfully connected to " <> ver
ExitUnsupportedPgVersion pgVer minPgVer ->
"Cannot run in this PostgreSQL version (" <> pgvName pgVer <> "), PostgREST needs at least " <> pgvName minPgVer
ExitDBNoRecoveryObs ->
"Automatic recovery disabled, exiting."
ExitDBFatalError ServerAuthError usageErr ->
"Failed to establish a connection. " <> jsonMessage usageErr
ExitDBFatalError ServerPgrstBug usageErr ->
"This is probably a bug in PostgREST, please report it at https://github.com/PostgREST/postgrest/issues. " <> jsonMessage usageErr
ExitDBFatalError ServerError42P05 usageErr ->
"If you are using connection poolers in transaction mode, try setting db-prepared-statements to false. " <> jsonMessage usageErr
ExitDBFatalError ServerError08P01 usageErr ->
"Connection poolers in statement mode are not supported." <> jsonMessage usageErr
SchemaCacheErrorObs dbSchemas extraPaths usageErr ->
"Failed to load the schema cache using "
<> "db-schemas=" <> T.intercalate "," (toList dbSchemas)
<> " and "
<> "db-extra-search-path=" <> T.intercalate "," extraPaths
<> ". " <> jsonMessage usageErr
SchemaCacheQueriedObs resultTime ->
"Schema cache queried in " <> showMillis resultTime <> " milliseconds"
SchemaCacheSummaryObs summary ->
"Schema cache loaded " <> summary
SchemaCacheLoadedObs resultTime ->
"Schema cache loaded in " <> showMillis resultTime <> " milliseconds"
ConnectionRetryObs delay ->
"Attempting to reconnect to the database in " <> (show delay::Text) <> " seconds..."
QueryPgVersionError usageErr ->
"Failed to query the PostgreSQL version. " <> jsonMessage usageErr
DBListenStart channel -> do
"Listening for database notifications on the " <> show channel <> " channel"
DBListenFail channel listenErr ->
"Failed listening for database notifications on the " <> show channel <> " channel. " <>
either showListenerConnError showListenerException listenErr
DBListenRetry delay ->
"Retrying listening for database notifications in " <> (show delay::Text) <> " seconds..."
DBListenerGotSCacheMsg channel ->
"Received a schema cache reload message on the " <> show channel <> " channel"
DBListenerGotConfigMsg channel ->
"Received a config reload message on the " <> show channel <> " channel"
DBQuery sql _ ->
T.decodeUtf8 sql
ConfigReadErrorObs usageErr ->
"Failed to query database settings for the config parameters." <> jsonMessage usageErr
QueryRoleSettingsErrorObs usageErr ->
"Failed to query the role settings. " <> jsonMessage usageErr
QueryErrorCodeHighObs usageErr ->
jsonMessage usageErr
ConfigInvalidObs err ->
"Failed reloading config: " <> err
ConfigSucceededObs ->
"Config reloaded"
PoolInit poolSize ->
"Connection Pool initialized with a maximum size of " <> show poolSize <> " connections"
PoolAcqTimeoutObs usageErr ->
jsonMessage usageErr
HasqlPoolObs (SQL.ConnectionObservation uuid status) ->
"Connection " <> show uuid <> (
case status of
SQL.ConnectingConnectionStatus -> " is being established"
SQL.ReadyForUseConnectionStatus -> " is available"
SQL.InUseConnectionStatus -> " is used"
SQL.TerminatedConnectionStatus reason -> " is terminated due to " <> case reason of
SQL.AgingConnectionTerminationReason -> "max lifetime"
SQL.IdlenessConnectionTerminationReason -> "max idletime"
SQL.ReleaseConnectionTerminationReason -> "release"
SQL.NetworkErrorConnectionTerminationReason _ -> "network error" -- usage error is already logged, no need to repeat the same message.
)
PoolRequest ->
"Trying to borrow a connection from pool"
PoolRequestFullfilled ->
"Borrowed a connection from the pool"
where
showMillis :: Double -> Text
showMillis x = toS $ showFFloat (Just 1) (x * 1000) ""
jsonMessage err = T.decodeUtf8 . LBS.toStrict . Error.errorPayload $ Error.PgError False err
showOnSingleLine txt = T.intercalate " " $ T.filter (/= '\t') <$> T.lines txt -- the errors from hasql-notifications come intercalated with "\t\n"
showListenerConnError :: SQL.ConnectionError -> Text
showListenerConnError = maybe "Connection error" (showOnSingleLine . T.decodeUtf8)
showListenerException :: Either SomeException () -> Text
showListenerException (Right _) = "Failed getting notifications" -- should not happen as the listener will never finish (hasql-notifications uses `forever` internally) with a Right result
showListenerException (Left e) = showOnSingleLine $ show e
+44 -67
View File
@@ -4,8 +4,7 @@ Description : PostgREST Request Planner
This module is in charge of building an intermediate This module is in charge of building an intermediate
representation between the HTTP request and the representation between the HTTP request and the
final response, which may or not result in SQL execution final resulting SQL query.
(computing OpenAPI or OPTIONS requests don't require database interaction)
A query tree is built in case of resource embedding. By inferring the A query tree is built in case of resource embedding. By inferring the
relationship between tables, join conditions are added for every embedded relationship between tables, join conditions are added for every embedded
@@ -23,6 +22,7 @@ module PostgREST.Plan
, InspectPlan(..) , InspectPlan(..)
, InfoPlan(..) , InfoPlan(..)
, CrudPlan(..) , CrudPlan(..)
, CallReadPlan(..)
) where ) where
import qualified Data.HashMap.Strict as HM import qualified Data.HashMap.Strict as HM
@@ -37,7 +37,12 @@ import Data.List (delete, lookup)
import Data.Maybe (fromJust) import Data.Maybe (fromJust)
import Data.Tree (Tree (..)) import Data.Tree (Tree (..))
import PostgREST.ApiRequest (ApiRequest (..)) import PostgREST.ApiRequest (Action (..),
ApiRequest (..),
DbAction (..),
InvokeMethod (..),
Mutation (..),
Payload (..))
import PostgREST.Config (AppConfig (..)) import PostgREST.Config (AppConfig (..))
import PostgREST.Error (ApiRequestError (..), import PostgREST.Error (ApiRequestError (..),
Error (..), Error (..),
@@ -65,6 +70,7 @@ import PostgREST.SchemaCache.Routine (MediaHandler (..),
Routine (..), Routine (..),
RoutineMap, RoutineMap,
RoutineParam (..), RoutineParam (..),
funcReturnsCompositeAlias,
funcReturnsScalar, funcReturnsScalar,
funcReturnsSetOfScalar, funcReturnsSetOfScalar,
funcReturnsSingle) funcReturnsSingle)
@@ -90,13 +96,12 @@ import Protolude hiding (from)
-- Setup for doctests -- Setup for doctests
-- >>> import Data.Ranged.Ranges (fullRange) -- >>> import Data.Ranged.Ranges (fullRange)
-- Plan for reading or writing to the db
data CrudPlan data CrudPlan
= WrappedReadPlan = WrappedReadPlan
{ wrReadPlan :: ReadPlanTree { wrReadPlan :: ReadPlanTree
, pTxMode :: SQL.Mode , pTxMode :: SQL.Mode
, wrHandler :: MediaHandler , wrHandler :: MediaHandler
, pMedia :: MediaType , wrMedia :: MediaType
, wrHdrsOnly :: Bool , wrHdrsOnly :: Bool
, crudQi :: QualifiedIdentifier , crudQi :: QualifiedIdentifier
} }
@@ -105,22 +110,22 @@ data CrudPlan
, mrMutatePlan :: MutatePlan , mrMutatePlan :: MutatePlan
, pTxMode :: SQL.Mode , pTxMode :: SQL.Mode
, mrHandler :: MediaHandler , mrHandler :: MediaHandler
, pMedia :: MediaType , mrMedia :: MediaType
, mrMutation :: Mutation , mrMutation :: Mutation
, crudQi :: QualifiedIdentifier , crudQi :: QualifiedIdentifier
} }
| CallReadPlan {
data CallReadPlan = CallReadPlan {
crReadPlan :: ReadPlanTree crReadPlan :: ReadPlanTree
, crCallPlan :: CallPlan , crCallPlan :: CallPlan
, pTxMode :: SQL.Mode , crTxMode :: SQL.Mode
, crProc :: Routine , crProc :: Routine
, crHandler :: MediaHandler , crHandler :: MediaHandler
, pMedia :: MediaType , crMedia :: MediaType
, crInvMthd :: InvokeMethod , crInvMthd :: InvokeMethod
, crQi :: QualifiedIdentifier , crQi :: QualifiedIdentifier
} }
-- Plan for reading db object metadadta
data InspectPlan = InspectPlan { data InspectPlan = InspectPlan {
ipMedia :: MediaType ipMedia :: MediaType
, ipTxmode :: SQL.Mode , ipTxmode :: SQL.Mode
@@ -128,51 +133,31 @@ data InspectPlan = InspectPlan {
, ipSchema :: Schema , ipSchema :: Schema
} }
-- A Plan may use the the database or not data DbActionPlan = DbCrud CrudPlan | DbCall CallReadPlan | MaybeDb InspectPlan
data ActionPlan data InfoPlan = RelInfoPlan QualifiedIdentifier | RoutineInfoPlan CallReadPlan | SchemaInfoPlan
= Db DbActionPlan data ActionPlan = Db DbActionPlan | NoDb InfoPlan
| NoDb InfoPlan
type IsDbExplain = Bool
-- A db plan can consist on read/write, rpc call or reading metadata (which may use the db or just use cached objects)
data DbActionPlan
= DbCrud IsDbExplain CrudPlan
| MayUseDb InspectPlan
-- Plans that don't use the database
data InfoPlan
= RelInfoPlan QualifiedIdentifier -- info about relation
| RoutineInfoPlan Routine -- info about function
| SchemaInfoPlan -- info about schema cache
actionPlan :: Action -> AppConfig -> ApiRequest -> SchemaCache -> Either Error ActionPlan actionPlan :: Action -> AppConfig -> ApiRequest -> SchemaCache -> Either Error ActionPlan
actionPlan act conf apiReq sCache = case act of actionPlan act conf apiReq sCache = case act of
ActDb dbAct -> Db <$> dbActionPlan dbAct conf apiReq sCache ActDb dbAct -> Db <$> dbActionPlan dbAct conf apiReq sCache
ActRelationInfo ident -> pure . NoDb $ RelInfoPlan ident ActRelationInfo ident -> pure . NoDb $ RelInfoPlan ident
ActRoutineInfo ident inv -> ActRoutineInfo ident inv -> NoDb . RoutineInfoPlan <$> callReadPlan ident conf sCache apiReq inv
let crPln = callReadPlan ident conf sCache apiReq inv in ActSchemaInfo -> pure $ NoDb SchemaInfoPlan
NoDb . RoutineInfoPlan . crProc <$> crPln
ActSchemaInfo -> pure $ NoDb SchemaInfoPlan
dbActionPlan :: DbAction -> AppConfig -> ApiRequest -> SchemaCache -> Either Error DbActionPlan dbActionPlan :: DbAction -> AppConfig -> ApiRequest -> SchemaCache -> Either Error DbActionPlan
dbActionPlan dbAct conf apiReq sCache = case dbAct of dbActionPlan dbAct conf apiReq sCache = case dbAct of
ActRelationRead identifier headersOnly -> ActRelationRead identifier headersOnly ->
toDbActPlan <$> wrappedReadPlan identifier conf sCache apiReq headersOnly DbCrud <$> wrappedReadPlan identifier conf sCache apiReq headersOnly
ActRelationMut identifier mut -> ActRelationMut identifier mut ->
toDbActPlan <$> mutateReadPlan mut apiReq identifier conf sCache DbCrud <$> mutateReadPlan mut apiReq identifier conf sCache
ActRoutine identifier invMethod -> ActRoutine identifier invMethod ->
toDbActPlan <$> callReadPlan identifier conf sCache apiReq invMethod DbCall <$> callReadPlan identifier conf sCache apiReq invMethod
ActSchemaRead tSchema headersOnly -> ActSchemaRead tSchema headersOnly ->
MayUseDb <$> inspectPlan apiReq headersOnly tSchema MaybeDb <$> inspectPlan apiReq headersOnly tSchema
where
toDbActPlan pl = case pMedia pl of
MTVndPlan{} -> DbCrud True pl
_ -> DbCrud False pl
wrappedReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> Bool -> Either Error CrudPlan wrappedReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> Bool -> Either Error CrudPlan
wrappedReadPlan identifier conf sCache apiRequest@ApiRequest{iPreferences=Preferences{..},..} headersOnly = do wrappedReadPlan identifier conf sCache apiRequest@ApiRequest{iPreferences=Preferences{..},..} headersOnly = do
qi <- findTable identifier sCache qi <- findTable identifier (dbTables sCache)
rPlan <- readPlan qi conf sCache apiRequest rPlan <- readPlan qi conf sCache apiRequest
(handler, mediaType) <- mapLeft ApiRequestError $ negotiateContent conf apiRequest qi iAcceptMediaType (dbMediaHandlers sCache) (hasDefaultSelect rPlan) (handler, mediaType) <- mapLeft ApiRequestError $ negotiateContent conf apiRequest qi iAcceptMediaType (dbMediaHandlers sCache) (hasDefaultSelect rPlan)
if not (null invalidPrefs) && preferHandling == Just Strict then Left $ ApiRequestError $ InvalidPreferences invalidPrefs else Right () if not (null invalidPrefs) && preferHandling == Just Strict then Left $ ApiRequestError $ InvalidPreferences invalidPrefs else Right ()
@@ -180,14 +165,14 @@ wrappedReadPlan identifier conf sCache apiRequest@ApiRequest{iPreferences=Prefe
mutateReadPlan :: Mutation -> ApiRequest -> QualifiedIdentifier -> AppConfig -> SchemaCache -> Either Error CrudPlan mutateReadPlan :: Mutation -> ApiRequest -> QualifiedIdentifier -> AppConfig -> SchemaCache -> Either Error CrudPlan
mutateReadPlan mutation apiRequest@ApiRequest{iPreferences=Preferences{..},..} identifier conf sCache = do mutateReadPlan mutation apiRequest@ApiRequest{iPreferences=Preferences{..},..} identifier conf sCache = do
qi <- findTable identifier sCache qi <- findTable identifier (dbTables sCache)
rPlan <- readPlan qi conf sCache apiRequest rPlan <- readPlan qi conf sCache apiRequest
mPlan <- mutatePlan mutation qi apiRequest sCache rPlan mPlan <- mutatePlan mutation qi apiRequest sCache rPlan
if not (null invalidPrefs) && preferHandling == Just Strict then Left $ ApiRequestError $ InvalidPreferences invalidPrefs else Right () if not (null invalidPrefs) && preferHandling == Just Strict then Left $ ApiRequestError $ InvalidPreferences invalidPrefs else Right ()
(handler, mediaType) <- mapLeft ApiRequestError $ negotiateContent conf apiRequest qi iAcceptMediaType (dbMediaHandlers sCache) (hasDefaultSelect rPlan) (handler, mediaType) <- mapLeft ApiRequestError $ negotiateContent conf apiRequest qi iAcceptMediaType (dbMediaHandlers sCache) (hasDefaultSelect rPlan)
return $ MutateReadPlan rPlan mPlan SQL.Write handler mediaType mutation qi return $ MutateReadPlan rPlan mPlan SQL.Write handler mediaType mutation qi
callReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> InvokeMethod -> Either Error CrudPlan callReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> InvokeMethod -> Either Error CallReadPlan
callReadPlan identifier conf sCache apiRequest@ApiRequest{iPreferences=Preferences{preferHandling, invalidPrefs, preferMaxAffected},..} invMethod = do callReadPlan identifier conf sCache apiRequest@ApiRequest{iPreferences=Preferences{preferHandling, invalidPrefs, preferMaxAffected},..} invMethod = do
let paramKeys = case invMethod of let paramKeys = case invMethod of
InvRead _ -> S.fromList $ fst <$> qsParams' InvRead _ -> S.fromList $ fst <$> qsParams'
@@ -256,16 +241,14 @@ findProc qi argumentsKeys allProcs contentMediaType isInvPost =
| hasSingleUnnamedParam proc = (ts,proc:fs) | hasSingleUnnamedParam proc = (ts,proc:fs)
| otherwise = (ts,fs) | otherwise = (ts,fs)
-- If the function is called with post and has a single unnamed parameter -- If the function is called with post and has a single unnamed parameter
-- it can be called depending on content type and the parameter type. -- it can be called depending on content type and the parameter type
-- The parameter must have no declared name (ppName == mempty). hasSingleUnnamedParam Function{pdParams=[RoutineParam{ppType}]} = isInvPost && case (contentMediaType, ppType) of
hasSingleUnnamedParam Function{pdParams=[RoutineParam{ppName, ppType}]} = (MTApplicationJSON, "json") -> True
isInvPost && ppName == mempty && case (contentMediaType, ppType) of (MTApplicationJSON, "jsonb") -> True
(MTApplicationJSON, "json") -> True (MTTextPlain, "text") -> True
(MTApplicationJSON, "jsonb") -> True (MTTextXML, "xml") -> True
(MTTextPlain, "text") -> True (MTOctetStream, "bytea") -> True
(MTTextXML, "xml") -> True _ -> False
(MTOctetStream, "bytea") -> True
_ -> False
hasSingleUnnamedParam _ = False hasSingleUnnamedParam _ = False
matchesParams proc = matchesParams proc =
let let
@@ -812,10 +795,10 @@ validateAggFunctions aggFunctionsAllowed (Node rp@ReadPlan {select} forest)
| otherwise = Node rp <$> traverse (validateAggFunctions aggFunctionsAllowed) forest | otherwise = Node rp <$> traverse (validateAggFunctions aggFunctionsAllowed) forest
-- | Lookup table in the schema cache before creating read plan -- | Lookup table in the schema cache before creating read plan
findTable :: QualifiedIdentifier -> SchemaCache -> Either Error QualifiedIdentifier findTable :: QualifiedIdentifier -> TablesMap -> Either Error QualifiedIdentifier
findTable qi@QualifiedIdentifier{..} sc@SchemaCache{dbTables} = findTable qi@QualifiedIdentifier{..} tableMap =
case HM.lookup qi dbTables of case HM.lookup qi tableMap of
Nothing -> Left $ SchemaCacheErr $ TableNotFound qiSchema qiName sc Nothing -> Left $ SchemaCacheErr $ TableNotFound qiSchema qiName (HM.elems tableMap)
Just _ -> Right qi Just _ -> Right qi
addFilters :: ResolverContext -> ApiRequest -> ReadPlanTree -> Either Error ReadPlanTree addFilters :: ResolverContext -> ApiRequest -> ReadPlanTree -> Either Error ReadPlanTree
@@ -965,17 +948,10 @@ addRanges ApiRequest{..} rReq =
addLogicTrees :: ResolverContext -> ApiRequest -> ReadPlanTree -> Either Error ReadPlanTree addLogicTrees :: ResolverContext -> ApiRequest -> ReadPlanTree -> Either Error ReadPlanTree
addLogicTrees ctx ApiRequest{..} rReq = addLogicTrees ctx ApiRequest{..} rReq =
foldr addLogicTreeToNode (Right rReq) logic foldr addLogicTreeToNode (Right rReq) qsLogic
where where
QueryParams.QueryParams{..} = iQueryParams QueryParams.QueryParams{..} = iQueryParams
logic =
case iAction of
ActDb (ActRelationRead _ _) -> qsLogic
ActDb (ActRoutine _ _) -> qsLogic
-- For mutations, take the non-root logic filters. These will only affect the embeddings and not the top level of the returned representation.
_ -> filter (not . null . fst) qsLogic
addLogicTreeToNode :: (EmbedPath, LogicTree) -> Either Error ReadPlanTree -> Either Error ReadPlanTree addLogicTreeToNode :: (EmbedPath, LogicTree) -> Either Error ReadPlanTree -> Either Error ReadPlanTree
addLogicTreeToNode = updateNode (\t (Node q@ReadPlan{from=fromTable, where_=lf} f) -> Node q{ReadPlan.where_=resolveLogicTree ctx{qi=fromTable} t:lf} f) addLogicTreeToNode = updateNode (\t (Node q@ReadPlan{from=fromTable, where_=lf} f) -> Node q{ReadPlan.where_=resolveLogicTree ctx{qi=fromTable} t:lf} f)
@@ -1048,6 +1024,7 @@ callPlan proc ApiRequest{} paramKeys args readReq = FunctionCall {
, funCArgs = args , funCArgs = args
, funCScalar = funcReturnsScalar proc , funCScalar = funcReturnsScalar proc
, funCSetOfScalar = funcReturnsSetOfScalar proc , funCSetOfScalar = funcReturnsSetOfScalar proc
, funCRetCompositeAlias = funcReturnsCompositeAlias proc
, funCFilterFields = getFilterFieldNames readReq , funCFilterFields = getFilterFieldNames readReq
, funCReturning = inferColsEmbedNeeds readReq [] , funCReturning = inferColsEmbedNeeds readReq []
} }
@@ -1144,7 +1121,7 @@ negotiateContent conf ApiRequest{iAction=act, iPreferences=Preferences{preferRep
m@(MTVndPlan (MTVndSingularJSON strip) _ _) -> mtPlanToNothing $ Just (BuiltinAggSingleJson strip, m) m@(MTVndPlan (MTVndSingularJSON strip) _ _) -> mtPlanToNothing $ Just (BuiltinAggSingleJson strip, m)
m@(MTVndPlan MTVndArrayJSONStrip _ _) -> mtPlanToNothing $ Just (BuiltinAggArrayJsonStrip, m) m@(MTVndPlan MTVndArrayJSONStrip _ _) -> mtPlanToNothing $ Just (BuiltinAggArrayJsonStrip, m)
-- TODO the plan should have its own MediaHandler instead of relying on MediaType -- TODO the plan should have its own MediaHandler instead of relying on MediaType
m@(MTVndPlan mType _ _) -> mtPlanToNothing $ ((,) . fst <$> lookupHandler mType) <*> pure m m@(MTVndPlan mType _ _) -> mtPlanToNothing $ (,) <$> (fst <$> lookupHandler mType) <*> pure m
-- all the other media types can be overridden -- all the other media types can be overridden
x -> lookupHandler x x -> lookupHandler x
mtPlanToNothing x = if configDbPlanEnabled conf then x else Nothing -- don't find anything if the plan media type is not allowed mtPlanToNothing x = if configDbPlanEnabled conf then x else Nothing -- don't find anything if the plan media type is not allowed

Some files were not shown because too many files have changed in this diff Show More