Admin server was crashing silently and requests hang indefinitely.
With this fix, we are now logging the admin server exceptions and also
close the socket afterwards so admin request don't hang.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
BREAKING CHANGE
Breaks the string comparison operators implemented in #3813. Those can
be replaced with regex searches using JSON Path `search()` function.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
For admin server, port server-setting was set to the same port as
app server port. This doesn't have any effect because it is overridden
by the already initialized admin server socket, but it's better to
explicitly correct it to avoid anything unexpected.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
This change ensures PostgREST starts listening on a server socket only after it loaded the schema cache and is ready to handle requests. It is no longer going to return 503 errors during startup until the schema cache is loaded.
Right now stateNextListenerDelay is only used in Listener module.
This change is a small refactoring moving nextListenerDelay to Listener to increase cohesion and decrease coupling.
This is useful when multiple instances run on the same machine, for
example behind a proxy. Unix sockets for web and admin servers can then
be put in the same folder for each instance.
Can be helpful when writing tests as well.
retryingSchemaCacheLoad should not clear existing schema cache upon failure - there is no reason to do that. If there is a communication issue with the database server or db is down, clients are going to get 502 anyway. If it was a glitch when loading the schema cache - the clients are going to use old (stale) schema cache for some time until next retry re-loads it successfully.
Replaces the `Maybe [Text]` with `[Text]`. The `Maybe` is unnecessary
because we handle `Just []` and `Nothing` the same way.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
The PGRST000 database connection error message was "Database connection
error. Retrying the connection.", but reconnection attempts are already
logged separately by the reconnection observation, and on fatal errors
(e.g. authentication failure) PostgREST does not retry at all. Drop the
"Retrying the connection." part, leaving "Database connection error.".
The config dump in Config.hs listed db-config, db-pre-config and the
db-pool-* settings out of alphabetical order, while the rest of the dump
was sorted. Now the whole dump is consistently alphabetical and update
the expected IO test configs to match.
Currently, authentication and response execution each unwrap ExceptT with separate runExceptT calls, which split the main request flow across nested pattern matching and Either handling. Control flow is complex and difficult to understand.
The goal of this change is to make request execution as sequential
monadic code with clear error handling.
To implement that, request handling is now run in ExceptT over WriterT (Last ByteString) IO monad stack. Auth role is written after authentication succeeds and further returned along the response. Thanks to it response observation generation is centralized at the end of request handling.
It was necessary to abstract monad stack in getAuthResult, lookupJwtCache, postgrestResponse, and withTiming to enable introduction of WriterT.
This change makes the API surface between MainTx and App smaller.
Currently, App reconstructs a database transaction by unpacking the isolation
level, transaction mode, DbHandler, and transaction runner returned by MainTx.
That exposes MainTx internals at the call site even though MainTx already owns
query setup, execution, decoding, and rollback behavior.
The goal is to keep transaction assembly in MainTx while App remains responsible
for pool execution, database error mapping, and response orchestration. DbTx now
carries the assembled SQL session, and App passes that session directly to the
connection pool.
Schema cache query timings are only needed immediately after a schema-cache reload to emit SchemaCacheQueriedObs. Storing them inside SchemaCache makes the cache carry transient observability data that is not part of the cached schema state and is never used by request handling.
This change makes querySchemaCache to return query timings in a tuple in parallel to SchemaCache and removes dbQueryTimings field.
This change gets rid of unnecessary explicit bindRandomPortTCP in initServerSocket. Returned port value was ignored in removed code anyway as assigned port retrieval from an open socket is handled elsewhere.
Remove internal schema cache load and relationship load sleep settings plus
the delay wrappers they enabled. Drop IO tests that depended on the removed
settings.
We decided against doing this in #4913, therefore removing it. With that,
also mentioning that control flow never reaches there to avoid confusion.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Right now metrics observation handler does not track database connections but updates a single Gauge based on HasqlPoolObs events. This is problematic because Hasql pool reports various connection events in multiple phases. The connection state machine is not simple and to precisely report the number of connections in various states, it is necessary to track their lifecycles.
This change adds a ConnTrack data structure and logic to track database connections lifecycles. At the moment it supports "connected" and "inUse" connection counts precisely. The "pgrst_db_pool_available" metric is implemented on top of ConnTrack instead of a simple Gauge.
Disabling the autovacuum daemon should also help reproducibility in
theory, although I don't know of any cases where we hit a problem with
that.
VACUUM changes the order of rows that PostgreSQL returns for some table
without explicit ordering, thus doing the latter to make it consistently
reproducible.
After ANALYZE estimates are 100% exact for the moment, so some requests
which returned 206 Partial Response now return 200 instead. The fact
that PostgREST returns 206 on an unfiltered endpoint can probably be
considered a bug.
This commit removes auth middleware for it hides
side effects and obscures logic. The auth operations
are now done in its own stage in the request-response
cycle.
It also removes the logging middleware because now
we instead use observation module to log the response.
Renames the "Functions" in the logs to "RPCs". This clarifies that
we log number of callable functions and not the number of SQL objects.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
PostgREST failed when querying role settings where current
role name contained uppercase letters. This commit resolves
it by quoting the CURRENT_USER.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Upgraded warp to 3.4.13 which fixed https://github.com/yesodweb/wai/issues/853
Changed interrupt handling so that instead of killing the main thread, listening sockets are closed which triggers warp graceful shutdown.
This adds a new log line that shows each schema cache query time individually, only on
`log-level=debug`. Like so:
```
$ PGRST_LOG_LEVEL=debug postgrest-with-pg-17 -f test/spec/fixtures/load.sql postgrest-run
....
10/Apr/2026:21:48:45 -0500: Schema cache queried in 192.2 milliseconds
10/Apr/2026:21:48:45 -0500: tables: 72.027 ms, keydeps: 20.118 ms, rels: 6.189 ms, funcs: 35.010 ms, comprels: 4.319 ms, dreps: 1.614 ms, mhandlers: 7.419 ms, tzones: 43.025 ms
```
This helps debug specific schema cache queries being slow like on
https://github.com/PostgREST/postgrest/issues/4613#issuecomment-4210191065 and
https://github.com/PostgREST/postgrest/issues/3046#issuecomment-3469059948.
It also closes https://github.com/PostgREST/postgrest/issues/3215,
which main motivation was to find out which query is slow.
Implementation details
---------------------
To time each query inside a transaction in pure SQL, we do:
```sql
-- start timer
select set_config('pgrst.tmp_x', clock_timestamp()::text, false);
-- run the query
select <query>
-- end timer
select set_config('pgrst.tmp_x', (clock_timestamp() - current_setting('pgrst.tmp_x', false)::timestamptz)::text, false);
-- .... repeated for every query
-- at the end we capture all the timings with
select extract('milliseconds' from current_setting('pgrst.tmp_x', false)::interval), extract(..;
```
Considerations
--------------
Only added this on `log-level=debug` because while the queries are fast
and the data is valuable, it triples the amount of queries we run during schema cache refresh,
which could be troublesome on slow networks. It's possible to reduce the
amount of queries by starting and stopping timers in one statement, but
this would still double the amount of queries and makes the code messy,
doesn't seem worth it.
Also it would pollute pg_stat_statements, it's only required to debug certain
extreme cases anyway.
retryingSchemaCacheLoad flushes the pool upon every retry before it starts reloading the schema. This is too early as schema reloading might take some time during which new connections might be acquired. The consequence is that:
* upon successful schema cache reload we might have some connections created with the old schema cache
* we close connections upon each retry and under load we will keep closing and re-opening connections until schema cache load succeeds
This change is to make sure we flush the pool only after successful schema cache querying but before loading (so that connections acquired during loading wait for it and do not interfere with timing the loading process).
DISCLAIMER:
This commit was authored entirely by a human without the assistance of LLMs.
Using debouncer to trigger schema cache loading makes it difficult to understand when exactly it is triggered.
Fixed integer type mapping in OpenAPI 2.0: replaced the invalid integer format with int32/int64 and added the toSwaggerFormat function to map PostgreSQL types to valid OpenAPI 2.0 formats:
smallint -> int32
integer -> int32
bigint -> int64
To avoid repeated querying of `pg_timezone_names` every time schema
cache is reset, `Prefer: timezone` can be disabled by setting
`db-timezone-enabled = false`.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Some helpers are provided for introspecting metrics already (used in JWT cache tests). This change provides facilities to additionally validate emited Observation events.
A new Spec module is also implemented, adding basic tests of schema cache reloading - their main goal is to excercise the new infrastructure.
