Admin server was crashing silently and requests hang indefinitely.
With this fix, we are now logging the admin server exceptions and also
close the socket afterwards so admin request don't hang.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Observation module exports observationMessage and isDbListenerBug functions. The first one is used only in Logger module and the second one - only in Listener.
This change moves observationMessage function to Logger module and isDbListenerBug function to Listener module, making all three modules more cohesive and lessening dependencies.
It also gives the compiler more opportunities for intra-module inlining and optimization.
Diagnosing problems with listener channel notifications not being handled properly by PostgREST connected to read replicas is difficult. Issues might be related to lost connections and listener not being connected to the right host after failover or database server restarts.
This patch adds logging of actual host:port used by libpq connection opened by the listener. It should make it easier to find out if PostgREST is connected to the right host.
Some helpers are provided for introspecting metrics already (used in JWT cache tests). This change provides facilities to additionally validate emited Observation events.
A new Spec module is also implemented, adding basic tests of schema cache reloading - their main goal is to excercise the new infrastructure.
Fixes#4646. Using the repro on #4646, this now produces the log:
```
11/Feb/2026:09:40:08 -0500: Warp server error: stack overflow
```
When:
```
$ curl localhost:3000/
curl: (52) Empty reply from server
```
(cherry picked from commit e95e815483)
retryingListen function potentially leaks database connections. This patch ensures the connections are released in case of listen/notify errors.
(cherry picked from commit 00c7cb1a22)
Closes https://github.com/PostgREST/postgrest/issues/4226.
This requires moving query generation to the top App.hs module.
At this point is also simple to log the transaction variables + the
pre-request function call but this is not done here to reduce scope.
Replaces the "API server listening on unix socket" with simpler
"API server listening on " observation. This allows refactoring
redundant code.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
Changes:
1. Refactoring and some cleanup of JWT handling code:
* Instead of caching AuthResult cache decoded claims (which signature was verified). Validating claims and determining role is done after cache lookup
* Cleaned up API so that usage of it is simplified: lookupJwtCache cache key >>= parseClaims configJwtAud time
* Handling of JwtCacheState initialization and updates of configuration is encapsulated in Auth.JwtCache module
2. Generic high performance (hopefully) scalable, dynamically resizeable cache implementation based on stm, stm-hamt and sieve algorithm. It also integrates with PostgREST measurements infrastructure providing usage stats (ie. hit ratio, evictions count)
It's important for observability to have an historic trace of the pool
size. Currently we expose it on the metrics endpoint, but not all
deployments use it.
This logs the pool size after the successful connection log to make it
more visible:
<timestamp>: Connection Pool initialized with a maximum size of 4 connections
It's not immediately clear on which port the API server is listening.
Also it's not clear that the "pgrst" channel is for database
notifications.
Goes from:
<timestamp>: Admin server listening on 0.0.0.0:3001
<timestamp>: Listening on 0.0.0.0:3000
<timestamp>: Listening for notifications on the "pgrst" channel
To:
<timestamp>: Admin server listening on 0.0.0.0:3001
<timestamp>: API server listening on 0.0.0.0:3000
<timestamp>: Listening for database notifications on the "pgrst" channel
Fixes https://github.com/PostgREST/postgrest/issues/3523.
Now if there's a failure when obtaining the pg version OR schema cache,
we do the same retrying process. This way we don't add two retries.
Refactors and renames the "connectionWorker" to "schemaCacheLoader".
This makes more sense since what we really want is the schema cache,
the version is the pre-requisite for ensuring our
schema cache queries work.
Additionally, we no longer log ` Attempting to connect to the database...`
at startup unnecessarily. This is only logged whenever there's a retry attempt.
Update hasql-notifications to include the fix on
https://github.com/diogob/hasql-notifications/issues/24.
Which now reveals the following error:
```
$ postgrest-with-postgresql-16 --replica -f test/spec/fixtures/load.sql postgrest-run
17/May/2024:18:35:38 -0500: Successfully connected to PostgreSQL 16.2 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.0, 64-bit
17/May/2024:18:35:38 -0500: Could not listen for notifications on the "pgrst" channel. ERROR: cannot execute LISTEN during recovery
17/May/2024:18:35:38 -0500: Retrying listening for notifications...
```
This is still not good because the LISTEN channel will be retried
forever without a backoff.
The fatal logic is now inside `usePool`. It centralizes the
logic which is better for Locality of Behavior.
Removes:
- The need to do checkIsFatal on other parts of the code
- SCFatalFail/ConnFatalFail states which are no longer needed.
With this:
- Is no longer necessary to pass observer as an argument
to every function that needs observations.
- We can invoke the observer on every function that uses AppConfig.
However it'd be better to just call the observer in the upper modules
(like on App.hs).
