The PGRST000 database connection error message was "Database connection
error. Retrying the connection.", but reconnection attempts are already
logged separately by the reconnection observation, and on fatal errors
(e.g. authentication failure) PostgREST does not retry at all. Drop the
"Retrying the connection." part, leaving "Database connection error.".
The config dump in Config.hs listed db-config, db-pre-config and the
db-pool-* settings out of alphabetical order, while the rest of the dump
was sorted. Now the whole dump is consistently alphabetical and update
the expected IO test configs to match.
This was brought over in the last commit, because the postgis was
re-using the same config as another test. Now it has a separate config,
so we can reduce it a bit.
Previously, information about each test-suite was repeated in 3 separate
places:
- as a label and as implicit knowledge in the test-suite itself,
- as a comment in Main.hs, and
- as a configuration in SpecHelper.hs.
With this change, there will be a single source of truth in the test
suite itself. This will allow a single test-suite to easily test
multiple different configurations.
Currently, authentication and response execution each unwrap ExceptT with separate runExceptT calls, which split the main request flow across nested pattern matching and Either handling. Control flow is complex and difficult to understand.
The goal of this change is to make request execution as sequential
monadic code with clear error handling.
To implement that, request handling is now run in ExceptT over WriterT (Last ByteString) IO monad stack. Auth role is written after authentication succeeds and further returned along the response. Thanks to it response observation generation is centralized at the end of request handling.
It was necessary to abstract monad stack in getAuthResult, lookupJwtCache, postgrestResponse, and withTiming to enable introduction of WriterT.
This reports the percentage change between the current head branch and
the main branch, which is exactly the number we'll want to make our
decisions on "success or fail" on.
CI failures will initially be reported for regressions of 5% or more on
an individual number.
Because we separate loadtest results per URL now, we can move the error
tests into the regular mixed bag of loadtests - we will be able to tell
from the misspelled URLs when we hit a regression in that area.
We should be able to do similar things for JWT tests, but we'll need
more infrastructure here.
We previously used "rate", i.e. number of requests per second, as the
primary metric to judge loadtest results. However, this has always been
varying from run to run quite a bit, especially in CI where other jobs
possibly run on the same VM host.
The run-to-run variance has massively increased after splitting the
results up per request. Example run in CI with rate on the PR
introducing this change (on which we would expect no change at all):
| rate [1/s] | main | head | Δ |
|:-----------------------------------|-------:|-------:|-----:|
| / | 9.4 | 9.5 | 1% |
| /actors | 870.4 | 1023.0 | 18% |
| /actors?actor=eq.1 | 188.5 | 198.6 | 5% |
| /actors?actor=eq.1&columns=name | 197.3 | 167.1 | -15% |
| /actors?select=*,roles(*,films(*)) | 153.9 | 144.9 | -6% |
| /films?columns=id,title | 157.9 | 182.6 | 16% |
| /films?columns=id,title,year,... | 87.0 | 87.1 | 0% |
| /roles | 204.5 | 267.3 | 31% |
| /rpc/call_me | 231.3 | 208.8 | -10% |
| /rpc/call_me?name=John | 212.2 | 201.7 | -5% |
From the data we can easily tell that the very reason that rate as a
paramter has only worked, so far, because the data was *heavily*
dominated by the requests on the root endpoint for OpenAPI. The longer
duration makes the request much less vulnerable for concurrent activity.
For all other requests its essentially not possible to judge the effect
of a PR this way.
One way to counter this would be to massively increase the time the
loadtest runs. More samples will result in a smoother average. However,
that's not practical for usability of CI. In the original PR #1812 I
already evaluated using the *minimum latency* as the most reliable
criteriumi, but this has never really caught on. The theory behind this
is: The variation in timings between requests is happening because of
concurrent activity, priority chosen by the scheduler, availability of
resources and such - all factors *outside* our control, and *irrelevant*
to the Haskell code we're writing.
Using the minimum latency is an estimation of how fast the code can run
*in the best case*. This might not be a number relevant for production,
but it's much more directly related to the code we write.
Here's to show how variation becomes *much* smaller with minimum latency
as the parameter:
| min latency [μs] | main | head | Δ |
|:-----------------------------------|---------:|-------:|-----:|
| / | 1275.3 | 1263.6 | -1% |
| /actors | 10.0 | 9.9 | -1% |
| /actors?actor=eq.1 | 50.7 | 48.3 | -5% |
| /actors?actor=eq.1&columns=name | 54.1 | 54.0 | 0% |
| /actors?select=*,roles(*,films(*)) | 63.2 | 61.9 | -2% |
| /films?columns=id,title | 51.1 | 50.7 | -1% |
| /films?columns=id,title,year,... | 121.9 | 121.8 | 0% |
| /roles | 42.9 | 42.6 | -1% |
| /rpc/call_me | 45.6 | 45.4 | 0% |
| /rpc/call_me?name=John | 44.4 | 44.2 | 0% |
Since we're separating results per request now, we can only sensibly
focus on *one* parameter - otherwise this would get really clunky
UI-wise. Especially for automated CI failures, minimum latency is the
logical choice.
This commit starts using minimum latency, i.e. P0, but any percentile
should be an improvement over the status quo. A later commit will change
to a different P-value.
Different requests hit different code paths and perform very
differently. By looking at each request type separately, we should be
able to get a much better idea of what kind of change in performance
we're looking at and where the root cause might be.
It will hopefully also allow us to migrate some of the other test-cases
into the main loadtest.
Ultimately, we only look at the `rate` column, so we can just as well
remove all other columns.
This makes the next step, when we split results by request type, much
less noisy.
This change makes the API surface between MainTx and App smaller.
Currently, App reconstructs a database transaction by unpacking the isolation
level, transaction mode, DbHandler, and transaction runner returned by MainTx.
That exposes MainTx internals at the call site even though MainTx already owns
query setup, execution, decoding, and rollback behavior.
The goal is to keep transaction assembly in MainTx while App remains responsible
for pool execution, database error mapping, and response orchestration. DbTx now
carries the assembled SQL session, and App passes that session directly to the
connection pool.
Schema cache query timings are only needed immediately after a schema-cache reload to emit SchemaCacheQueriedObs. Storing them inside SchemaCache makes the cache carry transient observability data that is not part of the cached schema state and is never used by request handling.
This change makes querySchemaCache to return query timings in a tuple in parallel to SchemaCache and removes dbQueryTimings field.
Config variables are tested already via reading the config files in
the `configs/` directory.
If more are to be tested, it should be done via adding a file in
`configs/` and compare it with its associated file in `configs/expected/`.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
This change gets rid of unnecessary explicit bindRandomPortTCP in initServerSocket. Returned port value was ignored in removed code anyway as assigned port retrieval from an open socket is handled elsewhere.
Remove internal schema cache load and relationship load sleep settings plus
the delay wrappers they enabled. Drop IO tests that depended on the removed
settings.
Add an IO test that drops a table while schema cache reload is delayed. It verifies the stale cache path returns PostgreSQL 42P01 and the refreshed cache returns PGRST205.
Instead of taking wild guesses at the runtime of the target generation
itself, we're just making sure to reset the system time to a fixed value
when we ultimately start PostgREST. This allows us to create the right
JWT expiry values ahead of time.
We decided against doing this in #4913, therefore removing it. With that,
also mentioning that control flow never reaches there to avoid confusion.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
No need to do this in as many jobs. Splitting all the github stuff from
docker stuff into two jobs is enough. This still allows to conditionally
enable docker jobs in contributors repos, depending on whether the
relevant docker credentials are provided - but avoids using too many
concurrent runners.
Right now metrics observation handler does not track database connections but updates a single Gauge based on HasqlPoolObs events. This is problematic because Hasql pool reports various connection events in multiple phases. The connection state machine is not simple and to precisely report the number of connections in various states, it is necessary to track their lifecycles.
This change adds a ConnTrack data structure and logic to track database connections lifecycles. At the moment it supports "connected" and "inUse" connection counts precisely. The "pgrst_db_pool_available" metric is implemented on top of ConnTrack instead of a simple Gauge.
This was required for v9 and earlier, but these don't build with the
current nix invocation anymore anyway. Even loadtesting against v10 does
not work, because `--version` is used in one of the wait scripts and
this was only added in v11.2.
So no need to pretend we'd support comparing against older versions.
The functions `drain_stdout` and `match_log` should be in `util.py`
so they can be reused in other modules.
Signed-off-by: Taimoor Zaeem <taimoorzaeem@gmail.com>
No need to spin up full VM runners for small automation tasks, when we
can use single-CPU runners in containers instead.
https://docs.github.com/en/actions/reference/runners/github-hosted-runners#single-cpu-runners
(some of this will potentially not work, because dependencies in the
slim image might not be available - however, it makes no sense to create
this as a PR, because all jobs touched here run on branches only. Thus
pushing directly to main)
Disabling the autovacuum daemon should also help reproducibility in
theory, although I don't know of any cases where we hit a problem with
that.
VACUUM changes the order of rows that PostgreSQL returns for some table
without explicit ordering, thus doing the latter to make it consistently
reproducible.
After ANALYZE estimates are 100% exact for the moment, so some requests
which returned 206 Partial Response now return 200 instead. The fact
that PostgREST returns 206 on an unfiltered endpoint can probably be
considered a bug.
