1.1.26

Fix: stale-404 cache poisoning

A transient 404 from the upstream was being written to disk and then served as STALE 404 indefinitely for that URI, even after the upstream had recovered. Symptom: GET /get_term_info?id=FBbt_00014013 returning 404 from the cache while the same request hit upstream returned 200.

Root cause

Three settings combined to make a single bad upstream response permanent:

• proxy_cache_valid 404 600s wrote the 404 into the cache.
• inactive=5y on proxy_cache_path kept the file on disk for five years past TTL.
• proxy_cache_use_stale … updating http_404 plus proxy_cache_background_update on served the stale 404 immediately on every request and prevented a recovered upstream from displacing it.

Changes

nginx.conf.template

• Removed proxy_cache_valid 404 600s. Status codes not listed in proxy_cache_valid are not cached, which is what we want for 404s.
• Removed http_404 from proxy_cache_use_stale, so a recovered upstream's response is no longer overridden by a stale cached 404.

New: purge-cached-404s.sh (shipped at /usr/local/bin/purge-cached-404s.sh in the image)

A surgical cleanup tool for caches that already contain poisoned 404 entries. Walks the cache directory and matches each file's stored response status line (anchored to start-of-line, so a 200 response whose body contains the literal text "HTTP/1.1 404" is not affected). Files removed from disk are simply treated as MISS on the next request — no nginx reload required.

docker exec -it owlery-cache purge-cached-404s.sh           # dry run, lists candidates
docker exec -it owlery-cache purge-cached-404s.sh --apply   # delete matching files

Honours $CACHE_DIR (default /var/cache/nginx/owlery).

Upgrade notes

After the new image is running on each cache, run purge-cached-404s.sh --apply once per container to evict any pre-existing poisoned 404 entries. The bulk of the cache is left intact; only entries whose stored response is 404 are removed.

1.1.25

Fix: stale-404 cache poisoning

A transient 404 from the upstream was being written to disk and then served as STALE 404 indefinitely for that URI, even after the upstream had recovered. Symptom: GET /get_term_info?id=FBbt_00014013 returning 404 from the cache while the same request hit upstream returned 200.

Root cause

Three settings combined to make a single bad upstream response permanent:

• proxy_cache_valid 404 600s wrote the 404 into the cache.
• inactive=5y on proxy_cache_path kept the file on disk for five years past TTL.
• proxy_cache_use_stale … updating http_404 plus proxy_cache_background_update on served the stale 404 immediately on every request and prevented a recovered upstream from displacing it.

Changes

nginx.conf.template

• Removed proxy_cache_valid 404 600s. Status codes not listed in proxy_cache_valid are not cached, which is what we want for 404s.
• Removed http_404 from proxy_cache_use_stale, so a recovered upstream's response is no longer overridden by a stale cached 404.

New: purge-cached-404s.sh (shipped at /usr/local/bin/purge-cached-404s.sh in the image)

A surgical cleanup tool for caches that already contain poisoned 404 entries. Walks the cache directory and matches each file's stored response status line (anchored to start-of-line, so a 200 response whose body contains the literal text "HTTP/1.1 404" is not affected). Files removed from disk are simply treated as MISS on the next request — no nginx reload required.

docker exec -it owlery-cache purge-cached-404s.sh           # dry run, lists candidates
docker exec -it owlery-cache purge-cached-404s.sh --apply   # delete matching files

Honours $CACHE_DIR (default /var/cache/nginx/owlery).

Upgrade notes

After the new image is running on each cache, run purge-cached-404s.sh --apply once per container to evict any pre-existing poisoned 404 entries. The bulk of the cache is left intact; only entries whose stored response is 404 are removed.

1.1.24

Fix nginx blocked-IP map generation by normalizing and deduplicating entries in blocked and whitelist map builders.

1.1.23

Fix: increase nginx map_hash_bucket_size to 128

v1.1.22

Fix envsubst export for FORCE_CACHE_REFRESH_ON_REQUEST so the container startup and CI nginx config validation work correctly.

v1.1.21

Add FORCE_CACHE_REFRESH_ON_REQUEST runtime environment support to bypass cache on demand and refresh entries when requested.

1.1.20

Full Changelog: 1.1.19...1.1.20

1.1.14

Traffic hardening release: increase worker connection capacity, add cache lock tuning, fail fast on upstream connect issues, and preserve long read/send windows for heavy queries.

1.1.13

Test release for caching long-running upstream responses after client timeouts/disconnects.

1.1.12

Full Changelog: 1.1.11...1.1.12