diff --git a/docs/CI_POSTMORTEM_4.2.46.md b/docs/CI_POSTMORTEM_4.2.46.md new file mode 100644 index 0000000..9107a85 --- /dev/null +++ b/docs/CI_POSTMORTEM_4.2.46.md @@ -0,0 +1,94 @@ +# CI Pipeline Postmortem - Release 4.2.46 (2026-01-29) + +## Summary + +Release 4.2.46 triggered the full 42-language test matrix. **31 of 43 test jobs failed** (72% failure rate). Root cause analysis revealed three distinct failure categories, none of which indicate actual SDK bugs. + +## Impact + +- **12 languages passed**: python, ruby, bash, cpp, fortran, d, nim, v, crystal, fsharp, haskell, prolog +- **31 languages failed**: All others +- Pipeline status reported "success" because test jobs have `allow_failure: true` + +## Root Causes + +### 1. QR Test Missing Dependencies (23 failures) + +**What**: QR code generation tests (`qr_generate`) require language-specific libraries (`qrcode` npm, python `qrcode`, etc.) that are not installed in the sandbox environment. + +**Why it failed**: The test treated missing dependencies as a test FAIL rather than a SKIP. The QR test files (e.g., `test/qr.js`) do `require('qrcode')` which fails with `Cannot find module 'qrcode'` in the sandbox. + +**Why python passed**: The `qrcode` pip package happens to be pre-installed in the sandbox image. + +**Fix in 4.2.48**: QR test now detects missing dependency errors and reports SKIP instead of FAIL. This is honest - we can't test what we can't run. + +### 2. Service Operations 5xx Errors (13 failures) + +**What**: `service_destroy` and `service_exec` tests failed with HTTP 5xx errors from the API, even after retry exhaustion. + +**Root cause - pool imbalance**: The cluster has two pool hosts: +- **Cammy** (`cammy-foxhop-net`): 32 vCPUs, load 37.96 (118% utilization), **21 services** +- **AI** (`ai-foxhop-net`): 32 vCPUs, load 9.26 (29% utilization), **0 services** + +All 21 services were placed on Cammy while AI sat idle. The API's service placement algorithm appears to always select the first available pool rather than load-balancing across pools. + +**Why `service_destroy` was especially bad**: A 5xx on destroy could mean the service was actually destroyed but the response was lost. The next retry would get "Service not found" which didn't match the success pattern `destroy|deleted|success`, and also didn't match any transient error pattern, so it fell into the "no-match" retry path (only 3 attempts, 2s delay). + +**Fixes in 4.2.48**: +- `service_destroy` now accepts "not found" as success (service was already destroyed) +- Backoff cap increased from 10s to 30s +- No-match retry increased from 3 to 5 attempts with 3s delay +- Added nginx error messages to 5xx detection (bad gateway, service unavailable, gateway timeout) + +**Remaining**: The pool imbalance is an API-side issue that needs a load-balancing fix in `api.unsandbox.com`. + +### 3. Inline Code Format Issues (5 failures) + +**What**: `exec_basic` failed for erlang, clojure, kotlin, objc, and deno. These languages can't run their "hello world" code via `build/un -s LANG 'code'`. + +**Why**: The hello code format in `test-sdk.sh` doesn't match what the sandbox expects: +- **Erlang**: Module-based syntax (`-module(main).`) doesn't work for inline execution via escript +- **Kotlin**: `fun main() { println("test-ok") }` needs a `.kt` file, not inline +- **Deno**: "deno" isn't a valid language name for the API's `-s` flag +- **Clojure**: Inline format incompatible with sandbox's Clojure runner +- **Obj-C**: Requires compilation, can't run inline + +**Status**: Not fixed in 4.2.48. These need per-language hello code adjustments or acceptance that inline execution isn't supported for all languages. + +## Failure Output Improvement + +**Problem**: When tests failed, the output showed only `head -5` of the output file, which often missed the actual error. HTTP status codes (500, 502, 503) were not displayed. + +**Fix in 4.2.48**: Failure output now shows: +1. Matched HTTP status codes and error messages (via grep) +2. First 5 lines of raw output +3. Clear section markers for readability + +## Change Detection Bug + +**Problem**: The version bump for 4.2.46 changed 14 files across 8 language directories, but the main branch pipeline only ran 8 SDK tests instead of the full matrix. A version bump is a release-level change that should trigger full testing. + +**Root cause**: `detect-changes.sh` line 140 checked for `tests?/|scripts/|\.gitlab-ci\.yml|clients/\{.*\}/` but did NOT check for `VERSION` or `Makefile` changes. The `clients/\{.*\}/` pattern with literal braces was also a bug - it would only match paths like `clients/{template}/` which don't exist. + +**Fix in 4.2.48**: Changed to `^(VERSION|Makefile|tests?/|scripts/|\.gitlab-ci\.yml)` - VERSION and Makefile changes now trigger full matrix. + +## Timeline + +| Time | Event | +|------|-------| +| 20:20 | 4.2.46 tag pushed, pipelines start | +| 20:25 | Child pipeline 13574 starts full 42-language matrix | +| 20:28 | Python passes 20/20 (first to complete) | +| 20:35 | First failures appear (service 5xx, QR deps) | +| 20:45 | Second wave failures as Cammy becomes more overloaded | +| 21:45 | Pipeline 13574 completes: 12 pass, 31 fail | +| 21:50 | Root cause analysis complete | +| 22:00 | Fixes committed, 4.2.48 tagged | + +## Lessons Learned + +1. **QR tests assumed dependencies exist** - sandbox doesn't have every language's QR library. Tests should detect and SKIP, not FAIL. +2. **Service placement needs load balancing** - a single overloaded host causes cascading failures across all test languages. +3. **Retry logic needs to understand operation semantics** - "not found" after a destroy attempt is success, not failure. +4. **Core file changes should trigger full matrix** - VERSION bumps are release events. +5. **Failure output must include HTTP status codes** - without them, diagnosing API issues requires manual log reading.