docs: Add CI debugging session chronicle

Documents the journey from 'yaml invalid' to green pipeline:
- 8 distinct issues
- 12 commits
- ~90 minutes
- Lessons learned

docs/CI_DEBUGGING_2026-01-17.md

@@ -0,0 +1,222 @@
+# CI/CD Pipeline Debugging Session - 2026-01-17
+
+## The Journey from "yaml invalid" to Green Pipeline
+
+This document chronicles a debugging session that took a broken GitLab CI pipeline from "Unable to create pipeline" to fully passing in approximately 90 minutes and 12 commits.
+
+## Initial State
+
+```
+Unable to create pipeline
+Local file `test-matrix.yml` does not exist!
+```
+
+## Issue #1: Missing `test-matrix.yml`
+
+**Problem**: GitLab validates `include` directives at pipeline creation time, before any jobs run. The pipeline config referenced a file that would be generated by a job, but GitLab needed it to exist in the repo.
+
+**Fix**: Created `test-matrix.yml` as a placeholder file.
+
+**Commit**: `fix(ci): Add test-matrix.yml placeholder for GitLab CI`
+
+---
+
+## Issue #2: Invalid YAML Syntax
+
+**Problem**: The placeholder file had comments but wasn't valid GitLab CI YAML.
+
+```
+Included file `test-matrix.yml` does not have valid YAML syntax!
+```
+
+**Fix**: Created a valid but inactive job definition:
+
+```yaml
+.test-matrix-placeholder:
+  script:
+    - echo "Placeholder"
+  rules:
+    - when: never
+```
+
+**Commit**: `fix(ci): Use valid GitLab CI YAML syntax in placeholder`
+
+---
+
+## Issue #3: No Runner Available
+
+**Problem**: Jobs were stuck in "pending" - no runners matched.
+
+```
+This job is stuck because the project doesn't have any runners online assigned to it.
+```
+
+**Fix**: Added `default: tags: [build]` to `.gitlab-ci.yml` to route jobs to the `build` runner.
+
+**Commit**: `fix(ci): Add build runner tag to all jobs`
+
+---
+
+## Issue #4: Shell Executor vs Docker
+
+**Problem**: The `build` runner uses a shell executor, not Docker. Alpine commands failed.
+
+```
+bash: line 159: apk: command not found
+```
+
+**Fix**: Removed all `image: alpine:latest` and `apk add` commands from `.gitlab-ci.yml`. The shell executor runs directly on the host.
+
+**Commit**: `fix(ci): Remove Docker/Alpine config for shell executor`
+
+---
+
+## Issue #5: Output File Conflict
+
+**Problem**: The `generate-matrix.sh` script wrote to `test-matrix.yml` internally, then the CI config also redirected to the same file.
+
+```
+cat: test-matrix.yml: input file is output file
+```
+
+**Fix**: Removed the `> test-matrix.yml` redirect from `.gitlab-ci.yml` since the script handles file creation internally.
+
+**Commit**: `fix(ci): Fix generate-matrix output conflict and remove Alpine deps`
+
+---
+
+## Issue #6: Missing C Development Libraries
+
+**Problem**: The build server lacked headers for compiling the C SDK.
+
+```
+src/un.c:27:10: fatal error: curl/curl.h: No such file or directory
+```
+
+**Fix**: Updated Salt states (`foxhop-states/gitlab/build-host/ubuntu.sls`) to install:
+- `libcurl4-openssl-dev`
+- `libwebsockets-dev`
+- `libjson-c-dev`
+
+Then ran `salt 'build.unturf.com' state.highstate` to apply.
+
+**Commit**: (in foxhop-states) `feat(build): Add C SDK dependencies for un-inception`
+
+---
+
+## Issue #7: Type Definitions Not Found
+
+**Problem**: Library types like `unsandbox_result_t` weren't defined when building CLI.
+
+```
+src/un.c:5714:28: error: unknown type name 'unsandbox_result_t'
+```
+
+**Root Cause**: The types are defined in `un.h`, which is only included when `UNSANDBOX_LIBRARY` is defined. But we were building CLI without that flag.
+
+**Attempted Fix**: Build with `-DUNSANDBOX_LIBRARY`
+
+**Commit**: `fix(c): Build CLI with library flag to include type definitions`
+
+---
+
+## Issue #8: No main() Function
+
+**Problem**: When `UNSANDBOX_LIBRARY` is defined, `main()` is excluded via `#ifndef`.
+
+```
+undefined reference to `main'
+```
+
+**Code Structure**:
+```c
+#ifdef UNSANDBOX_LIBRARY
+#include "un.h"  // Types defined here
+#endif
+
+// Library API functions (used types but weren't wrapped)
+void unsandbox_free_result(unsandbox_result_t *result) { ... }
+
+#ifndef UNSANDBOX_LIBRARY
+int main(int argc, char *argv[]) { ... }  // CLI entry point
+#endif
+```
+
+**The Dilemma**:
+- Without `UNSANDBOX_LIBRARY`: main() included, but types undefined → compile error
+- With `UNSANDBOX_LIBRARY`: types defined, but main() excluded → link error
+
+**Fix**: Wrapped library API functions (lines 5714-8179) in `#ifdef UNSANDBOX_LIBRARY`:
+
+```c
+#ifdef UNSANDBOX_LIBRARY
+/* Library API - Only compiled when UNSANDBOX_LIBRARY is defined */
+void unsandbox_free_result(unsandbox_result_t *result) { ... }
+// ... all library functions ...
+#endif /* UNSANDBOX_LIBRARY */
+
+#ifndef UNSANDBOX_LIBRARY
+int main(int argc, char *argv[]) { ... }
+#endif
+```
+
+Now:
+- **CLI build** (no flag): Compiles main(), skips library API
+- **Library build** (with flag): Compiles library API, skips main()
+
+**Commit**: `fix(c): Wrap library API functions in ifdef UNSANDBOX_LIBRARY`
+
+---
+
+## Final Result
+
+```
+Pipeline #10647: SUCCESS
+All 9 jobs passed:
+  - detect-changes
+  - generate-matrix
+  - build
+  - science-validate-examples
+  - science-lint-sdks
+  - science-benchmark-clients
+  - validate-examples
+  - generate-documentation
+  - report
+```
+
+## Lessons Learned
+
+1. **GitLab validates includes at parse time** - Dynamic includes via job artifacts don't work as expected. Use placeholder files.
+
+2. **Know your executor** - Shell executors run on the host; Docker executors run in containers. Don't mix commands.
+
+3. **Salt state changes need explicit application** - Pushing to the states repo doesn't auto-apply. Run `state.highstate`.
+
+4. **C preprocessor conditionals are tricky** - When code has `#ifdef` for library vs CLI modes, ensure all dependent code is properly wrapped.
+
+5. **Incremental debugging works** - Each fix revealed the next issue. 12 commits, 8 distinct problems, one green pipeline.
+
+## Commit History
+
+```
+b6d6f16 fix(c): Wrap library API functions in ifdef UNSANDBOX_LIBRARY
+f72b6b4 fix(c): Build CLI with library flag to include type definitions
+4594a0f ci: Retry after curl deps installed
+3c38598 ci: Retry after build server deps installed
+83f2691 feat(ci): Implement inception testing pattern
+c769427 fix(ci): Fix generate-matrix output conflict and remove Alpine deps
+6ac4f53 fix(ci): Remove Docker/Alpine config for shell executor
+22df152 fix(ci): Add build runner tag to all jobs
+164ebcc fix(ci): Use valid GitLab CI YAML syntax in placeholder
+512840c fix(ci): Add test-matrix.yml placeholder for GitLab CI
+```
+
+## Architecture Note: Inception Testing
+
+The CI now implements "inception testing" - using the C CLI (`build/un`) to test all other SDK implementations through the unsandbox API:
+
+```
+build server → un (C binary) → unsandbox API → un.py/un.js/etc → unsandbox API → test code
+```
+
+This means the build server only needs: `gcc`, `libcurl`, `libwebsockets`, `libssl`. No Python, Ruby, PHP, etc. required locally - everything runs through unsandbox.