diff --git a/docs/CI_DEBUGGING_2026-01-17.md b/docs/CI_DEBUGGING_2026-01-17.md new file mode 100644 index 0000000..3e413a2 --- /dev/null +++ b/docs/CI_DEBUGGING_2026-01-17.md @@ -0,0 +1,222 @@ +# CI/CD Pipeline Debugging Session - 2026-01-17 + +## The Journey from "yaml invalid" to Green Pipeline + +This document chronicles a debugging session that took a broken GitLab CI pipeline from "Unable to create pipeline" to fully passing in approximately 90 minutes and 12 commits. + +## Initial State + +``` +Unable to create pipeline +Local file `test-matrix.yml` does not exist! +``` + +## Issue #1: Missing `test-matrix.yml` + +**Problem**: GitLab validates `include` directives at pipeline creation time, before any jobs run. The pipeline config referenced a file that would be generated by a job, but GitLab needed it to exist in the repo. + +**Fix**: Created `test-matrix.yml` as a placeholder file. + +**Commit**: `fix(ci): Add test-matrix.yml placeholder for GitLab CI` + +--- + +## Issue #2: Invalid YAML Syntax + +**Problem**: The placeholder file had comments but wasn't valid GitLab CI YAML. + +``` +Included file `test-matrix.yml` does not have valid YAML syntax! +``` + +**Fix**: Created a valid but inactive job definition: + +```yaml +.test-matrix-placeholder: + script: + - echo "Placeholder" + rules: + - when: never +``` + +**Commit**: `fix(ci): Use valid GitLab CI YAML syntax in placeholder` + +--- + +## Issue #3: No Runner Available + +**Problem**: Jobs were stuck in "pending" - no runners matched. + +``` +This job is stuck because the project doesn't have any runners online assigned to it. +``` + +**Fix**: Added `default: tags: [build]` to `.gitlab-ci.yml` to route jobs to the `build` runner. + +**Commit**: `fix(ci): Add build runner tag to all jobs` + +--- + +## Issue #4: Shell Executor vs Docker + +**Problem**: The `build` runner uses a shell executor, not Docker. Alpine commands failed. + +``` +bash: line 159: apk: command not found +``` + +**Fix**: Removed all `image: alpine:latest` and `apk add` commands from `.gitlab-ci.yml`. The shell executor runs directly on the host. + +**Commit**: `fix(ci): Remove Docker/Alpine config for shell executor` + +--- + +## Issue #5: Output File Conflict + +**Problem**: The `generate-matrix.sh` script wrote to `test-matrix.yml` internally, then the CI config also redirected to the same file. + +``` +cat: test-matrix.yml: input file is output file +``` + +**Fix**: Removed the `> test-matrix.yml` redirect from `.gitlab-ci.yml` since the script handles file creation internally. + +**Commit**: `fix(ci): Fix generate-matrix output conflict and remove Alpine deps` + +--- + +## Issue #6: Missing C Development Libraries + +**Problem**: The build server lacked headers for compiling the C SDK. + +``` +src/un.c:27:10: fatal error: curl/curl.h: No such file or directory +``` + +**Fix**: Updated Salt states (`foxhop-states/gitlab/build-host/ubuntu.sls`) to install: +- `libcurl4-openssl-dev` +- `libwebsockets-dev` +- `libjson-c-dev` + +Then ran `salt 'build.unturf.com' state.highstate` to apply. + +**Commit**: (in foxhop-states) `feat(build): Add C SDK dependencies for un-inception` + +--- + +## Issue #7: Type Definitions Not Found + +**Problem**: Library types like `unsandbox_result_t` weren't defined when building CLI. + +``` +src/un.c:5714:28: error: unknown type name 'unsandbox_result_t' +``` + +**Root Cause**: The types are defined in `un.h`, which is only included when `UNSANDBOX_LIBRARY` is defined. But we were building CLI without that flag. + +**Attempted Fix**: Build with `-DUNSANDBOX_LIBRARY` + +**Commit**: `fix(c): Build CLI with library flag to include type definitions` + +--- + +## Issue #8: No main() Function + +**Problem**: When `UNSANDBOX_LIBRARY` is defined, `main()` is excluded via `#ifndef`. + +``` +undefined reference to `main' +``` + +**Code Structure**: +```c +#ifdef UNSANDBOX_LIBRARY +#include "un.h" // Types defined here +#endif + +// Library API functions (used types but weren't wrapped) +void unsandbox_free_result(unsandbox_result_t *result) { ... } + +#ifndef UNSANDBOX_LIBRARY +int main(int argc, char *argv[]) { ... } // CLI entry point +#endif +``` + +**The Dilemma**: +- Without `UNSANDBOX_LIBRARY`: main() included, but types undefined → compile error +- With `UNSANDBOX_LIBRARY`: types defined, but main() excluded → link error + +**Fix**: Wrapped library API functions (lines 5714-8179) in `#ifdef UNSANDBOX_LIBRARY`: + +```c +#ifdef UNSANDBOX_LIBRARY +/* Library API - Only compiled when UNSANDBOX_LIBRARY is defined */ +void unsandbox_free_result(unsandbox_result_t *result) { ... } +// ... all library functions ... +#endif /* UNSANDBOX_LIBRARY */ + +#ifndef UNSANDBOX_LIBRARY +int main(int argc, char *argv[]) { ... } +#endif +``` + +Now: +- **CLI build** (no flag): Compiles main(), skips library API +- **Library build** (with flag): Compiles library API, skips main() + +**Commit**: `fix(c): Wrap library API functions in ifdef UNSANDBOX_LIBRARY` + +--- + +## Final Result + +``` +Pipeline #10647: SUCCESS +All 9 jobs passed: + - detect-changes + - generate-matrix + - build + - science-validate-examples + - science-lint-sdks + - science-benchmark-clients + - validate-examples + - generate-documentation + - report +``` + +## Lessons Learned + +1. **GitLab validates includes at parse time** - Dynamic includes via job artifacts don't work as expected. Use placeholder files. + +2. **Know your executor** - Shell executors run on the host; Docker executors run in containers. Don't mix commands. + +3. **Salt state changes need explicit application** - Pushing to the states repo doesn't auto-apply. Run `state.highstate`. + +4. **C preprocessor conditionals are tricky** - When code has `#ifdef` for library vs CLI modes, ensure all dependent code is properly wrapped. + +5. **Incremental debugging works** - Each fix revealed the next issue. 12 commits, 8 distinct problems, one green pipeline. + +## Commit History + +``` +b6d6f16 fix(c): Wrap library API functions in ifdef UNSANDBOX_LIBRARY +f72b6b4 fix(c): Build CLI with library flag to include type definitions +4594a0f ci: Retry after curl deps installed +3c38598 ci: Retry after build server deps installed +83f2691 feat(ci): Implement inception testing pattern +c769427 fix(ci): Fix generate-matrix output conflict and remove Alpine deps +6ac4f53 fix(ci): Remove Docker/Alpine config for shell executor +22df152 fix(ci): Add build runner tag to all jobs +164ebcc fix(ci): Use valid GitLab CI YAML syntax in placeholder +512840c fix(ci): Add test-matrix.yml placeholder for GitLab CI +``` + +## Architecture Note: Inception Testing + +The CI now implements "inception testing" - using the C CLI (`build/un`) to test all other SDK implementations through the unsandbox API: + +``` +build server → un (C binary) → unsandbox API → un.py/un.js/etc → unsandbox API → test code +``` + +This means the build server only needs: `gcc`, `libcurl`, `libwebsockets`, `libssl`. No Python, Ruby, PHP, etc. required locally - everything runs through unsandbox.