un-inception/docs/CI_DEBUGGING_2026-01-17.md

CI/CD Pipeline Debugging Session - 2026-01-17

The Journey from "yaml invalid" to Green Pipeline

This document chronicles a debugging session that took a broken GitLab CI pipeline from "Unable to create pipeline" to fully passing in approximately 90 minutes and 12 commits.

Initial State

Unable to create pipeline
Local file `test-matrix.yml` does not exist!

Issue #1: Missing test-matrix.yml

Problem: GitLab validates include directives at pipeline creation time, before any jobs run. The pipeline config referenced a file that would be generated by a job, but GitLab needed it to exist in the repo.

Fix: Created test-matrix.yml as a placeholder file.

Commit: fix(ci): Add test-matrix.yml placeholder for GitLab CI

---

Issue #2: Invalid YAML Syntax

Problem: The placeholder file had comments but wasn't valid GitLab CI YAML.

Included file `test-matrix.yml` does not have valid YAML syntax!

Fix: Created a valid but inactive job definition:

.test-matrix-placeholder:
  script:
    - echo "Placeholder"
  rules:
    - when: never

Commit: fix(ci): Use valid GitLab CI YAML syntax in placeholder

---

Issue #3: No Runner Available

Problem: Jobs were stuck in "pending" - no runners matched.

This job is stuck because the project doesn't have any runners online assigned to it.

Fix: Added default: tags: [build] to .gitlab-ci.yml to route jobs to the build runner.

Commit: fix(ci): Add build runner tag to all jobs

---

Issue #4: Shell Executor vs Docker

Problem: The build runner uses a shell executor, not Docker. Alpine commands failed.

bash: line 159: apk: command not found

Fix: Removed all image: alpine:latest and apk add commands from .gitlab-ci.yml. The shell executor runs directly on the host.

Commit: fix(ci): Remove Docker/Alpine config for shell executor

---

Issue #5: Output File Conflict

Problem: The generate-matrix.sh script wrote to test-matrix.yml internally, then the CI config also redirected to the same file.

cat: test-matrix.yml: input file is output file

Fix: Removed the > test-matrix.yml redirect from .gitlab-ci.yml since the script handles file creation internally.

Commit: fix(ci): Fix generate-matrix output conflict and remove Alpine deps

---

Issue #6: Missing C Development Libraries

Problem: The build server lacked headers for compiling the C SDK.

src/un.c:27:10: fatal error: curl/curl.h: No such file or directory

Fix: Updated Salt states (foxhop-states/gitlab/build-host/ubuntu.sls) to install:

• libcurl4-openssl-dev
• libwebsockets-dev
• libjson-c-dev

Then ran salt 'build.unturf.com' state.highstate to apply.

Commit: (in foxhop-states) feat(build): Add C SDK dependencies for un-inception

---

Issue #7: Type Definitions Not Found

Problem: Library types like unsandbox_result_t weren't defined when building CLI.

src/un.c:5714:28: error: unknown type name 'unsandbox_result_t'

Root Cause: The types are defined in un.h, which is only included when UNSANDBOX_LIBRARY is defined. But we were building CLI without that flag.

Attempted Fix: Build with -DUNSANDBOX_LIBRARY

Commit: fix(c): Build CLI with library flag to include type definitions

---

Issue #8: No main() Function

Problem: When UNSANDBOX_LIBRARY is defined, main() is excluded via #ifndef.

undefined reference to `main'

Code Structure:

#ifdef UNSANDBOX_LIBRARY
#include "un.h"  // Types defined here
#endif

// Library API functions (used types but weren't wrapped)
void unsandbox_free_result(unsandbox_result_t *result) { ... }

#ifndef UNSANDBOX_LIBRARY
int main(int argc, char *argv[]) { ... }  // CLI entry point
#endif

The Dilemma:

• Without UNSANDBOX_LIBRARY: main() included, but types undefined → compile error
• With UNSANDBOX_LIBRARY: types defined, but main() excluded → link error

Fix: Wrapped library API functions (lines 5714-8179) in #ifdef UNSANDBOX_LIBRARY:

#ifdef UNSANDBOX_LIBRARY
/* Library API - Only compiled when UNSANDBOX_LIBRARY is defined */
void unsandbox_free_result(unsandbox_result_t *result) { ... }
// ... all library functions ...
#endif /* UNSANDBOX_LIBRARY */

#ifndef UNSANDBOX_LIBRARY
int main(int argc, char *argv[]) { ... }
#endif

Now:

• CLI build (no flag): Compiles main(), skips library API
• Library build (with flag): Compiles library API, skips main()

Commit: fix(c): Wrap library API functions in ifdef UNSANDBOX_LIBRARY

---

Final Result

Pipeline #10647: SUCCESS
All 9 jobs passed:
  - detect-changes
  - generate-matrix
  - build
  - science-validate-examples
  - science-lint-sdks
  - science-benchmark-clients
  - validate-examples
  - generate-documentation
  - report

Lessons Learned

1. GitLab validates includes at parse time - Dynamic includes via job artifacts don't work as expected. Use placeholder files.

2. Know your executor - Shell executors run on the host; Docker executors run in containers. Don't mix commands.

3. Salt state changes need explicit application - Pushing to the states repo doesn't auto-apply. Run state.highstate.

4. C preprocessor conditionals are tricky - When code has #ifdef for library vs CLI modes, ensure all dependent code is properly wrapped.

5. Incremental debugging works - Each fix revealed the next issue. 12 commits, 8 distinct problems, one green pipeline.

Commit History

b6d6f16 fix(c): Wrap library API functions in ifdef UNSANDBOX_LIBRARY
f72b6b4 fix(c): Build CLI with library flag to include type definitions
4594a0f ci: Retry after curl deps installed
3c38598 ci: Retry after build server deps installed
83f2691 feat(ci): Implement inception testing pattern
c769427 fix(ci): Fix generate-matrix output conflict and remove Alpine deps
6ac4f53 fix(ci): Remove Docker/Alpine config for shell executor
22df152 fix(ci): Add build runner tag to all jobs
164ebcc fix(ci): Use valid GitLab CI YAML syntax in placeholder
512840c fix(ci): Add test-matrix.yml placeholder for GitLab CI

Architecture Note: Inception Testing

The CI now implements "inception testing" - using the C CLI (build/un) to test all other SDK implementations through the unsandbox API:

build server → un (C binary) → unsandbox API → un.py/un.js/etc → unsandbox API → test code

This means the build server only needs: gcc, libcurl, libwebsockets, libssl. No Python, Ruby, PHP, etc. required locally - everything runs through unsandbox.