# CI/CD Pipeline Debugging Session - 2026-01-17

## The Journey from "yaml invalid" to Green Pipeline

This document chronicles a debugging session that took a broken GitLab CI pipeline from "Unable to create pipeline" to fully passing in approximately 90 minutes and 12 commits.

## Initial State

```
Unable to create pipeline
Local file `test-matrix.yml` does not exist!
```

## Issue #1: Missing `test-matrix.yml`

**Problem**: GitLab validates `include` directives at pipeline creation time, before any jobs run. The pipeline config referenced a file that would be generated by a job, but GitLab needed it to exist in the repo.

**Fix**: Created `test-matrix.yml` as a placeholder file.

**Commit**: `fix(ci): Add test-matrix.yml placeholder for GitLab CI`

---

## Issue #2: Invalid YAML Syntax

**Problem**: The placeholder file had comments but wasn't valid GitLab CI YAML.

```
Included file `test-matrix.yml` does not have valid YAML syntax!
```

**Fix**: Created a valid but inactive job definition:

```yaml
.test-matrix-placeholder:
  script:
    - echo "Placeholder"
  rules:
    - when: never
```

**Commit**: `fix(ci): Use valid GitLab CI YAML syntax in placeholder`

---

## Issue #3: No Runner Available

**Problem**: Jobs were stuck in "pending" - no runners matched.

```
This job is stuck because the project doesn't have any runners online assigned to it.
```

**Fix**: Added `default: tags: [build]` to `.gitlab-ci.yml` to route jobs to the `build` runner.

**Commit**: `fix(ci): Add build runner tag to all jobs`

---

## Issue #4: Shell Executor vs Docker

**Problem**: The `build` runner uses a shell executor, not Docker. Alpine commands failed.

```
bash: line 159: apk: command not found
```

**Fix**: Removed all `image: alpine:latest` and `apk add` commands from `.gitlab-ci.yml`. The shell executor runs directly on the host.

**Commit**: `fix(ci): Remove Docker/Alpine config for shell executor`

---

## Issue #5: Output File Conflict

**Problem**: The `generate-matrix.sh` script wrote to `test-matrix.yml` internally, then the CI config also redirected to the same file.

```
cat: test-matrix.yml: input file is output file
```

**Fix**: Removed the `> test-matrix.yml` redirect from `.gitlab-ci.yml` since the script handles file creation internally.

**Commit**: `fix(ci): Fix generate-matrix output conflict and remove Alpine deps`

---

## Issue #6: Missing C Development Libraries

**Problem**: The build server lacked headers for compiling the C SDK.

```
src/un.c:27:10: fatal error: curl/curl.h: No such file or directory
```

**Fix**: Updated Salt states (`foxhop-states/gitlab/build-host/ubuntu.sls`) to install:
- `libcurl4-openssl-dev`
- `libwebsockets-dev`
- `libjson-c-dev`

Then ran `salt 'build.unturf.com' state.highstate` to apply.

**Commit**: (in foxhop-states) `feat(build): Add C SDK dependencies for un-inception`

---

## Issue #7: Type Definitions Not Found

**Problem**: Library types like `unsandbox_result_t` weren't defined when building CLI.

```
src/un.c:5714:28: error: unknown type name 'unsandbox_result_t'
```

**Root Cause**: The types are defined in `un.h`, which is only included when `UNSANDBOX_LIBRARY` is defined. But we were building CLI without that flag.

**Attempted Fix**: Build with `-DUNSANDBOX_LIBRARY`

**Commit**: `fix(c): Build CLI with library flag to include type definitions`

---

## Issue #8: No main() Function

**Problem**: When `UNSANDBOX_LIBRARY` is defined, `main()` is excluded via `#ifndef`.

```
undefined reference to `main'
```

**Code Structure**:
```c
#ifdef UNSANDBOX_LIBRARY
#include "un.h"  // Types defined here
#endif

// Library API functions (used types but weren't wrapped)
void unsandbox_free_result(unsandbox_result_t *result) { ... }

#ifndef UNSANDBOX_LIBRARY
int main(int argc, char *argv[]) { ... }  // CLI entry point
#endif
```

**The Dilemma**:
- Without `UNSANDBOX_LIBRARY`: main() included, but types undefined → compile error
- With `UNSANDBOX_LIBRARY`: types defined, but main() excluded → link error

**Fix**: Wrapped library API functions (lines 5714-8179) in `#ifdef UNSANDBOX_LIBRARY`:

```c
#ifdef UNSANDBOX_LIBRARY
/* Library API - Only compiled when UNSANDBOX_LIBRARY is defined */
void unsandbox_free_result(unsandbox_result_t *result) { ... }
// ... all library functions ...
#endif /* UNSANDBOX_LIBRARY */

#ifndef UNSANDBOX_LIBRARY
int main(int argc, char *argv[]) { ... }
#endif
```

Now:
- **CLI build** (no flag): Compiles main(), skips library API
- **Library build** (with flag): Compiles library API, skips main()

**Commit**: `fix(c): Wrap library API functions in ifdef UNSANDBOX_LIBRARY`

---

## Final Result

```
Pipeline #10647: SUCCESS
All 9 jobs passed:
  - detect-changes
  - generate-matrix
  - build
  - science-validate-examples
  - science-lint-sdks
  - science-benchmark-clients
  - validate-examples
  - generate-documentation
  - report
```

## Lessons Learned

1. **GitLab validates includes at parse time** - Dynamic includes via job artifacts don't work as expected. Use placeholder files.

2. **Know your executor** - Shell executors run on the host; Docker executors run in containers. Don't mix commands.

3. **Salt state changes need explicit application** - Pushing to the states repo doesn't auto-apply. Run `state.highstate`.

4. **C preprocessor conditionals are tricky** - When code has `#ifdef` for library vs CLI modes, ensure all dependent code is properly wrapped.

5. **Incremental debugging works** - Each fix revealed the next issue. 12 commits, 8 distinct problems, one green pipeline.

## Commit History

```
b6d6f16 fix(c): Wrap library API functions in ifdef UNSANDBOX_LIBRARY
f72b6b4 fix(c): Build CLI with library flag to include type definitions
4594a0f ci: Retry after curl deps installed
3c38598 ci: Retry after build server deps installed
83f2691 feat(ci): Implement inception testing pattern
c769427 fix(ci): Fix generate-matrix output conflict and remove Alpine deps
6ac4f53 fix(ci): Remove Docker/Alpine config for shell executor
22df152 fix(ci): Add build runner tag to all jobs
164ebcc fix(ci): Use valid GitLab CI YAML syntax in placeholder
512840c fix(ci): Add test-matrix.yml placeholder for GitLab CI
```

## Architecture Note: Inception Testing

The CI now implements "inception testing" - using the C CLI (`build/un`) to test all other SDK implementations through the unsandbox API:

```
build server → un (C binary) → unsandbox API → un.py/un.js/etc → unsandbox API → test code
```

This means the build server only needs: `gcc`, `libcurl`, `libwebsockets`, `libssl`. No Python, Ruby, PHP, etc. required locally - everything runs through unsandbox.