Now collects separate time series for each pool:
- ai-foxhop-net: load1/5/15, available, total
- cammy-foxhop-net: load1/5/15, available, total
Saves to raw-pools.csv and includes per-pool stats in pool-metrics.json
Fixed perf-aggregate-report job failure caused by artifact files
conflicting with git checkout.
The issue:
- Job downloads artifacts from perf-report (reports/4.2.5/*)
- Tries to checkout main with 'git checkout -B main origin/main'
- Fails because artifact files would be overwritten
The solution:
1. Stash all artifact files and generated content
2. Checkout main branch cleanly
3. Pull latest changes from origin/main
4. Pop stash to restore artifacts
5. Add and commit aggregated report
This preserves the artifact files while properly syncing with remote main.
Convert complex shell commands to multi-line blocks to fix YAML parsing:
- Changed `test -z "$VAR" && ... || true` patterns to proper if/else blocks
- Used `|` syntax for multi-line shell scripts
- Fixed variable expansion in $FILES for build/un command
- Separated git diff --cached logic into if/else block
This resolves the error:
"jobs:perf-aggregate-report:script config should be a string or
a nested array of strings up to 10 levels deep"
YAML validated successfully with Python yaml.safe_load()
- Add perf-aggregate-report CI job to analyze variance across releases
- Implement dynamic version discovery using git tags (ever-growing)
- Generate charts via UN sandbox using matplotlib
- Create AGGREGATED-PERFORMANCE.md with comprehensive methodology
- Add Makefile target for local report generation
- Include 3 visualization charts showing variance trends
Key findings: 2-3x performance variance due to orchestrator placement
on CPU-bound pool causing non-deterministic scheduling.
- Wait 60s after test matrix completes for data to settle
- Checkout main branch BEFORE generating reports to avoid unstaged changes
- Use git checkout -B to force-create branch from origin/main
- Add input validation for tag format
- Check for required tools (curl, jq, bc)
- Skip if report already exists (idempotent)
- Add colored logging for better visibility
- Handle push failures gracefully
- Use 'rules' instead of 'only' for modern GitLab CI
- Add GIT_DEPTH: 0 for full history access
- Add scripts/generate-perf-charts.py for matplotlib visualizations
- Update generate-perf-report.sh for versioned directories (reports/TAG/)
- Add make perf-charts and make perf-all targets
- Add GitLab CI perf-report job to auto-commit after tagged releases
- Generated 6 charts for 4.2.0: dashboard, duration, histogram, leaders, etc.
Each release now gets its own reports/TAG/ directory with:
- perf.json (raw timing data)
- perf.md (markdown report)
- chart-*.png (6 visualizations)
- Implement detect-changes stage: identifies which SDKs changed
- Implement generate-matrix stage: creates dynamic test matrix based on changes
- Only test SDKs that changed (5x faster than testing all 42)
- Parallel test execution via GitLab matrix strategy
- Science jobs for pool burning: validate-examples, lint-all-sdks, benchmark-clients
- Zero cost execution: uses warm pool + idle capacity
- Comprehensive reporting with JUnit XML and markdown summaries
Pipeline flow:
detect-changes → generate-matrix → build → test (parallel) → science → report
The unfair advantage:
- GitLab sees changes, tests only what's needed
- GitHub shows traditional Actions (external view)
- Internal: 5x faster, $0 per execution
- External: looks normal (strategic asymmetry)
