diff --git a/reports/aggregate-27-28.md b/reports/aggregate-27-28.md new file mode 100644 index 0000000..45ef61f --- /dev/null +++ b/reports/aggregate-27-28.md @@ -0,0 +1,496 @@ +# UN Inception: Aggregated Performance Analysis + +**Analysis Date:** 1769299539.4496503 +**Reports Analyzed:** 4.2.27, 4.2.28 + +--- + +## Executive Summary + +Analysis of 2 performance reports with the **new separated infrastructure architecture**: + +**Infrastructure Changes (effective 4.2.27):** +- API and orchestrator now run on **dedicated infrastructure** (no longer on compute pool) +- Compute workloads distributed across **2 pool types** + +**Pool Specifications:** +- **Xeon Pool**: Intel Xeon, 32 vCPU, 300GB RAM +- **i9 Pool**: Intel i9, 32 vCPU, 32GB RAM + +This separation eliminates the previous resource contention issues where the orchestrator competed with test jobs for CPU cycles. + +--- + +## Key Findings + +### 1. Extreme Metric Variance + +| Release | Avg Duration | Slowest | Fastest | Change from Previous | +|---------|--------------|---------|---------|----------------------| +| 4.2.27 | 119s | commonlisp (176s) | php (50s) | baseline | +| 4.2.28 | 116s | commonlisp (145s) | awk (102s) | -3s (-2.5%) | + +**Observation:** Average duration decreased **2.5%** from 119s to 116s. + +With the orchestrator now on dedicated infrastructure, we're seeing: +- More stable execution patterns +- Reduced contention between orchestrator and compute jobs +- More consistent resource allocation across pools + +--- + +### 2. Unstable Language Rankings + +The same language changes dramatically in rank between runs: + + +**PERL:** + - 4.2.27: 53s + - 4.2.28: 143s + - **Range:** 53s → 143s (169.8% variance) + +**LUA:** + - 4.2.27: 51s + - 4.2.28: 140s + - **Range:** 51s → 140s (174.5% variance) + +**BASH:** + - 4.2.27: 52s + - 4.2.28: 137s + - **Range:** 52s → 137s (163.5% variance) + +**R:** + - 4.2.27: 54s + - 4.2.28: 137s + - **Range:** 54s → 137s (153.7% variance) + +**FORTRAN:** + - 4.2.27: 175s + - 4.2.28: 103s + - **Range:** 103s → 175s (69.9% variance) + + +--- + +### 3. Execution Order Non-Determinism + +**Fastest Languages by Run:** + +4.2.27: php, lua, bash, perl, r +4.2.28: awk, zig, powershell, objc, nim + +**Slowest Languages by Run:** + +4.2.27: commonlisp, fortran, d, zig, powershell +4.2.28: commonlisp, perl, lua, r, bash + +**Conclusion:** No consistent "fast" or "slow" languages across runs. This proves: +- Execution order is random or system-dependent +- Resource availability varies dramatically +- Each run experiences different contention patterns + +--- + +## Infrastructure Architecture + +### Current Setup (Separated) + +The API and orchestrator now run on **dedicated infrastructure**, separate from the compute pools: + +``` +✅ CURRENT: [API + ORCHESTRATOR] on dedicated node + [TEST JOBS] distributed across compute pools +``` + +**Pool Configuration:** +- **Xeon Pool**: Intel Xeon, 32 vCPU, 300GB RAM (memory-intensive workloads) +- **i9 Pool**: Intel i9, 32 vCPU, 32GB RAM (CPU-intensive workloads) + +**Benefits of this architecture:** +1. Orchestrator has dedicated CPU for scheduling/coordination +2. Test jobs don't compete with orchestrator for resources +3. More predictable timing and reproducible results +4. Better resource isolation and capacity planning +5. Workloads can be assigned to appropriate pool based on requirements + +--- + +## Concurrency Hypothesis + +### Theory: Matrix Hydra Execution Limits + +Given 42 languages with 15 tests each, if there were a **concurrency limit**, we'd expect: + +**Observed avg duration:** 33-70s +**If truly serialized (1 job at a time):** ~500s minimum +**If unlimited parallel:** ~50-70s + +This suggests jobs run in **parallel batches**, but the batch size varies: + +#### Possible Concurrency Models: + +1. **Kubernetes Executor (default 32-64 parallel):** Each release has different load +2. **GitLab runner queue saturation:** Some runs hit limits, others don't +3. **Node CPU throttling:** Kubernetes QoS class limits being applied +4. **No explicit limit, but OS scheduler bottleneck:** ~64 thread context limit + +### Evidence from Timing Patterns + +If concurrency was fixed at N parallel jobs: +- `Total time = ceiling(42 / N) * (average job time)` +- For 4.2.0 (33s avg): ~42 concurrent or very efficient scheduling +- For 4.2.3 (63s avg): ~20 concurrent (slower overall, more contention) +- For 4.2.4 (70s avg): ~18 concurrent (even more contention) + +**Implication:** Concurrency limit is either: +- **Dynamic** (based on available resources) +- **Not enforced** (unlimited, but OS scheduler creates natural limit) +- **Degrading** (orchestrator consuming more CPU over versions) + +--- + +## Detailed Language Analysis + +### Most Variable Languages + + +PERL: 53s → 143s (+169.8%) + +LUA: 51s → 140s (+174.5%) + +BASH: 52s → 137s (+163.5%) + +R: 54s → 137s (+153.7%) + +FORTRAN: 103s → 175s (+69.9%) + +ELIXIR: 61s → 131s (+114.8%) + +D: 103s → 171s (+66.0%) + +ERLANG: 61s → 129s (+111.5%) + +ZIG: 103s → 169s (+64.1%) + +PHP: 50s → 110s (+120.0%) + + +These languages are most affected by resource contention. Likely reasons: +- **Dynamic languages** (Python, Ruby, JavaScript): Startup time varies with GC/JIT +- **Compiled languages with heavy linking** (C++, Rust): Linker contention +- **Language VMs** (Java, Elixir): VM startup sensitive to system load + +--- + +## Recommendations + +### Current Architecture (Optimized) + +With the API and orchestrator now separated from the compute pool, focus on: + +1. **Monitor the separated architecture** + - Track variance reduction compared to previous runs (pre-4.2.27) + - Verify orchestrator has sufficient resources + - Monitor pool utilization across both pool types (Xeon vs i9) + +2. **Optimize pool utilization** + - Balance workloads across different pool types + - Consider workload affinity (memory-heavy vs CPU-heavy languages) + - Monitor queue depths per pool + +3. **Continue collecting metrics** + - Compare variance before/after separation + - Identify any remaining bottlenecks + - Track improvements in reproducibility + +### Expected Improvements + +With dedicated orchestrator infrastructure: +- Reduced timing variance between runs +- More predictable performance benchmarks +- Easier capacity planning and cost prediction +- Better debug reproducibility + +--- + +## Raw Data: Language Variance Table + +| Language | Min (s) | Max (s) | Avg (s) | Range (s) | Variance % | +|----------|---------|---------|---------|-----------|------------| +| LUA | 51 | 140 | 95.5 | 89 | 174.5% | +| PERL | 53 | 143 | 98.0 | 90 | 169.8% | +| BASH | 52 | 137 | 94.5 | 85 | 163.5% | +| R | 54 | 137 | 95.5 | 83 | 153.7% | +| PHP | 50 | 110 | 80.0 | 60 | 120.0% | +| ELIXIR | 61 | 131 | 96.0 | 70 | 114.8% | +| ERLANG | 61 | 129 | 95.0 | 68 | 111.5% | +| JAVASCRIPT | 58 | 113 | 85.5 | 55 | 94.8% | +| RUBY | 58 | 112 | 85.0 | 54 | 93.1% | +| TYPESCRIPT | 58 | 112 | 85.0 | 54 | 93.1% | +| PYTHON | 65 | 114 | 89.5 | 49 | 75.4% | +| FORTRAN | 103 | 175 | 139.0 | 72 | 69.9% | +| D | 103 | 171 | 137.0 | 68 | 66.0% | +| ZIG | 103 | 169 | 136.0 | 66 | 64.1% | +| POWERSHELL | 103 | 162 | 132.5 | 59 | 57.3% | +| COBOL | 103 | 153 | 128.0 | 50 | 48.5% | +| DENO | 111 | 157 | 134.0 | 46 | 41.4% | +| JAVA | 103 | 145 | 124.0 | 42 | 40.8% | +| KOTLIN | 104 | 144 | 124.0 | 40 | 38.5% | +| CLOJURE | 112 | 154 | 133.0 | 42 | 37.5% | + + +--- + +## Visualizations + +### Duration Degradation Trend +![Duration Trend](aggregated-duration-trend.png) + +**Shows:** Average test duration increasing 2.1x from 4.2.0 → 4.2.4 + +### Language Variance Heatmap +![Language Variance](aggregated-language-variance.png) + +**Shows:** Top 15 most unstable languages, with Elixir, TCL, and C showing >300% variance + +### Ranking Instability +![Ranking Changes](aggregated-ranking-changes.png) + +**Shows:** The same languages moving dramatically in performance rankings across releases + +--- + +## Conclusion + +These 2 releases (4.2.27 and 4.2.28) are the first with the **new separated infrastructure architecture**: + +- API and orchestrator on dedicated infrastructure +- Compute workloads distributed across 2 pool types (Xeon + i9) + +**Early observations:** +1. Average duration relatively stable (119s -> 116s, -2.5%) +2. Some variance still present due to pool heterogeneity +3. Compare these results with pre-4.2.27 releases to measure improvement + +**Next steps:** +- Continue monitoring variance trends +- Collect more data points with separated architecture +- Fine-tune workload distribution across pool types + +--- + +## Reproducibility & Methodology + +### Pipeline Overview + +This aggregated report is generated from individual performance reports collected during CI/CD runs. The pipeline combines data analysis, statistical variance calculation, and visualization rendering. + +**Architecture:** +``` +Individual Reports → Aggregation Script → Chart Generation (via UN) → Final Report + (perf.json) (Python) (matplotlib) (Markdown) +``` + +### Data Sources + +**Input Files:** +- `reports/4.2.27/perf.json` - 653 tests, generated 2026-01-24T23:27:26Z +- `reports/4.2.28/perf.json` - 645 tests, generated 2026-01-24T23:31:17Z + + +Each `perf.json` contains: +- Pipeline metadata (tag, timestamp, pipeline IDs) +- Summary statistics (avg, min, max durations) +- Per-language results (42 languages × ~15 tests each) +- Queue times & execution durations + +**Data Collection:** +1. GitLab CI triggers test matrix (42 languages in parallel) +2. Each language job reports timing via GitLab API +3. `scripts/generate-perf-report.sh` queries API & generates `perf.json` +4. Report committed to `reports/{TAG}/` directory + +### Analysis Pipeline + +**Step 1: Variance Analysis** (`scripts/aggregate-performance-reports.py`) + +```python +# Load all reports +for version_dir in Path('reports').iterdir(): + reports[version] = json.loads((version_dir / 'perf.json').read_text()) + +# Extract language timings +for version, perf_data in reports.items(): + for lang_entry in perf_data['languages']: + language_timings[version][lang_entry['language']] = lang_entry['duration_seconds'] + +# Calculate variance per language +for lang in all_languages: + durations = [language_timings[v][lang] for v in versions if lang in language_timings[v]] + percent_variance = ((max(durations) - min(durations)) / min(durations) * 100) +``` + +**Step 2: Chart Generation** (`scripts/generate-aggregated-charts.py`) + +Charts are generated using **matplotlib inside UN sandbox** (not local environment): + +```bash +# Copy reports with version-tagged names +cp reports/4.2.0/perf.json perf-4.2.0.json +cp reports/4.2.3/perf.json perf-4.2.3.json +cp reports/4.2.4/perf.json perf-4.2.4.json + +# Execute chart generation via UN (includes matplotlib) +build/un -a \ + -f perf-4.2.0.json \ + -f perf-4.2.3.json \ + -f perf-4.2.4.json \ + scripts/generate-aggregated-charts.py + +# Artifacts returned: *.png files +``` + +**Why UN for Charts?** +- Matplotlib not installed locally (by design) +- UN sandbox provides pre-configured Python environment with matplotlib +- Ensures reproducibility across different machines +- Same approach used in GitLab CI/CD pipeline + +**Step 3: Report Generation** + +```bash +# Generate markdown report (no matplotlib needed locally) +python3 scripts/aggregate-performance-reports.py reports AGGREGATED-PERFORMANCE.md +``` + +### Reproducing This Report + +**Prerequisites:** +- Git repository checked out +- `build/un` binary (UN Inception CLI client) +- Python 3.x (for report generation, not charts) +- Access to `reports/` directory with historical data + +**Command:** +```bash +make perf-aggregate-report +``` + +**Or manually:** +```bash +# Step 1: Generate charts +cp reports/4.2.0/perf.json perf-4.2.0.json +cp reports/4.2.3/perf.json perf-4.2.3.json +cp reports/4.2.4/perf.json perf-4.2.4.json +build/un -a -f perf-4.2.0.json -f perf-4.2.3.json -f perf-4.2.4.json scripts/generate-aggregated-charts.py +rm -f perf-*.json +mv *.png reports/ + +# Step 2: Generate markdown report +python3 scripts/aggregate-performance-reports.py reports AGGREGATED-PERFORMANCE.md +``` + +### Stepping Back in Time + +To regenerate this report with historical data: + +1. **Checkout the specific commit:** + ```bash + git checkout + ``` + +2. **Verify reports exist:** + ```bash + ls -la reports/4.2.0/perf.json + ls -la reports/4.2.3/perf.json + ls -la reports/4.2.4/perf.json + ``` + +3. **Run analysis:** + ```bash + make perf-aggregate-report + ``` + +### CI/CD Integration + +This report auto-generates on release tags via GitLab CI: + +```yaml +perf-aggregate-report: + stage: report + needs: [perf-report] + script: + - echo "Generating aggregated analysis..." + - cp reports/4.2.0/perf.json perf-4.2.0.json + - cp reports/4.2.3/perf.json perf-4.2.3.json + - cp reports/4.2.4/perf.json perf-4.2.4.json + - build/un -a -f perf-4.2.0.json -f perf-4.2.3.json -f perf-4.2.4.json scripts/generate-aggregated-charts.py + - python3 scripts/aggregate-performance-reports.py reports AGGREGATED-PERFORMANCE.md + - git add reports/ AGGREGATED-PERFORMANCE.md + - git commit -m "perf: Update aggregated performance analysis [ci skip]" + - git push origin main + rules: + - if: '$CI_COMMIT_TAG =~ /^\d+\.\d+\.\d+$/' +``` + +**When new release tagged:** Pipeline automatically updates aggregated report with new data point. + +### Statistical Methods + +**Variance Calculation:** +- Per-language min/max/avg across all releases +- Percent variance: `((max - min) / min) * 100` +- Languages with <2 data points excluded + +**Ranking Analysis:** +- Languages sorted by duration per release +- Top 10 slowest tracked across releases +- Ranking position changes indicate non-determinism + +**Concurrency Estimation:** +- Average duration vs theoretical serialized time +- Estimated parallel capacity: `ceiling(42 langs / avg_duration) * per_job_time` +- Variance suggests dynamic (not fixed) concurrency + +### Tools & Dependencies + +**Local Environment:** +- Python 3.x (standard library only) +- `build/un` - UN Inception CLI +- Git (for version control) +- Bash (for scripting) + +**UN Sandbox Environment:** +- Python 3.x with matplotlib, numpy +- Pre-configured visualization environment +- Isolated execution (no local dependencies) + +**GitLab CI:** +- GitLab Runner with `build` tag +- Environment variables: `UNSANDBOX_PUBLIC_KEY`, `UNSANDBOX_SECRET_KEY` +- Deploy key for auto-commit + +### Data Integrity + +**Validation:** +- JSON schema validation on input files +- Version tag format validation (`X.Y.Z`) +- Minimum 2 releases required for variance analysis + +**Timestamps:** +- All reports include generation timestamp +- Commit history provides audit trail +- CI pipeline IDs link back to source runs + +### Contact & Questions + +For questions about this methodology or to report issues: +- Repository: `git.unturf.com/engineering/unturf/un-inception` +- Methodology issues: Open issue with `[methodology]` tag +- Data integrity concerns: Check commit history & CI pipeline logs + +--- + +**Generated by UN Inception Performance Analysis Pipeline** +**Analysis Date:** 2026-01-24T19:05:47.107351 +**Report Version:** 1.0.0