Add blog post on UN Inception performance variance & chaos engineering

- New post: un-inception-performance-variance-chaos-engineering
- Reference AGGREGATED-PERFORMANCE.md report from un-inception repo
- Include 3 performance variance charts (duration, variance, ranking)
- Propose 10 additional chaos engineering experiments
- Update Makefile with agent-friendly format targets (rst2md, llms, jsonfeed, etc)
- Update Lumos cover letter status to Applied

Post explores 2-3x performance variance across releases, orchestrator
placement issues, and reframes variance as chaos engineering opportunity.

Makefile

@@ -1,4 +1,4 @@
-.PHONY: html help clean regenerate serve devserver publish pygments jsonresume jsonresume-themes jsonresume-pdf
+.PHONY: html help clean regenerate serve devserver publish pygments jsonresume jsonresume-themes jsonresume-pdf rst2md llms jsonfeed jsonblog plaintext formats
 
 BASEDIR=$(CURDIR)
 INPUTDIR=$(BASEDIR)/content
@@ -23,6 +23,12 @@ help:
 	@echo '   make serve                          serve site at http://localhost:8000'
 	@echo '   make devserver                      serve and regenerate together      '
 	@echo '   make pygments                       generate pygments stylesheets      '
+	@echo '   make rst2md                         convert RST to MD in output       '
+	@echo '   make llms                           generate llms.txt for AI agents   '
+	@echo '   make jsonfeed                       generate JSON Feed (/feeds/all.json)'
+	@echo '   make jsonblog                       generate JSON Blog (/blog.json)   '
+	@echo '   make plaintext                      generate plain text (index.txt)   '
+	@echo '   make formats                        generate all agent-friendly formats'
 	@echo '                                                                          '
 
 THEME_REPO_URL ?= git@github.com:russellballestrini/pelican-svbhack.git
@@ -158,3 +164,33 @@ pygments:
 	sed -i 's/.highlight { background: #272822; color: #f8f8f2 }/.highlight { color: #f8f8f2 }/' pelican-themes/pelican-svbhack/static/css/pygments-dark.css
 	sed -i 's/.highlight { background: #f8f8f8; }/.highlight { }/' pelican-themes/pelican-svbhack/static/css/pygments-light.css
 	@echo "Done! Pygments stylesheets generated and patched."
+
+rst2md:
+	@echo "Converting RST to Markdown in output directory..."
+	venv/bin/python lib/rst2md.py $(OUTPUTDIR)
+	@echo "Done! Markdown files generated alongside RST sources."
+
+llms:
+	@echo "Generating llms.txt for AI agents..."
+	venv/bin/python lib/generate_llms_txt.py $(OUTPUTDIR) $(SITEURL)
+	@echo "Done! llms.txt generated in output directory."
+
+jsonfeed:
+	@echo "Generating JSON Feed..."
+	venv/bin/python lib/generate_jsonfeed.py $(OUTPUTDIR) $(SITEURL)
+	@echo "Done! JSON Feed generated at /feeds/all.json"
+
+jsonblog:
+	@echo "Generating JSON Blog..."
+	venv/bin/python lib/generate_jsonblog.py $(OUTPUTDIR) $(SITEURL)
+	@echo "Done! JSON Blog generated at /blog.json"
+
+plaintext:
+	@echo "Generating plain text versions..."
+	venv/bin/python lib/generate_plaintext.py $(OUTPUTDIR)
+	@echo "Done! Plain text files generated (index.txt)."
+
+formats: rst2md plaintext jsonfeed jsonblog llms
+	@echo "All agent-friendly formats generated!"
+
+SITEURL ?= https://russell.ballestrini.net

content/2026-01-19-un-inception-performance-variance-chaos-engineering.rst

@@ -0,0 +1,366 @@
+un inception: performance variance & chaos engineering in ci/cd
+###################################################################
+
+:author: Russell Ballestrini
+:slug: un-inception-performance-variance-chaos-engineering
+:date: 2026-01-19 12:00
+:tags: DevOps, CI/CD, Performance, Chaos Engineering, Testing
+:status: published
+
+**The full** `AGGREGATED-PERFORMANCE.md report <https://git.unturf.com/engineering/unturf/un-inception/-/blob/main/AGGREGATED-PERFORMANCE.md>`_ **lives in the un-inception repo.**
+
+I recently analyzed performance variance across 3 releases of UN Inception (4.2.0, 4.2.3, 4.2.4) & discovered something fascinating: the same test suite running the same workload shows 2-3x performance variance. Not random noise. A symptom of architectural misplacement.
+
+the setup
+=========
+
+UN Inception tests 42 programming languages with ~15 tests each. That's over 600 test cases running in parallel on GitLab CI. Each release generates a performance report tracking:
+
+- Average duration per language
+- Queue times & execution times
+- Slowest & fastest languages
+- Overall pipeline duration
+
+Sounds deterministic, right? Same code, same tests, same infrastructure. Should produce consistent metrics.
+
+**Spoiler:** It doesn't.
+
+the variance
+============
+
+**Elixir's journey:**
+
+- 4.2.0: 20 seconds (fast, efficient)
+- 4.2.3: 69 seconds (3.5x slower)
+- 4.2.4: 105 seconds (5x slower than baseline)
+
+**425% variance.** For identical tests.
+
+**C's performance:**
+
+- 4.2.0: 23 seconds
+- 4.2.3: 107 seconds (4.7x slower)
+- 4.2.4: 23 seconds (back to baseline)
+
+**365% variance.** C didn't change. The infrastructure did.
+
+**Rust, Clojure, Scheme:** All showing 120-360% variance.
+
+No consistent pattern. Different languages win & lose in each run. The fastest language in one release becomes the slowest in the next.
+
+.. image:: /uploads/2026/01/aggregated-duration-trend.png
+   :alt: Duration degradation over releases
+   :align: center
+   :width: 100%
+
+the root cause
+==============
+
+After digging through the reports, the culprit became clear: **orchestrator placement on CPU-bound pool**.
+
+**What's happening:**
+
+1. GitLab runner (orchestrator) needs CPU to schedule & coordinate jobs
+2. Test jobs need CPU to compile, run tests, report results
+3. Both compete for limited CPU cycles on the same nodes
+4. Context switching & cache thrashing create unpredictable timing
+5. Matrix generation order becomes random as scheduler equilibrates
+
+**The anti-pattern:**
+
+.. code-block:: text
+
+   ❌ BAD:  [ORCHESTRATOR] + [TEST JOB 1] + [TEST JOB 2] ... on same CPU pool
+   ✅ GOOD: [ORCHESTRATOR] on dedicated node, [TESTS] on separate pool
+
+This violates fundamental SRE principles. You don't run the traffic cop in the middle of the highway.
+
+why it's fun for chaos engineering
+===================================
+
+Here's the twist: **from a chaos engineering perspective, this setup is perfect.**
+
+It reproduces real-world conditions:
+
+- Resource contention (just like production)
+- Non-deterministic scheduling (like when traffic spikes)
+- Race conditions that only appear under load
+- Timing bugs that slip through local testing
+
+**No two runs are identical.** True chaos.
+
+This tests:
+
+- Retry logic robustness
+- Flaky test detection systems
+- Performance monitoring accuracy
+- SLA adherence under adversarial conditions
+
+For stress testing & finding edge cases? Keep this setup. For production CI/CD & meaningful benchmarks? Fix it immediately.
+
+the experiments
+===============
+
+The aggregated report uses dynamic version discovery & generates visualizations via UN's sandbox environment. Charts render using matplotlib in an isolated execution context, then return as artifacts.
+
+**Key insight:** The methodology itself demonstrates UN's value. Generate reports with embedded execution, no local dependencies required.
+
+Here's the pipeline:
+
+.. code-block:: bash
+
+   # Discover all releases dynamically from git tags
+   VERSIONS=$(git tag | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' | sort -V)
+
+   # Copy performance data for each release
+   for v in $VERSIONS; do
+     cp reports/$v/perf.json perf-$v.json
+   done
+
+   # Generate charts via UN (matplotlib in sandbox)
+   build/un -a -f perf-*.json scripts/generate-aggregated-charts.py
+
+   # Generate markdown report
+   python3 scripts/aggregate-performance-reports.py reports AGGREGATED-PERFORMANCE.md
+
+**Ever-growing analysis.** Each new release automatically includes its data in the aggregated report. No hardcoded versions.
+
+what other tests could we do?
+==============================
+
+This performance variance analysis opens doors for additional experiments:
+
+1. orchestrator isolation study
+--------------------------------
+
+**Hypothesis:** Separating orchestrator from compute pool reduces variance to <10%.
+
+**Test:**
+
+- Run 10 releases with orchestrator on CPU-bound pool (current setup)
+- Run 10 releases with orchestrator on dedicated node
+- Compare variance distributions
+- Measure reduction in non-determinism
+
+**Expected outcome:** Variance drops dramatically. Consistent language rankings emerge.
+
+2. concurrency limit sweep
+---------------------------
+
+**Hypothesis:** Fixed concurrency limits stabilize performance at cost of total duration.
+
+**Test:**
+
+.. code-block:: yaml
+
+   # GitLab .gitlab-ci.yml variations
+   trigger-test-matrix:
+     parallel: [8, 16, 32, 64, 128, unlimited]
+
+Run the full test suite at each concurrency level. Measure:
+
+- Total pipeline duration
+- Per-language variance
+- Resource utilization
+- Queue wait times
+
+**Expected outcome:** Sweet spot exists where variance minimizes without excessive serialization.
+
+3. language startup time profiling
+-----------------------------------
+
+**Hypothesis:** Dynamic languages show higher variance due to JIT/GC startup inconsistency.
+
+**Test:**
+
+- Add instrumentation to measure VM startup vs test execution time
+- Separate cold start (first test) from warm runs (subsequent tests)
+- Compare compiled languages (C, Rust, Go) vs dynamic (Python, Ruby, JavaScript)
+
+**Expected outcome:** Dynamic languages show 2-3x variance in startup, compiled languages show <10% variance.
+
+4. cache warming experiments
+-----------------------------
+
+**Hypothesis:** Container image caching reduces variance for languages with heavy dependencies.
+
+**Test:**
+
+- Pre-warm Docker layer cache before matrix execution
+- Measure performance with cold cache vs warm cache
+- Track which languages benefit most from caching
+
+**Expected outcome:** Languages with heavy ecosystems (JavaScript, Python) show largest improvements.
+
+5. time-of-day variance analysis
+---------------------------------
+
+**Hypothesis:** Infrastructure load varies by time, affecting CI performance.
+
+**Test:**
+
+- Trigger identical test runs every 2 hours for 1 week
+- Tag each run with timestamp & day-of-week
+- Correlate variance with calendar patterns
+
+**Expected outcome:** Business hours show higher variance. Weekends & nights show more consistency.
+
+6. resource contention simulation
+----------------------------------
+
+**Hypothesis:** Artificial load mimics production chaos patterns.
+
+**Test:**
+
+- Run stress-ng or similar on CI nodes during test execution
+- Vary CPU/memory/IO pressure levels
+- Measure impact on test variance & failure rates
+
+**Expected outcome:** Controlled chaos reveals which tests are brittle under load.
+
+7. network partition tolerance
+-------------------------------
+
+**Hypothesis:** Tests that depend on external resources show higher variance.
+
+**Test:**
+
+- Add network delay/jitter using tc (traffic control)
+- Randomly drop packets at varying rates
+- Measure which language implementations handle degradation gracefully
+
+**Expected outcome:** Tests with proper timeout/retry logic maintain performance. Others fail or timeout.
+
+8. compiler optimization impact
+--------------------------------
+
+**Hypothesis:** Optimization flags affect not just speed but variance.
+
+**Test:**
+
+For compiled languages, run tests with:
+
+.. code-block:: text
+
+   - No optimization (-O0)
+   - Standard optimization (-O2)
+   - Aggressive optimization (-O3)
+   - Debug symbols vs stripped
+
+Measure both performance & variance at each level.
+
+**Expected outcome:** Higher optimization reduces variance by minimizing branching & improving cache locality.
+
+9. memory pressure cascade
+---------------------------
+
+**Hypothesis:** Memory exhaustion on one test affects subsequent tests.
+
+**Test:**
+
+- Run tests with varying memory limits (cgroup constraints)
+- Deliberately trigger OOM conditions
+- Measure if failures cascade to unrelated tests
+
+**Expected outcome:** Isolated test failures when using proper containerization. Cascading failures indicate shared state.
+
+10. performance archaeology
+----------------------------
+
+**Hypothesis:** Historical variance patterns reveal infrastructure changes.
+
+**Test:**
+
+- Extend analysis back 50+ releases
+- Correlate variance spikes with git commits, infrastructure changes, & CI config updates
+- Build timeline of "what changed when"
+
+**Expected outcome:** Variance spikes align with infrastructure migrations, runner updates, or Kubernetes upgrades.
+
+the value of chaos
+==================
+
+Most teams want stable, predictable CI/CD pipelines. Understandably. But **intentional chaos has value:**
+
+- Reveals hidden assumptions
+- Tests recovery mechanisms
+- Finds race conditions
+- Validates monitoring & alerting
+- Builds confidence in system resilience
+
+The UN Inception variance isn't a bug. It's a feature. A window into how systems behave under real-world conditions.
+
+**For production benchmarks:** Fix the orchestrator placement, set explicit concurrency, isolate resources.
+
+**For chaos testing:** Keep it exactly as-is. Let the scheduler gods do their worst.
+
+reproducibility
+===============
+
+The entire analysis pipeline is reproducible:
+
+.. code-block:: bash
+
+   # Clone the repo
+   git clone https://git.unturf.com/engineering/unturf/un-inception.git
+   cd un-inception
+
+   # Generate the report
+   make perf-aggregate-report
+
+This generates:
+
+- ``AGGREGATED-PERFORMANCE.md`` - Full analysis
+- ``reports/aggregated-*.png`` - Visualizations
+
+Charts render via UN's sandbox environment. No local matplotlib installation needed.
+
+**Step back in time:**
+
+.. code-block:: bash
+
+   git checkout <commit-sha>
+   make perf-aggregate-report
+
+Regenerate historical reports using the exact code & data from any point in history.
+
+future work
+===========
+
+Next steps for this analysis:
+
+1. Implement orchestrator isolation & measure impact
+2. Add real-time variance tracking to CI dashboard
+3. Correlate variance with infrastructure metrics (CPU, memory, network)
+4. Build prediction model for expected variance ranges
+5. Alert when variance exceeds thresholds (signal vs noise)
+6. Extend to other projects beyond UN Inception
+
+The goal isn't zero variance. That's impossible in distributed systems. The goal is **understood, bounded, predictable variance.**
+
+When variance exceeds expectations, that's signal. Something changed. Time to investigate.
+
+conclusion
+==========
+
+Performance variance isn't always bad. Context matters.
+
+For UN Inception, the variance revealed an architectural anti-pattern (orchestrator placement) while simultaneously creating an excellent chaos testing environment.
+
+**The lesson:** Measure, analyze, understand. Don't just chase lower numbers. Understand why the numbers vary.
+
+And when in doubt, **run more experiments.**
+
+.. image:: /uploads/2026/01/aggregated-language-variance.png
+   :alt: Language variance across releases
+   :align: center
+   :width: 100%
+
+read the full report
+====================
+
+The complete analysis with methodology, raw data, & reproducibility instructions:
+
+`AGGREGATED-PERFORMANCE.md on git.unturf.com <https://git.unturf.com/engineering/unturf/un-inception/-/blob/main/AGGREGATED-PERFORMANCE.md>`_
+
+The report auto-updates on each release tag via GitLab CI. Living documentation that grows with the project.

content/pages/cover-letters/2026-01-17-lumos-devops-engineer-russell-ballestrini.rst

@@ -3,7 +3,7 @@ Lumos - DevOps Engineer
 
 :date: 2026-01-17
 :slug: 2026-01-17-lumos-devops-engineer-russell-ballestrini-cover-letter
-:status: draft
+:status: Applied 2026-01-18
 
 To the Lumos team: