- Add perf-aggregate-report CI job to analyze variance across releases
- Implement dynamic version discovery using git tags (ever-growing)
- Generate charts via UN sandbox using matplotlib
- Create AGGREGATED-PERFORMANCE.md with comprehensive methodology
- Add Makefile target for local report generation
- Include 3 visualization charts showing variance trends
Key findings: 2-3x performance variance due to orchestrator placement
on CPU-bound pool causing non-deterministic scheduling.
- Add setup-ci-deploy-key target to automate SSH deploy key creation
- Use protected=true so DEPLOY_KEY only exposed on protected tags
- Add test-ci-deploy-key target to verify SSH access
- Add remove-ci-deploy-key target for cleanup
- Include instructions about protecting release tags in GitLab
- Add scripts/generate-perf-charts.py for matplotlib visualizations
- Update generate-perf-report.sh for versioned directories (reports/TAG/)
- Add make perf-charts and make perf-all targets
- Add GitLab CI perf-report job to auto-commit after tagged releases
- Generated 6 charts for 4.2.0: dashboard, duration, histogram, leaders, etc.
Each release now gets its own reports/TAG/ directory with:
- perf.json (raw timing data)
- perf.md (markdown report)
- chart-*.png (6 visualizations)
Add complete testing infrastructure that validates clients in:
1. CLI MODE: Test as standalone command-line tool
- Argument parsing (--help, --version)
- File execution (un.py code.py)
- Environment variables (-e VAR=val)
- Commands (execute, session, service)
2. LIBRARY MODE: Test as importable SDK
- Client object creation
- Method availability (execute, create_session)
- Authentication/HMAC generation
- Return types and data structures
3. INTEGRATION MODE: Test API contract validation
- Valid/invalid authentication (200/401)
- Language support verification
- Error handling (rate limits, timeouts)
- Artifacts and file operations
- Environment variable passing
4. FUNCTIONAL MODE: Real-world usage scenarios
- Fibonacci calculation
- Data analysis (pandas, etc.)
- Web requests (semitrusted mode)
- File I/O operations
- Subprocess handling
- JSON parsing
- Error handling
- Async/await code
Added:
- TEST-TEMPLATES.md: Complete test templates for Python, Go, JavaScript
(easily adaptable to all 42+ languages)
- Test structure examples for each mode
- Python pytest, Go testing, Jest patterns
- Integration patterns for API validation
- Functional test scenarios
- Enhanced Makefile with multi-mode targets:
- make test-python: All 4 modes for Python
- make test-python-cli: Only CLI mode
- make test-python-library: Only Library mode
- make test-all-cli: CLI for all languages
- make test-all: All modes for all languages
- make test-integration-all: Cross-language API validation
How it works:
- Each client implementation tests as CLI AND library
- Integration validates auth, error codes, API contract
- Functional tests prove real-world usage works
- Makefile targets guide developers to create per-language tests
- CI can run all 4 modes or specific modes on changes
Add comprehensive testing infrastructure for UN clients:
1. Smart Change Detection (detect-changes.sh)
- Detects changes in BOTH root-level (un.py, un.go) AND clients/ directory
- Maps file extensions and directory names to languages
- Triggers test_all when infrastructure changes
2. Language-Specific CI Matrix (generate-matrix.sh compatible)
- Only runs tests for languages with changes
- Example: modify clients/python/ → pytest runs, Go/Ruby skipped
3. Testing Strategy Document (TESTING-STRATEGY.md)
- Complete testing matrix by language tier (compiled, interpreted, inception)
- Unit, integration, embedding, and parity tests
- Inception pattern for languages without local interpreters
- Common failures and fixes
- Rollout schedule for client/ migration
4. Makefile Targets
- 'make test-python', 'make test-go', etc. for local development
- 'make test-all' for comprehensive testing
- 'make test-integration' for API contract validation
- 'make test-ci-locally' to simulate CI pipeline
5. Updated CLAUDE.md
- Documents SDK architecture (in growth state)
- Explains three purposes: CLI, library, embeddable
- References TESTING-STRATEGY.md for details
This enables:
✓ Per-language testing (only run what changed)
✓ Local developer workflow (make test-LANG)
✓ 42+ language feature parity validation
✓ Cross-language integration testing
