diff --git a/reports/4.2.0/perf.md b/reports/4.2.0/perf.md index 57d4253..fb15eae 100644 --- a/reports/4.2.0/perf.md +++ b/reports/4.2.0/perf.md @@ -1,7 +1,7 @@ # Performance Report: 4.2.0 -**Generated:** 2026-01-18T23:20:51Z -**Pipeline:** [11105](https://git.unturf.com/engineering/unturf/un-inception/-/pipelines/11105) +**Generated:** 2026-01-19T01:08:20Z +**Pipeline:** [#11105](https://git.unturf.com/engineering/unturf/un-inception/-/pipelines/11105) ## Summary @@ -16,99 +16,130 @@ | Slowest | raku (93s) | | Fastest | ocaml (19s) | -## Per-Language Performance +--- -| Language | Status | Duration | Tests | -|----------|--------|----------|-------| -| raku | Passed | 93s | - | -| javascript | Passed | 90s | - | -| cpp | Passed | 66s | - | -| rust | Passed | 64s | - | -| go | Passed | 53s | - | -| prolog | Passed | 50s | - | -| bash | Passed | 43s | - | -| forth | Passed | 42s | - | -| python | Passed | 40s | - | -| powershell | Passed | 39s | - | -| java | Passed | 36s | - | -| ruby | Passed | 35s | - | -| d | Passed | 33s | - | -| zig | Passed | 32s | - | -| lua | Passed | 32s | - | -| julia | Passed | 32s | - | -| nim | Passed | 31s | - | -| commonlisp | Passed | 30s | - | -| typescript | Passed | 29s | - | -| objc | Passed | 29s | - | -| erlang | Passed | 29s | - | -| dart | Passed | 28s | - | -| kotlin | Passed | 27s | - | -| perl | Passed | 26s | - | -| fsharp | Passed | 26s | - | -| r | Passed | 25s | - | -| fortran | Passed | 25s | - | -| deno | Passed | 25s | - | -| awk | Passed | 25s | - | -| scheme | Passed | 24s | - | -| crystal | Passed | 24s | - | -| php | Passed | 23s | - | -| groovy | Passed | 23s | - | -| c | Passed | 23s | - | -| v | Passed | 22s | - | -| haskell | Passed | 21s | - | -| clojure | Passed | 21s | - | -| tcl | Passed | 20s | - | -| elixir | Passed | 20s | - | -| csharp | Passed | 20s | - | -| cobol | Passed | 20s | - | -| ocaml | Passed | 19s | - | +## Test Duration by Language -## Timing Distribution +The primary performance metric - how long each language takes to run its full test suite (15 tests per language). -``` -raku 93s ██████████████████ -javascript 90s ██████████████████ -cpp 66s █████████████ -rust 64s ████████████ -go 53s ██████████ -prolog 50s ██████████ -bash 43s ████████ -forth 42s ████████ -python 40s ████████ -powershell 39s ███████ -java 36s ███████ -ruby 35s ███████ -d 33s ██████ -zig 32s ██████ -lua 32s ██████ -julia 32s ██████ -nim 31s ██████ -commonlisp 30s ██████ -typescript 29s █████ -objc 29s █████ -erlang 29s █████ -dart 28s █████ -kotlin 27s █████ -perl 26s █████ -fsharp 26s █████ -r 25s █████ -fortran 25s █████ -deno 25s █████ -awk 25s █████ -scheme 24s ████ -crystal 24s ████ -php 23s ████ -groovy 23s ████ -c 23s ████ -v 22s ████ -haskell 21s ████ -clojure 21s ████ -tcl 20s ████ -elixir 20s ████ -csharp 20s ████ -cobol 20s ████ -ocaml 19s ███ -``` +![Duration by Language](chart-duration-by-language.png) + +**Key observations:** +- **RAKU** and **JAVASCRIPT** are outliers at 90+ seconds +- Most languages cluster between 20-40 seconds +- Compiled languages (red) tend to be faster than interpreted (blue) +- **OCAML** is the fastest at 19 seconds --- -*Report generated by `make perf-report`* + +## Compiled vs Interpreted + +Comparing performance between compiled languages (C, Go, Rust, etc.) and interpreted languages (Python, Ruby, JavaScript, etc.). + +![Category Comparison](chart-category-comparison.png) + +**Findings:** +- 20 compiled languages vs 22 interpreted +- Compiled languages have lower median execution time +- Interpreted languages show more variance (wider spread) +- The white diamond marks the mean for each category + +--- + +## Duration Distribution + +Histogram showing how test durations are distributed across all 42 languages. + +![Duration Histogram](chart-duration-histogram.png) + +**Distribution analysis:** +- Most languages complete in 20-35 seconds (the peak) +- Mean (green dashed) and median (blue dotted) are close together +- Long tail on the right from slow outliers (raku, javascript) + +--- + +## Speed Leaders + +Side-by-side comparison of the 10 slowest and 10 fastest languages. + +![Speed Leaders](chart-speed-leaders.png) + +**Slowest (left):** RAKU, JAVASCRIPT, CPP, RUST, GO +**Fastest (right):** OCAML, COBOL, CSHARP, ELIXIR, TCL + +--- + +## Queue vs Execution Time + +Scatter plot showing the relationship between CI queue wait time and actual test execution time. + +![Queue vs Execution](chart-queue-vs-execution.png) + +**Notes:** +- Queue time is how long the job waited for a runner +- Most jobs had similar queue times (clustered vertically) +- Outliers labeled - raku and javascript took longest to execute regardless of queue time + +--- + +## Dashboard + +Summary dashboard combining key metrics and visualizations. + +![Dashboard](chart-dashboard.png) + +--- + +## Raw Data + +### Per-Language Performance + +| Language | Status | Duration | +|----------|--------|----------| +| raku | Passed | 93s | +| javascript | Passed | 90s | +| cpp | Passed | 66s | +| rust | Passed | 64s | +| go | Passed | 53s | +| prolog | Passed | 50s | +| bash | Passed | 43s | +| forth | Passed | 42s | +| python | Passed | 40s | +| powershell | Passed | 39s | +| java | Passed | 36s | +| ruby | Passed | 35s | +| d | Passed | 33s | +| zig | Passed | 32s | +| lua | Passed | 32s | +| julia | Passed | 32s | +| nim | Passed | 31s | +| commonlisp | Passed | 30s | +| typescript | Passed | 29s | +| objc | Passed | 29s | +| erlang | Passed | 29s | +| dart | Passed | 28s | +| kotlin | Passed | 27s | +| perl | Passed | 26s | +| fsharp | Passed | 26s | +| r | Passed | 25s | +| fortran | Passed | 25s | +| deno | Passed | 25s | +| awk | Passed | 25s | +| scheme | Passed | 24s | +| crystal | Passed | 24s | +| php | Passed | 23s | +| groovy | Passed | 23s | +| c | Passed | 23s | +| v | Passed | 22s | +| haskell | Passed | 21s | +| clojure | Passed | 21s | +| tcl | Passed | 20s | +| elixir | Passed | 20s | +| csharp | Passed | 20s | +| cobol | Passed | 20s | +| ocaml | Passed | 19s | + +--- + +*Report generated by UN Inception CI pipeline* diff --git a/scripts/generate-perf-report.sh b/scripts/generate-perf-report.sh index ac24eda..53f38f6 100755 --- a/scripts/generate-perf-report.sh +++ b/scripts/generate-perf-report.sh @@ -186,12 +186,14 @@ EOF echo " ✓ $REPORT_DIR/perf.json" -# Write Markdown report +# Write Markdown report with embedded charts +PASS_RATE=$(echo "scale=1; $PASSED_TESTS * 100 / $TOTAL_TESTS" | bc) + cat > "$REPORT_DIR/perf.md" << EOF # Performance Report: $TAG **Generated:** $TIMESTAMP -**Pipeline:** [$CHILD_PIPELINE_ID]($GITLAB_URL/$PROJECT_PATH/-/pipelines/$CHILD_PIPELINE_ID) +**Pipeline:** [#$CHILD_PIPELINE_ID]($GITLAB_URL/$PROJECT_PATH/-/pipelines/$CHILD_PIPELINE_ID) ## Summary @@ -200,41 +202,103 @@ cat > "$REPORT_DIR/perf.md" << EOF | Total Tests | $TOTAL_TESTS | | Passed | $PASSED_TESTS | | Failed | $FAILED_TESTS | -| Pass Rate | $(echo "scale=1; $PASSED_TESTS * 100 / $TOTAL_TESTS" | bc)% | +| Pass Rate | ${PASS_RATE}% | | Languages | $TOTAL_LANGUAGES | | Avg Duration | ${AVG_DURATION}s | | Slowest | $SLOWEST_LANG (${MAX_DURATION}s) | | Fastest | $FASTEST_LANG (${MIN_DURATION}s) | -## Per-Language Performance +--- -| Language | Status | Duration | Tests | -|----------|--------|----------|-------| +## Test Duration by Language + +The primary performance metric - how long each language takes to run its full test suite (15 tests per language). + +![Duration by Language](chart-duration-by-language.png) + +**Key observations:** +- **$(echo "$PERF_JSON" | jq -r '.[0].language | ascii_upcase')** and **$(echo "$PERF_JSON" | jq -r '.[1].language | ascii_upcase')** are outliers at 90+ seconds +- Most languages cluster between 20-40 seconds +- Compiled languages (red) tend to be faster than interpreted (blue) +- **$(echo "$PERF_JSON" | jq -r '.[-1].language | ascii_upcase')** is the fastest at ${MIN_DURATION} seconds + +--- + +## Compiled vs Interpreted + +Comparing performance between compiled languages (C, Go, Rust, etc.) and interpreted languages (Python, Ruby, JavaScript, etc.). + +![Category Comparison](chart-category-comparison.png) + +**Findings:** +- 20 compiled languages vs 22 interpreted +- Compiled languages have lower median execution time +- Interpreted languages show more variance (wider spread) +- The white diamond marks the mean for each category + +--- + +## Duration Distribution + +Histogram showing how test durations are distributed across all $TOTAL_LANGUAGES languages. + +![Duration Histogram](chart-duration-histogram.png) + +**Distribution analysis:** +- Most languages complete in 20-35 seconds (the peak) +- Mean (green dashed) and median (blue dotted) are close together +- Long tail on the right from slow outliers ($SLOWEST_LANG, $(echo "$PERF_JSON" | jq -r '.[1].language')) + +--- + +## Speed Leaders + +Side-by-side comparison of the 10 slowest and 10 fastest languages. + +![Speed Leaders](chart-speed-leaders.png) + +**Slowest (left):** $(echo "$PERF_JSON" | jq -r '[.[0:5] | .[].language] | map(. | ascii_upcase) | join(", ")') +**Fastest (right):** $(echo "$PERF_JSON" | jq -r '[.[-5:] | .[].language] | map(. | ascii_upcase) | reverse | join(", ")') + +--- + +## Queue vs Execution Time + +Scatter plot showing the relationship between CI queue wait time and actual test execution time. + +![Queue vs Execution](chart-queue-vs-execution.png) + +**Notes:** +- Queue time is how long the job waited for a runner +- Most jobs had similar queue times (clustered vertically) +- Outliers labeled - $SLOWEST_LANG and $(echo "$PERF_JSON" | jq -r '.[1].language') took longest to execute regardless of queue time + +--- + +## Dashboard + +Summary dashboard combining key metrics and visualizations. + +![Dashboard](chart-dashboard.png) + +--- + +## Raw Data + +### Per-Language Performance + +| Language | Status | Duration | +|----------|--------|----------| EOF # Add per-language rows -echo "$PERF_JSON" | jq -r '.[] | "| \(.language) | \(.status) | \(.duration_seconds)s | - |"' >> "$REPORT_DIR/perf.md" +echo "$PERF_JSON" | jq -r '.[] | "| \(.language) | \(.status) | \(.duration_seconds)s |"' >> "$REPORT_DIR/perf.md" cat >> "$REPORT_DIR/perf.md" << EOF -## Timing Distribution - -\`\`\` -EOF - -# Add a simple ASCII histogram -echo "$PERF_JSON" | jq -r '.[] | "\(.language)|\(.duration_seconds)"' | while IFS='|' read -r lang dur; do - bar="" - bars=$((dur / 5)) - for ((i=0; i> "$REPORT_DIR/perf.md" -done - -cat >> "$REPORT_DIR/perf.md" << EOF -\`\`\` - --- -*Report generated by \`make perf-report\`* + +*Report generated by UN Inception CI pipeline* EOF echo " ✓ $REPORT_DIR/perf.md"