diff --git a/README.md b/README.md index 19b42c5..37481a6 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # uncommonlisp -A Scheme-like Lisp interpreter in one Python file. +A Scheme interpreter in one Python file, with a bytecode compiler. ``` λ> (define (fib n) @@ -13,28 +13,59 @@ A Scheme-like Lisp interpreter in one Python file. ## Usage ```bash -python3 uncommonlisp.py # interactive REPL -python3 uncommonlisp.py script.lsp # run a file -python3 uncommonlisp.py -e '(+ 1 2)' # eval an expression +python3 uncommonlisp.py # interactive REPL +python3 uncommonlisp.py script.lsp # run a file +python3 uncommonlisp.py -e '(+ 1 2)' # eval an expression +python3 uncommonlisp.py --fast script.lsp # auto-compile (7-19x faster) ``` +## Bytecode compiler + +uncommonlisp includes a stack-based bytecode compiler and VM. Enable it with `--fast` or `(auto-compile! #t)`: + +```bash +python3 uncommonlisp.py --fast examples/fibonacci.lsp +``` + +```scheme +(auto-compile! #t) +(define (ack m n) + (cond ((= m 0) (+ n 1)) + ((= n 0) (ack (- m 1) 1)) + (else (ack (- m 1) (ack m (- n 1)))))) +(compiled? ack) ; => #t +(ack 3 4) ; => 125 +``` + +The compiler handles: `if`, `begin`, `and`, `or`, `when`, `unless`, `cond`, `define`, `set!`, `lambda`, `let`, named-let, `let*`, `letrec`, `do`, `call/cc`, function calls with tail-call optimization. Macros are expanded at compile time. 20 specialized opcodes for hot builtins (`+`, `-`, `*`, `=`, `<`, `car`, `cdr`, `cons`, `null?`, etc.) avoid function call overhead. + +Features: +- **Explicit frame stack** — compiled-to-compiled calls don't grow the Python stack +- **Full continuations** — `call/cc` supports upward continuations; generators work +- **Constant folding** — `(+ 1 2)` folds to `3` at compile time +- **Peephole optimizer** — eliminates dead code (VOID+POP, JUMP-to-next) +- **`(disassemble proc)`** — inspect generated bytecode + ## What's implemented **Core language** - Full lexical scoping and closures -- Tail-call optimization (TCO) via explicit loop — deep recursion never blows the stack +- Tail-call optimization (TCO) — deep recursion never blows the stack - Hygienic macros via `syntax-rules` with ellipsis (`...`) support -- `define-macro` / `defmacro` for procedural macro transformers -- `call/cc` (escape continuations) +- `define-macro` / `defmacro` for procedural macros +- `call/cc` — full continuations (escape + upward) in compiled code - `values` / `call-with-values` - `dynamic-wind`, `guard`, `with-exception-handler` - `quasiquote` / `unquote` / `unquote-splicing` with proper nesting -- R7RS internal defines → letrec* body semantics (mutual recursion in bodies) -- R7RS error objects — `error-object?` `error-object-message` `error-object-irritants` -- Exact rational arithmetic via `Fraction` — `(/ 1 3)` → `1/3`, `(+ 1/4 3/4)` → `1` -- String ports — `open-input-string` `open-output-string` `get-output-string` `read` on ports -- Module system — `module` / `import` with explicit export lists +- R7RS internal defines with letrec* body semantics +- R7RS error objects +- Exact rational arithmetic — `(/ 1 3)` → `1/3`, `(+ 1/4 3/4)` → `1` +- String ports — `open-input-string` `open-output-string` `read` on ports +- Mutable strings — `string-set!` `string-fill!` `string-copy!` +- Module system — `module` / `import` with export lists - `define-record-type` with `(inherit parent)` for single-inheritance +- Pretty-print — `pp` / `pretty-print` +- Tracing — `(trace fn)` / `(untrace fn)` **Special forms** `define` `set!` `lambda` `λ` `if` `cond` `case` `and` `or` `when` `unless` @@ -45,107 +76,70 @@ python3 uncommonlisp.py -e '(+ 1 2)' # eval an expression `module` `import` `define-record-type` **Built-ins** -- Arithmetic: `+` `-` `*` `/` `quotient` `remainder` `modulo` `expt` `sqrt` `abs` `floor` `ceiling` `round` `truncate` `min` `max` `gcd` `lcm` `log` `exp` `sin` `cos` `tan` `atan` `numerator` `denominator` and more -- Rationals: `(/ 1 3)` → `1/3`, literal `1/3` syntax, exact arithmetic, `exact` / `inexact` conversion +- Arithmetic: `+` `-` `*` `/` `quotient` `remainder` `modulo` `expt` `sqrt` `abs` `floor` `ceiling` `round` `truncate` `min` `max` `gcd` `lcm` `log` `exp` trig functions, `numerator` `denominator` +- Rationals: `(/ 1 3)` → `1/3`, literal `1/3` syntax, `exact` / `inexact` conversion - Comparison: `=` `<` `>` `<=` `>=` `zero?` `positive?` `negative?` `odd?` `even?` -- Booleans: `not` `boolean?` `boolean=?` -- Equality: `eq?` `eqv?` `equal?` -- Pairs & lists: `cons` `car` `cdr` `set-car!` `set-cdr!` `list` `list*` `length` `append` `reverse` `list-ref` `list-tail` `memq` `memv` `member` `assq` `assv` `assoc` `iota` `map` `for-each` `filter` `fold-left` `fold-right` `reduce` `any` `every` `count` `flat-map` `sort` `sort-by` `partition` `find` `take` `drop` `take-while` `drop-while` `take-right` `drop-right` `zip` `flatten` `concatenate` `list-tabulate` `unfold` and more -- SRFI-1: `last` `first`–`fifth` `delete` `lset-union` `lset-intersection` `lset-difference` `proper-list?` `dotted-list?` -- Strings: `string-length` `string-ref` `substring` `string-append` `string-upcase` `string-downcase` `string->list` `list->string` `string->symbol` `symbol->string` `string->number` (with `#x`/`0x`/`#b`/`#o` prefix detection) `number->string` `string-contains` `string-split` `string-join` `string-trim` `string-replace` `format` and more -- Characters: `char->integer` `integer->char` `char-alphabetic?` `char-numeric?` `char-upcase` `char-downcase` -- Vectors: `make-vector` `vector` `vector-ref` `vector-set!` `vector->list` `list->vector` -- Hash tables: `make-hash-table` `hash-table-set!` `hash-table-ref` `hash-table-ref/default` `hash-table-delete!` `hash-table-exists?` `hash-table-keys` `hash-table-values` `hash-table->alist` `hash-table-walk` and more -- Type predicates: `number?` `integer?` `real?` `rational?` `string?` `symbol?` `pair?` `null?` `list?` `char?` `vector?` `boolean?` `procedure?` `exact?` `inexact?` -- I/O: `display` `write` `newline` `read` `read-char` `peek-char` `read-line` `load` `with-output-to-string` `open-input-string` `open-output-string` `get-output-string` -- Error objects: `error-object?` `error-object-message` `error-object-irritants` -- Pretty-print: `pp` / `pretty-print` — indented output respecting line width +- Pairs & lists: `cons` `car` `cdr` `list` `length` `append` `reverse` `map` `for-each` `filter` `fold-left` `fold-right` `reduce` `any` `every` `sort` `partition` `find` `take` `drop` `zip` `flatten` and more +- SRFI-1: `last` `first`–`fifth` `delete` `lset-union` `lset-intersection` `lset-difference` `unfold` `list-tabulate` +- Strings: `string-length` `string-ref` `string-set!` `substring` `string-append` `string-copy` `string-copy!` `string-fill!` `string->list` `string->number` `format` and more +- Characters: `char->integer` `integer->char` `char-alphabetic?` `char-upcase` `char-downcase` +- Vectors: `make-vector` `vector` `vector-ref` `vector-set!` `vector-copy` `vector-copy!` +- Hash tables: `make-hash-table` `hash-table-set!` `hash-table-ref` `hash-table-keys` `hash-table-values` `hash-table-walk` and more +- I/O: `display` `write` `newline` `read` `read-char` `read-line` `open-input-string` `open-output-string` `with-output-to-string` +- File system: `file-exists?` `delete-file` `rename-file` `directory-files` `current-directory` +- System: `command-line` `get-environment-variable` `current-time` `exit` - Python interop: `py-eval` `py-exec` `py-import` `py-call` `py-attr` +- Compiler: `compile` `compiled?` `disassemble` `auto-compile!` -**Prelude** (loaded automatically) -`when` `unless` `case` `while` `for` `define-record-type` (with inheritance) `1+` `1-` `add1` `sub1` `square` `cube` `compose` `atom?` `range` `flatten` `string-map` `string-for-each` `call-with-string-output-port` - -**Standard library** (`stdlib.lsp`, load explicitly) -Syntax-rules versions of `let`/`and`/`or`/`cond`/`case`/`do`, `fluid-let`, `receive` (SRFI-8), `begin0`, `while`/`until`, `dotimes`/`dolist`, `push!`/`pop!`, `and-let*` (SRFI-2), string utilities, list utilities (`sum` `product` `maximum` `minimum` `average` `enumerate` `transpose` `chunks` `interleave`), numeric utilities (`factorial` `fib` `prime?` `primes-up-to` `clamp`), alist/hash utilities, tree utilities, simple object system, coroutines via `call/cc` +**Standard library** (`stdlib.lsp`) +Additional macros, string/list/numeric/tree utilities, alist/hash helpers, simple object system, SRFI-2/8/64 test framework. ## Examples +```bash +python3 uncommonlisp.py --fast examples/fibonacci.lsp +python3 uncommonlisp.py --fast examples/generator.lsp +python3 uncommonlisp.py --fast examples/mergesort.lsp +python3 uncommonlisp.py examples/objects.lsp +``` + ```scheme -; Closures -(define (make-counter) - (let ((n 0)) - (lambda () (set! n (+ n 1)) n))) +;; Generator using full continuations +(auto-compile! #t) +(define (make-gen thunk) + (let ((k #f) (done #f)) + (lambda () + (if done 'done + (call/cc (lambda (return) + (if k (k return) + (begin (thunk (lambda (val) + (call/cc (lambda (next) + (set! k next) (return val))))) + (set! done #t) (return 'done))))))))) -(define c (make-counter)) -(c) ; => 1 -(c) ; => 2 - -; Hygienic macro (syntax-rules) -(define-syntax my-or - (syntax-rules () - ((my-or) #f) - ((my-or e) e) - ((my-or e1 e2 ...) - (let ((t e1)) - (if t t (my-or e2 ...)))))) - -; define-record-type -(define-record-type point - (make-point x y) - point? - (x point-x) - (y point-y set-point-y!)) - -(define p (make-point 3 4)) -(point-x p) ; => 3 - -; Named let (looping) -(let loop ((i 0) (acc '())) - (if (= i 5) - (reverse acc) - (loop (+ i 1) (cons (* i i) acc)))) -; => (0 1 4 9 16) - -; Hash tables -(define freq - (let ((h (make-hash-table))) - (for-each (lambda (x) - (hash-table-set! h x (+ 1 (hash-table-ref/default h x 0)))) - '(a b a c b a)) - h)) -(hash-table-ref freq 'a) ; => 3 - -; Tail calls — no stack overflow even at depth 1,000,000 -(define (count-down n) - (if (= n 0) 'done (count-down (- n 1)))) -(count-down 1000000) ; => done - -; Python interop -(define re (py-import "re")) -(py-call (py-attr re 'findall) "[0-9]+" "abc123def456") -; => ["123", "456"] +(define counter (make-gen (lambda (yield) + (let loop ((i 0)) (yield i) (loop (+ i 1)))))) +(counter) ; => 0 +(counter) ; => 1 +(counter) ; => 2 ``` -## Running tests +## Running tests & benchmarks ```bash -make test # run 455 tests +make test # run 529 tests make test-verbose # verbose output -``` - -## Running benchmarks - -```bash -python3 bench.py # compare against CPython baseline -python3 bench.py -v # show result values too +make bench # compare interpreter vs bytecode vs CPython +make lint # syntax check all Python files ``` ## File layout ``` -uncommonlisp.py interpreter (self-contained, one file) -stdlib.lsp extended standard library (load manually) -tests.py test suite (396 tests) +uncommonlisp.py interpreter + bytecode compiler (one file, ~2600 lines) +stdlib.lsp extended standard library +tests.py test suite (529 tests) bench.py benchmarks vs CPython -Makefile make test / make repl +examples/ example programs +Makefile make test / make bench / make repl ``` diff --git a/examples/fibonacci.lsp b/examples/fibonacci.lsp new file mode 100644 index 0000000..142117a --- /dev/null +++ b/examples/fibonacci.lsp @@ -0,0 +1,21 @@ +;;; fibonacci.lsp — iterative and recursive Fibonacci +;;; Run: python3 uncommonlisp.py --fast examples/fibonacci.lsp + +(define (fib-iter n) + (let loop ((a 0) (b 1) (i 0)) + (if (= i n) a (loop b (+ a b) (+ i 1))))) + +(define (fib-rec n) + (if (<= n 1) n (+ (fib-rec (- n 1)) (fib-rec (- n 2))))) + +(display "fib(30) iterative: ") +(display (fib-iter 30)) +(newline) + +(display "fib(20) recursive: ") +(display (fib-rec 20)) +(newline) + +(display "First 15 Fibonacci numbers: ") +(display (map fib-iter (iota 15))) +(newline) diff --git a/examples/generator.lsp b/examples/generator.lsp new file mode 100644 index 0000000..72adb09 --- /dev/null +++ b/examples/generator.lsp @@ -0,0 +1,50 @@ +;;; generator.lsp — coroutine-style generators using full continuations +;;; Run: python3 uncommonlisp.py --fast examples/generator.lsp + +(define (make-generator thunk) + (let ((k #f) (done #f)) + (lambda () + (if done 'done + (call/cc + (lambda (return) + (if k + (k return) + (begin + (thunk (lambda (val) + (call/cc (lambda (next) + (set! k next) + (return val))))) + (set! done #t) + (return 'done))))))))) + +;; Range generator +(define (range-gen start end) + (make-generator + (lambda (yield) + (let loop ((i start)) + (when (< i end) + (yield i) + (loop (+ i 1))))))) + +(define gen (range-gen 0 5)) +(display "Generator: ") +(let loop () + (let ((v (gen))) + (unless (eq? v 'done) + (display v) (display " ") + (loop)))) +(newline) + +;; Fibonacci generator (infinite) +(define (fib-gen) + (make-generator + (lambda (yield) + (let loop ((a 0) (b 1)) + (yield a) + (loop b (+ a b)))))) + +(define fibs (fib-gen)) +(display "First 10 fibs: ") +(do ((i 0 (+ i 1))) ((= i 10)) + (display (fibs)) (display " ")) +(newline) diff --git a/examples/mergesort.lsp b/examples/mergesort.lsp new file mode 100644 index 0000000..99672f3 --- /dev/null +++ b/examples/mergesort.lsp @@ -0,0 +1,27 @@ +;;; mergesort.lsp — functional merge sort +;;; Run: python3 uncommonlisp.py --fast examples/mergesort.lsp + +(define (merge a b) + (cond ((null? a) b) + ((null? b) a) + ((< (car a) (car b)) + (cons (car a) (merge (cdr a) b))) + (else + (cons (car b) (merge a (cdr b)))))) + +(define (split lst) + (let loop ((l lst) (a '()) (b '())) + (if (null? l) + (list a b) + (loop (cdr l) b (cons (car l) a))))) + +(define (msort lst) + (if (or (null? lst) (null? (cdr lst))) + lst + (let ((halves (split lst))) + (merge (msort (car halves)) + (msort (cadr halves)))))) + +(define data (reverse (iota 20))) +(display "Input: ") (display data) (newline) +(display "Sorted: ") (display (msort data)) (newline) diff --git a/examples/objects.lsp b/examples/objects.lsp new file mode 100644 index 0000000..8ff9409 --- /dev/null +++ b/examples/objects.lsp @@ -0,0 +1,49 @@ +;;; objects.lsp — record types, closures as objects, hash tables +;;; Run: python3 uncommonlisp.py examples/objects.lsp + +;; Record types with inheritance +(define-record-type + (make-point x y) + point? + (x point-x) + (y point-y)) + +(define-record-type + (inherit ) + (make-point3d x y z) + point3d? + (x point3d-x) + (y point3d-y) + (z point3d-z)) + +(define p (make-point 3 4)) +(define p3 (make-point3d 1 2 3)) + +(display "point: ") (display (point-x p)) (display ",") (display (point-y p)) (newline) +(display "point3d: ") (display (point3d-x p3)) (display ",") + (display (point3d-y p3)) (display ",") (display (point3d-z p3)) (newline) +(display "point3d is point? ") (display (point? p3)) (newline) + +;; Closure-based counter +(define (make-counter) + (let ((n 0)) + (lambda (msg . args) + (cond ((eq? msg 'inc) (set! n (+ n 1)) n) + ((eq? msg 'dec) (set! n (- n 1)) n) + ((eq? msg 'get) n) + ((eq? msg 'reset) (set! n 0) n) + (else (error "unknown message" msg)))))) + +(define c (make-counter)) +(c 'inc) (c 'inc) (c 'inc) +(display "Counter after 3 increments: ") (display (c 'get)) (newline) +(c 'dec) +(display "Counter after decrement: ") (display (c 'get)) (newline) + +;; Hash table +(define ht (make-hash-table)) +(hash-table-set! ht 'name "uncommonlisp") +(hash-table-set! ht 'version "1.0.0") +(hash-table-set! ht 'features '(bytecode continuations macros)) +(display "Name: ") (display (hash-table-ref ht 'name)) (newline) +(display "Features: ") (display (hash-table-ref ht 'features)) (newline) diff --git a/uncommonlisp.py b/uncommonlisp.py index 5328bf8..0a7d71a 100644 --- a/uncommonlisp.py +++ b/uncommonlisp.py @@ -1457,9 +1457,64 @@ def _bc_lambda(body, params, rest, env, name=None): for nm in def_names: inner.emit(OP_VOID); inner.emit(OP_BIND, nm) _bc_body(body_list, inner, env, tail=True) inner.emit(OP_RETURN) + _peephole(inner) return inner +_JUMP_OPS = frozenset([OP_JUMP, OP_JUMP_IF_FALSE, OP_JUMP_IF_FALSE_KEEP, + OP_JUMP_IF_TRUE_KEEP]) + +def _peephole(code): + """Peephole optimization: eliminate dead code and redundant ops.""" + instrs = code.instrs + n = len(instrs) + if n < 2: return + # Mark instructions to remove + remove = set() + for i in range(n - 1): + op, arg = instrs[i] + nop, _ = instrs[i + 1] + # VOID POP → remove both + if op == OP_VOID and nop == OP_POP: + remove.add(i); remove.add(i + 1) + # Dead code after RETURN (unless it's a jump target) + if op == OP_RETURN and nop not in (OP_RETURN,) and i + 1 not in _jump_targets(instrs): + # Only remove if next instruction is not a jump target + if nop not in (OP_PUSH_ENV, OP_POP_ENV): # be conservative + pass # skip for safety — jump target analysis is complex + # JUMP to next instruction → remove + for i in range(n): + op, arg = instrs[i] + if op == OP_JUMP and arg == i + 1: + remove.add(i) + if not remove: return + # Build index mapping: old → new + mapping = {}; new_idx = 0 + for i in range(n): + mapping[i] = new_idx + if i not in remove: new_idx += 1 + mapping[n] = new_idx # for jumps pointing past the end + # Rebuild with adjusted jumps + new_instrs = [] + for i in range(n): + if i in remove: continue + op, arg = instrs[i] + if op in _JUMP_OPS and isinstance(arg, int): + new_instrs.append((op, mapping.get(arg, arg))) + else: + new_instrs.append((op, arg)) + code.instrs = new_instrs + + +def _jump_targets(instrs): + """Return set of instruction indices that are jump targets.""" + targets = set() + for op, arg in instrs: + if op in _JUMP_OPS and isinstance(arg, int): + targets.add(arg) + return targets + + class _ContInvoked(Exception): """Raised when a full continuation is invoked.""" __slots__ = ('cont', 'val') @@ -2609,6 +2664,28 @@ def main(): args = sys.argv[1:] + # --help / -h + if '--help' in args or '-h' in args: + print('''uncommonlisp — a Scheme in one Python file + +Usage: uncommonlisp [options] [script.lsp] [args...] + uncommonlisp -e '(+ 1 2)' + uncommonlisp (interactive REPL) + +Options: + -e EXPR evaluate expression and print result + -f, --fast auto-compile all defines (bytecode VM, 7-19x faster) + -h, --help show this help + -v, --version show version + +Features: R7RS core, bytecode compiler, full continuations, macros, + syntax-rules, modules, rationals, string ports, SRFI-1/2/8/64.''') + return + + # --version / -v + if '--version' in args or '-v' in args: + print(f'uncommonlisp 1.0.0'); return + # --fast / -f: enable auto-compile if '--fast' in args or '-f' in args: _auto_compile[0] = True