CLPython - an implementation of Python in Common Lisp
CLPython is an open-source implementation of Python
written in Common Lisp.
With CLPython you can run Python programs in a Lisp environment. Libraries written
in Lisp
are available to Python code, and Python libraries can be accessed by Lisp code.
Requirements
CLPython runs successfully on the following platforms:
CLPython does not run yet on the following platforms::
There are dependencies on Closer to MOP and ptester.
Download
To get the source code from CVS:
- cvs -d :pserver:cvspublic@cvspublic.franz.com:/cvs-public login
- Password:cvspublic
- cvs -d :pserver:cvspublic@cvspublic.franz.com:/cvs-public checkout clpython
Install
To compile and load CLPython you need asdf: first create a link from the repository to file clpython.asd, then load the system using using (asdf:operate 'asdf:load-op :clpython).
Read the sections below on getting started with CLPython, and some technical details on the implementation.
To run the test suite, evaluate (asdf:operate 'asdf:test-op :clpython). The test result printed at the end should be the message "Errors detected in this test: 5" (which are the 5 or so known issues) and several hundred test successes. Unintended test errors will have the note "unexpected".
Mailing Lists
There are two mailing lists, both low-traffic:
- clpython-devel: for general discussion and bug reports;
- clpython-announce: for announcements of new releases.
About
CLPython is developed by Willem Broekema with support from Franz Inc. and released as open source under the LLGPL.
Completeness
Almost all Python language features are implemented, like generators, classes, metaclasses, modules, list comprehensions, and of course lambda. A few fairly new Python features introduced with Python 2.5 are missing:
- Absolute and relative imports: from .. import x
- Generators that use yield as expression: y = (yield x)
Some of the modules in the Python standard library are written in C; they have to be ported to Lisp before they can be used in CLPython. Not much time has been spent on this yet. Here's a rough overview of the porting progress:
Getting started with CLPython
| This is how to run a single Python file, e.g. the file b2.py from the Pie-thon benchmarks which calculates the digits of pi: |
clpython(1): (run #p"lib/parrotbench/b2.py") 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 2 3 8 4 6 2 6 4 ... |
| The same in a read-eval-print loop ("interpreter"): | clpython(2): (repl) [CLPython -- type `:q' to quit, `:help' for help] >>> import sys >>> sys.path.append("./lib/parrotbench/") >>> import b2 #<module `b2' from file #P"lib/parrotbench/b2.py"> >>> b2.main() 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 2 3 8 4 6 2 6 4 ... |
| You can enter arbitrary Python statements, including function and class definitions. The variables "_", "__" and "___" are bound to the last three results: |
>>> 1+2 3 >>> _ + 2 5 |
| You can enter arbitrary Lisp code at the prompt, like a function definition: | >>> (defun foo (x y) (+ x y)) clpython.app.repl::foo |
| But such definitions are not yet accessible to Python: | >>> foo Error: NameError: Variable `foo' is unbound. |
| Input that can be interpreted as both Python or Lisp code, like #'foo will normally be interpreted as a Python comment, but if a line starts with a space, it is always interpreted as Lisp code: | >>> #'foo #<Interpreted Function foo> |
| In that case also the variable "_" is set. That gives a way to call Lisp functions from within Python: |
>>> _ #<function foo @ #x11412672> >>> foo = _ >>> foo(1,2) 3 |
| >>> print "Have fun!" Have fun! |
CLPython - Some Technical Details
Python Object Representation
Python objects are represented by an equivalent Lisp value where possible, and as CLOS instances otherwise:
| Python data type | Representation in CLPython |
| Class | CLOS Class |
| Instance of user-defined class | CLOS instance |
| Function | Function |
| Dict | Hashtable |
| (Unicode) String | Unicode string |
| List | Adjustable vector |
| Tuple | Consed list |
| Long, Integer, Boolean | Integer |
| Complex | Complex |
| Float | Double-float |
| Complex | Complex |
Compiler
CLPython first translates Python code into an abstract syntax tree (AST), and then translates the AST into Lisp code. Most of the compilation work is carried out by macros:
if 4 > 3:
print 'y'
else:
print 'n'
...macroexpands into a cond form:
(cond
((py-val->lisp-bool (py-> 4 3))
(py-print nil (list "y") nil))
(t
(py-print nil (list "n") nil)))
...as defined by the macro:
(defmacro [if-stmt] (if-clauses else-clause)
`(cond ,@(loop for (cond body) in if-clauses
collect `((py-val->lisp-bool ,cond) ,body))
,@(when else-clause
`((t ,else-clause)))))
Note that the expansion contains calls to functions py-val->lisp-bool, py-> and py-print. These function are part of CLPython "runtime environment", and not part of the generated code. CLPython must be loaded every time this code is executed.
Compiler optimizations
Sometimes the generated Python code can be simplified because values of expressions are known at compile time. This is where compiler macros play a role. In the previous example, as 4 > 3 always holds, first the compiler macro for py-> replaces (py-> 4 3) by the Python value True. Then the compiler macro for py-val->lisp-bool sees True is a constant value, and replaces (py-val->lisp-bool True) by t. The Lisp compiler then deduces that always the first branch of the if expression is taken, and replace the whole (cond ...) by (py-print nil (list "y") nil), which is a call to the print function in the CLPython runtime environment. (Actually there is a compiler macro for py-print too, which also optimizes certain cases.)
In this example the compiler macros were able to remove a lot of the Lisp code at compile time. In practice there is often not that much that can be decided at compile time, due to Python as language being very dynamic. For example: in the expression 5 + x the value of x can be anything. As classes are able to redefine how the + operator behaves (with the __add__ and __radd__ methods), the value of 5 + x can be anything as well. Unless the context gives more information about the type of x, the Lisp code must contain a call to the generic addition function py-+.
Nevertheless, the compiler macro will inline "common" case, and make the generic call only for "uncommon" arguments. If small integers are common for the + operator, the compiler macro for py-+ could emit:
(if (typep x 'fixnum)
(+ 5 x)
(py-+ 5 x))
The check for x being fixnum is very fast, as is the addition in that case. If x is not a fixnum it could another kind of number, or even a Pythonic object simulating numeric behavior. The generic py-+ will handle those types, and raise an exception if addition fails.
Compiled vs. Interpreted Code
CLPython can run Python code in two Lisp modes, interpreted or compiled. In the latter case Lisp code is translated into assembly (or byte code, depending on the Lisp implementation). The advantage of interpreted code is that debugging is easier (the stack trace contains more information), but execution of compiled code is much faster. When a Python module is compiled, the functions are compiled into assembly code and written to an implementation-dependent fasl file.