Copyright © 2002 Pascal Costanza. All rights reserved. Permission to copy, transmit, and store this work, unmodified and in its entirety, is granted.
You can use this document as an online reference or print it out. All links are represented as explicit URLs that will be retained when printed. Please send any kind of feedback to costanza@web.de.
My current situation is as follows. I have worked with the Java programming language for the last seven years, mainly in projects in which extensions to the Java programming language have been developed. Before that I have mainly used languages from the Wirth family (mainly Modula-2 and Oberon), so in the beginning I was quite happy with some of the advantages that Java has over the latter languages.
During the past year I have come to realize that Java is still a very (indeed extremely) limited programming language, so I started to look at alternatives. Because of my involvement with the Feyerabend project (http://www.dreamsongs.com/Feyerabend/Feyerabend.html), Lisp naturally came about as one of those alternatives. (Richard Gabriel started the Feyerabend project, and he was also one of the people that kicked off the Common Lisp standardization in the beginning of the 1980's.)
Although there are a lot of nice programming languages available (for example, in alphabetical order: gbeta, Objective CAML, Python, Ruby), I have quickly got the impression that Lisp is, in some sense, the mother of all languages. The main reason for this statement is that Lisp includes a complete theory of computation by treating code and data uniformly. (This is more powerful than "just" being Turing-complete. For more information, see the section on the "Technical Background of Lisp" below.) This effectively means that there are no (conceptual) restrictions whatsoever in Lisp: if you can do something in any one language, you can also do it in Lisp. Furthermore, Lisp checks all types at runtime, so no static type system can get in your way.
You can rightfully generalize these observations and state the following: the mindset of Lisp asserts that expressive power is the single most important property of a programming language. Nothing should get in your way when you want to use that power. A programming language should not impose its view of the world on the programmer. It is the programmer who should be able to adapt the language to his/her needs, and not the other way around.
This has a great appeal to me and so I have decided that Lisp is the language of my choice. However, there is a drawback when you decide to start looking at Lisp more seriously: there are no really good introductions into the language available on the net that suited my needs - or at least I haven't found them. Of course, there are some introductions available, but either they are too low-level and try to teach programming for the very beginners, or they are too academic and only touch theoretically interesting topics. What I would have liked to see is an introduction that provides you with enough background information in order to understand the concepts and then gets you going as quickly as possible. (Something like "Lisp for experienced programmers".)
Because I haven't found anything I have had to work through the available stuff on my own. Now, I would like to present a summary of what I have found out in order to ease things for other people.
Why is this introduction called "highly opinionated"? Lisp is a complex language, or rather a family of languages. There cannot be "the one definitive guide" to Lisp. What I present here is what I would have liked to have seen in the first place. Other people probably prefer a different approach. I don't want to write an introduction that provides everything for everyone. I only want to present information that is useful for people like me. At least, this is the material that worked for me, so it might suit other people as well.
Especially, this is not something like "Common Lisp for Dummies in 21 days". You may need several iterations of the material presented here or elsewhere until you actually get the feel of the "Common Lisp experience". (...but I don't have to stress that you need a lot more than 21 days to master any serious language, right? ;)
Lisp has a long history. It was invented in the 1950's and has continually developed and been improved over the decades. At several stages there have been various dialects of Lisp, so Lisp is indeed not a single language but rather a family of languages. (If you think of languages like C, C++, Objective C, Java and C# as being languages of the same "C-like" family of languages, then you'll be quite close.)
In the 1980's and 1990's, two main dialects established themselves as the only widely available and usable dialects: Common Lisp and Scheme. Scheme was invented in the 1970's by Gerald Jay Sussman and Guy L. Steele as a result of trying to understand object-orientation and incorporate it into Lisp. Scheme essentially introduced lexical closures (as a generalization of objects) and continuations (as a generalization of function calls and message passing) at the conceptual level, and "extreme" optimization of tail calls and tail recursions at the technical level. The nice thing about Scheme is that it is a beautiful diamond with properties that appeal mainly to academics who are looking for truth and beauty (i.e. minimality and orthogonality). However, it misses many practical things that you need in every-day programming and that would spoil the conceptual beauty.
See the following links if you are interested in explanations for some of the terms mentioned above.
Lexical scope, dynamic scope, (lexical) closures: http://c2.com/cgi/wiki?ScopeAndClosures
Continuations: http://c2.com/cgi/wiki?CallWithCurrentContinuation
Tail calls, tail recursion: http://c2.com/cgi/wiki?TailCallOptimization
At the end of the 1970's, several variants of Lisp were in use. Common Lisp started as a remedy for an exaggerated diversification - a unified version of Lisp with the goal of integrating the advantages of the several Lisps that existed at this time and avoiding the disadvantages. Furthermore, Common Lisp was a big step forward because it also incorporated the lessons learned from Scheme. (Lexical scoping made its way into Common Lisp. Continuations were not included because they turned out to be too complicated for practical use. Optimization of tail recursions didn't need to be standardized because it was a natural progression to use it as an implementation technique.)
So this gives a hint of what to look for when you consider using Lisp today. If you want to do practical "real-world" things you should choose Common Lisp. If you mainly want to do academic research and, for example, want to experiment with continuations, you could also opt for Scheme. Both languages are still in use today and have a strong support from their respective user communities.
I have decided to use Common Lisp for several reasons. Some of them will pop out during the course of this introduction. What I don't like about Scheme is that it is a "right-thing" language that has strong notions about how things should be done. Common Lisp seems to be far more flexible.
However, many publications and descriptions of Scheme also make sense for Common Lisp - for example, descriptions of what lexical closures are - so I have nevertheless included some references to Scheme in the following sections.
(In essence, my message is as follows: it doesn't really matter if you choose Common Lisp or Scheme, it only depends on your specific needs. So just let your gut feelings guide you. In fact, both languages are flexible enough to be used both in research and in the "real world". However, the bulk of my introduction deals with Common Lisp, so if you opt for Scheme you have to look for other places to find introductory material. See the links section below for some pointers.)
A note on how to work through this material: I have tried to make it natural to read this text sequentially. Of course, you are free to skip sections and especially switch back and forth between part I and II. However, some of the later sections require you to have been exposed at least to some example Lisp code. The text includes enough pointers for that purpose.
A good one-page overview of what Common Lisp offers can be found at http://www.lisp.org/table/objects.htm. This description focusses on object-oriented features of CLOS (the Common Lisp Object System that is part of Common Lisp) but also gives a good summary of the functional and imperative features of Common Lisp.
The Common Lisp Language Overview by Xanalys is another good summary to be found at http://www.lispworks.com/products/lisp-overview.html.
Lisp's history is different from that of other programming languages, especially those of the Algol and C family. Therefore, the Lisp community has developed different notions and terminology at various levels. Here are some clarifications to avoid confusion.
Some of Lisp's built-in functions have seemingly funny names. For example, "car" is used to take the first element of a list, "cdr" to take the remaining (rest) elements. Lists can be built by the use of "cons", "mapcar" is used to iterate over a list, and so on. In particular, members of the Algol- and Wirth-family of languages usually question why the Lisp designers have not chosen to use more "natural" names, like "first", "rest", "foreach", and so on. At least, this was one of my first reactions to being exposed to these names (and I am obviously not the only one).
Please forget about your preconceptions of naturalness - Lisp simply has a different culture, mainly for historical reasons. For example, you can also rightfully claim that English is a more natural language than, say, French if and only if you are a native speaker. It takes only about one or two days of actual coding in Lisp in order to get used to its nomenclature, and afterwards this whole thing essentially becomes a non-issue. Common Lisp also offers some alternative names, for example "first" and "rest" for "car" and "cdr", so you have a choice. However, you should prepare for reading code that still uses the traditional names.
The same holds for Lisp's seemingly excessive use of parentheses. It's just a matter of choosing a text editor with good support for Lisp-style indentation, and then you quickly forget about parentheses. You will seldomly worry about correct usage of syntax in Lisp. Please remember that due to their syntax, C-like languages and Pascal might also run into trouble when curly braces or begin/end keywords are used incorrectly in conjunction with control statements (like if, while, switch/case, and so on). However, if you are an experienced programmer in those languages, my guess is that you rarely have any problems with these issues. The same holds for Lisp syntax. It's just a matter of getting used to it.
Some people still have the notion that Lisp provides only lists as data structures. This notion comes from their understanding of very old dialects of Lisp, and from the fact that the name Lisp itself is short for "List Processing". In fact, Common Lisp offers everything that you have found useful in other programming languages, like structs (similar to C-like structs), arrays, true multi-dimensional arrays, objects (like in Smalltalk and Java, with fields and methods), strings, and so on.
Similarly, some people still think that Lisp is an interpreted language. The truth is that there are hardly any implementations of Lisp out there that are purely interpreted systems. All serious implementations include a compiler that produces machine code directly. So Lisp also usually distinguishes between compile-time and runtime. This difference becomes important for example in the case of macro programming.
Still, there are some differences in this regard to so-called "compiled languages" like C and most Pascal dialects. Most of the time, you don't explicitly compile a Lisp program and execute it afterwards. Instead you interact with a Lisp development environment that compiles Lisp code on the fly. So Lisp is closer in spirit to a kind of just-in-time compiled language. Lisp has the same interactive nature as, for example, Smalltalk (which, to some extent, could rightfully be regarded as a member of the Lisp family of languages).
The most important thing is that Lisp is a highly efficient programming language. Common Lisp vendors have worked very hard over the years to provide excellent performance.
(Please note that modern IDEs for Java, like JBuilder, NetBeans, Eclipse and the like are steadily progressing towards the same level of interactivity as that of Lisp and Smalltalk - so obviously this is considered valuable even by the communities of Algol- and C-like languages.)
A recent study of the performance characteristics of Lisp compared to Java and C++ can be found at the following URL: Erann Gat, Lisp as an Alternative to Java, http://www.flownet.com/gat/papers/. (If you have a problem with the amount of programming experience reported for the Java programmers in that paper, please look at http://www.flownet.com/gat/papers/ljfaq.html for further clarification.)
A term that is often used in descriptions of Lisp is "form". A form is a basic element of Lisp's syntax. Here is an example:
(format t "Hello, World")
This is a form that prints out "Hello, World" on the console. (The parameter "t" is the standard boolean value for "truth" in Common Lisp and is interpreted by the format function as standard output.) So the term "form" captures the notions of expressions and statements in other programming languages. (A Lisp purist would claim that Lisp doesn't have statements but only expressions, but this distinction doesn't matter in practice. In other programming languages, the border between statements and expressions is usually blurred by allowing for so-called "expression statements" - expressions whose values are discarded. You can do the same in Lisp, and Lisp even defines expressions that don't have values (or have "undefined values"). So in practice, Lisp and other languages make the same distinction and use similar ways to blur it, only with different "default behavior".)
There are also things in Lisp which are called "special forms". In order to understand this term you have to know that in Lisp usually all forms are applications of functions. The example above is an application of the function "format" with the parameters "t" and "Hello, World". The following example would be an application of the function "foo" with parameters "bar" and "goo".
(foo bar goo)
However, there are forms in Lisp that don't denote function applications. They are either macros or "special forms". Special forms just denote a very restricted set of built-in operations. (They are treated "specially" by the Common Lisp implementation. Note that built-in functions and macros are not special forms!) It's always clear from the first element of a form if it is a function form, a macro form or a special form. The distinction between the various forms has some theoretical and also practical relevance, but you don't have to worry about these things upfront. You will notice the difference as soon as it becomes important, and then you will easily recognize the means to deal with it.
There has been and there probably still is a big controversy in the Lisp community about small vs. big languages. This might sound confusing to people from other communities (at least this was the case for me). So what's the problem?
A nice thing about Lisp is that it doesn't noticeably distinguish between built-in functions and user-defined functions. Here are two examples.
(car mylist)
(cure mylist)
The function car is predefined in Lisp whereas cure does not have a standard definition, so it should be a user-defined function. The important point is that you can't tell the difference from the syntax. Compare this to the following example from the Java programming language.
synchronized (lock) {
beginTransaction(...);
...
endTransaction(...);
}
The synchronized statement in Java is a built-in feature that allows you to write blocks of code that are synchronized with regard to a particular object. The transactional methods in this example also define a block of code that (presumably) respect transaction semantics, but you can immediately notice that this is not a built-in feature of the Java programming language. (Yes, Lisp offers block-oriented statements - sorry - forms; and yes, it's possible to define your own block-oriented forms that still look like built-in features of Lisp.)
Now, in order to understand the small vs. big language issue you have to take this difference into account.
So what does it mean to standardize something like Java, or similar programming languages? You first have to standardize the language and you also have to standardize at least some of the libraries. As stated above, there is a noticeable difference between built-in features and libraries in such languages, so these are clearly two different tasks.
Not so in the Lisp world. Because there is no noticeable difference between built-in and library features, the standardization of libraries essentially becomes part of the language standardization (and in some sense, vice versa).
So the issue of small vs. big languages boils down (in Algol- and C-like terms) to the issue of small vs. big libraries - and there is really nothing more to this topic!
So this amounts to the following: Scheme is a small language in the sense that it only provides a very small standard library whereas Common Lisp is a big language because it provides a lot of useful and standardized features. However, if you look at the very core of both languages then they are more or less equivalent in size. (Furthermore, if you look at "serious" Scheme implementations they also provide feature-rich libraries that are, nevertheless, not standardized.)
There are still conceptional differences that don't contribute to the small vs. big language issue. This is only to state that choosing between Scheme and Common Lisp is not just an issue of language size.
For example, you can sometimes read that Scheme is a "Lisp-1" and Common Lisp is a "Lisp-2" - or that Scheme is a one-cell system and Common Lisp is a two-cell system. What they're talking about is the fact that values of variables on the one hand and function definitions on the other hand are stored in the same way in Scheme whereas they are stored differently in Common Lisp. This isn't really relevant for practical purposes, because obviously you can write real programs in both languages. They both offer means to deal with the consequences of being either Lisp-1 or Lisp-2. You can safely ignore this issue until you are actually exposed to it, and then again you will quickly recognize how to deal with it.
(The Scheme community regards it as aesthetically more pleasing to store values and functions in the same way because this has the appeal of minimality and orthogonality. Common Lisp is more concerned with practical issues and it obviously makes more sense to distinguish between the two kinds of storages in this regard. I don't know the exact details, and I don't care because it doesn't matter to me in practice. However, if you would like to learn more about this issue you can read about it in detail in the following paper.
Richard P. Gabriel and Kent M. Pitman, Technical Issues of Separation in Function Cells and Value Cells, http://www.dreamsongs.com/Separation.html)
It's remarkable that there is extensive information available about the whole history of Lisp, including Common Lisp and Scheme, in two very well-written papers. At some stages, these two papers even read like crime novels - highly recommended!
John McCarthy: History of Lisp (ACM SIGPLAN History of Programming Languages Conference, 1978), http://www-formal.stanford.edu/jmc/history/lisp.html
Guy L. Steele Jr. and Richard P. Gabriel: The Evolution of Lisp (ACM Conference on the History of Programming Languages II, 1992), somewhere in the middle of http://www.dreamsongs.com/Essays.html
The latter paper also includes a pretty complete bibliography on Lisp
(Please don't confuse these with other papers about the history of Lisp by Herbert Stoyan which I have found hard to read and in my opinion come to strange conclusions.)
As I stated before, Lisp includes a complete theory of computation by treating code and data uniformly - and this is one of the main features that makes Lisp so powerful. There are two papers I can recommend that give you a good explanation of what this actually means. (They deal with the concept of "meta-circular interpreters" - don't let the term frighten you, it's not as complicated as it sounds.) IMHO, you should at least read the first one to be able to really appreciate the power of Lisp.
Paul Graham, The Roots of Lisp, http://www.paulgraham.com/rootsoflisp.html
...and, by the way, this article also gives a good description of the basic ("primitive") forms in Lisp.
If you like "The Roots of Lisp" and would like to learn more about meta-circular interpreters you can read a deep discussion and vision of how far you can get with them in the following paper. Alongside, you will learn useful facts about lexical closures, dynamic binding, and the like.
Guy L. Steele Jr. and Gerald Jay Sussman: The Art of the Interpreter or, the Modularity Complex (Parts Zero, One and Two), http://library.readscheme.org/page1.html
(This paper makes hints about a follow-up paper but it has never been written nor published. So don't look for it: try to tie up the loose ends as an exercise. ;)
You can find a long list of Common Lisp implementations at http://www.lisp.org/table/systems.htm. However, this list doesn't seem to have been updated in a while. Here is a list of implementations to consider IMHO, in alphabetical order.
Allegro Common Lisp for Windows, Unix, Linux and Mac OS X, http://www.franz.com, commercial, Windows version includes an IDE
CLISP, apparently almost all platforms, http://clisp.cons.org, GNU public license, IDE only on Unix
CMUCL, Unix, http://www.cons.org/cmucl/
CormanLisp for Windows, http://www.cormanlisp.com, commercial, includes IDE
LispWorks for Windows, Unix and Linux, http://www.lispworks.com, commercial, Windows and Linux free for personal use, includes IDE
Macintosh Common Lisp, for Mac OS, Mac OS X in preparation, http://www.digitool.com, commercial, includes IDE
OpenMCL, Linux and Darwin (Mac OS X), http://openmcl.clozure.com/
PowerLisp for Mac OS, http://www.cormanlisp.com (sic!), commercial, free for personal use, includes IDE, no on-going development
Steel Bank Common Lisp, Unix, http://sbcl.sourceforge.net/
I have taken a closer look at only some of these implementations and there are probably other ones out there. Many Lisp programmers prefer to use emacs or xemacs as a development environment for Lisp but I prefer more modern IDEs, so this has surely restricted my search space. (No need to argue, this is just an irrational, subjective matter of taste.)
Specific recommendations for Apple Macintosh: PowerLisp is a free version that is good to play around with in order to get a taste of Common Lisp. However, it misses some features from the ANSI standard and runs only in classic mode, and there are no plans to port it to Mac OS X. Macintosh Common Lisp is obviously more advanced and the people at Digitool are generous and supportive enough to let you try out MCL for 30 days. CLISP is available for Mac OS X but misses an IDE, which is essential for me. Franz, Inc. are about to provide an IDE with Allegro Common Lisp for Mac OS X some time in the future but have no fixed plans, so don't hold your breath.
I have opted for Macintosh Common Lisp, especially because this system also has a nice and supportive user community built around the MCL mailing list. (However, this is not a product endorsement - please check out your requirements on your own!)
In this part of the introduction I mainly give hints and references that should help you in getting to know the several facets of Common Lisp. (That is, I'm mainly talking about its standard library.)
In every-day programming, you usually need some reference material to be able to look up function definitions, syntax clarifications, and so on. Some people prefer books and some prefer online material.
The current Common Lisp standard is specified as an ANSI standard that has been released in 1995 and is the definitive source for Common Lisp. However, it is hard to read because it mainly only gives the specs and, for example, doesn't provide good rationale.
At the beginning of the 1990's, Guy L. Steele edited the second edition of the book "Common Lisp the Language" that was a report on the then-current state of Common Lisp standardization, which is quite close to ANSI Common Lisp. (This edition superceded the first edition from the 1980's and so is generally referred to as CLtL2. The first edition - just CLtL - shouldn't be used anymore.) Although CLtL2 misses some features from ANSI Common Lisp - there are also some minor differences - it can be generally recommended as an introduction because it has the big advantage that it provides good descriptions beyond a mere specification. This greatly eases your understanding of the material. Still it is highly recommended to have the ANSI specs at hand in order to find out about the exact details of particular definitions. Furthermore, the ANSI document contains a very useful glossary of terms.
Fortunately, both ANSI Common Lisp and CLtL2 are available online as HTML (and PDF/PS) documents and can even be downloaded (including the HTML versions) for use on your computer. Here are the links.
ANSI Common Lisp: http://www.lispworks.com/reference/HyperSpec/
(The editors of this "HyperSpec" stress that this is not the definitive specification but just derived from the official ANSI document. However, I guess that this is only a legal issue and that you can most probably count on the HyperSpec. In fact, it is generated from the same TeX sources.)
Another hyperlinked version of ANSI Common Lisp, but in a different style, is offered by Franz, Inc. at http://www.franz.com/support/documentation/6.2/ansicl/ansicl.htm
CLtL2: http://www-2.cs.cmu.edu/afs/cs.cmu.edu/project/ai-repository/ai/html/cltl/cltl2.html
(Beware: CLtL2 has appendices about the so-called "Series" macros and "Generators and Gatherers". These features were not included in ANSI Common Lisp. I don't know anything about these features and I haven't found any rationale why they were included in CLtL2 and dropped again in ANSI Common Lisp. My guess is that they were too experimental. The "Series" macros can be found http://series.sourceforge.net as a separate library, but I can give no recommendations whatsoever. I have chosen to ignore these features for the time being.)
If you prefer printed matter, you have to put a little effort into getting some. CLtL2 seems to be out of print but you could try to obtain a used copy at Ebay. It's also still available at some (online) bookstores, like Amazon. You may also try to buy it at the publisher's website at http://www.bh.com/digitalpress/.
The ANSI document is available at http://global.ihs.com/, but I don't expect it to be particularly handy and/or useful because it has over 1000 pages. Furthermore at $368 it's really expensive.
Paul Graham's "ANSI Common Lisp" (see below) is a good book but I don't find it useful for reference purposes. Usually I prefer printed specifications (like the Java language specification) but in the case of Common Lisp, the electronic versions are just fine.
Paul Graham has written a good book on ANSI Common Lisp called "ANSI Common Lisp". It is a good read and fortunately, the author has made the first two chapters available on his homepage. I can especially recommend the second chapter because it quickly gives you a good hands-on experience of what programming in Common Lisp is like.
Paul Graham, ANSI Common Lisp, http://www.paulgraham.com/acl.html, Chapter 2, http://www.paulgraham.com/lib/paulgraham/acl2.txt
If you like his style you might consider buying the book by following the links on that page.
Most of the information you need afterwards should be fairly easy to look up in CLtL2 and/or the ANSI specs. See below for some information that is a little harder to obtain IMHO.
In order to completely understand the remaining sections of part II you should have read either "The Roots of Lisp" (http://www.paulgraham.com/rootsoflisp.html) or "ANSI Common Lisp Chapter 2" (http://www.paulgraham.com/lib/paulgraham/acl2.txt) by now, or you should have been exposed at least to example Lisp code in some other ways.
Sooner or later you will have to deal with the following issues that involve some minor issues.
Common Lisp offers possibilities to define functions that are able to deal with optional arguments, rest arguments, keyword arguments and arguments with default values in various combinations. The exact specifications for how to define these things can be found in CLtL2 in chapter 5.2.2 ("Lambda Expressions") and in the ANSI specs in chapter 3.4 ("Lambda Lists"), especially chapter 3.4.1 ("Ordinary Lambda Lists").
(Lambda expressions are unnamed functions. Named functions are similar to procedures, functions and/or methods in other programming languages. Unnamed functions in Lisp are similar to blocks in Smalltalk or, to a certain extent, to anonymous inner classes in Java. See CLtL2, chapter 5.2 ("Functions") and its subchapters for clarification.)
In Common Lisp, standard output is usually performed by way of the format function. It is somewhat similar in spirit to printf in C (although I have to admit I don't have extensive knowledge about printf!). I haven't found a good general discussion about format in any paper so you basically have to read the respective entry in CLtL2.
In CLtL2, format is described in chapter 22.3.3 ("Formatted Output to Character Streams"). The ANSI specs cover it in chapter 22.3 ("Formatted Output").
Strings are a little bit complicated in Common Lisp for the following reasons. Common Lisp has been standardized before the advent of Unicode - however, it was already clear that ASCII or some other limited set of characters is not enough. So the Common Lisp standard specifies a general means of dealing with strings that leaves some issues open to concrete implementations.
Usually, this shouldn't be a problem but it can become one when you want to interact with applications written in other programming languages. Allegro Common Lisp, CLISP and LispWorks offer Unicode support, other implementations don't (yet), as far as I know. (However, of course you can implement Unicode support on your own - Lisp is flexible enough for these purposes. ;-)
If you need to know more details about string support, consult CLtL2 and the ANSI specs in conjunction with the documentation provided with the Common Lisp implementation of your choice.
The handling of filenames differs from platform to platform. Again, Common Lisp has standardized a kind of framework that allows implementors to fill in the holes as required by the concrete operating system. So, again, you need to consult CLtL2, the ANSI specs and the docs of your Common Lisp implementation in order to get detailed information.
Common Lisp's macro feature is one of the highlights of this language - it is a very powerful means to write your own programming language constructs beyond mere functions.
Macros are usually considered an advanced topic because of their power and because they are seemingly hard to understand. However, I have found them to be conceptually quite simple and easy to write.
A very good description of macros and how to use them can be found in the book "On Lisp" by Paul Graham. It is out of print, but can be downloaded for free from the following URL.
Paul Graham, On Lisp, http://www.paulgraham.com/onlisp.html
Here are some cautions and notes by me.
Macros in Common Lisp are not like macros in C! As far as I understand, macros in C process sequences of characters and don't know anything about the structure of the programs they manipulate. In contrast, macros in Common Lisp operate on the very structure of programs. This is possible, again, because code and data are treated uniformly in Lisp. So, a program is a list of forms and, of course, macros can manipulate lists of forms like any other list.
So if your instinctive reaction to the term "macro" is to back out please think twice - this might just be caused by prejudices developed in conjunction with other, more underdeveloped, programming languages. ;-)
(Since I am not going to say anything about C macros anymore, I will use the term "macro" solely for Lisp macros from now on.)
Macros can be understood as functions which accept code as arguments and in turn generate new code. If an application of a macro is found at compile-time then the respective form is passed to the macro and its result replaces this very form. Compilation then continues with the new form. (This process is called "macro expansion".)
So here is a macro that defines "cure" to be the same as car.
(defmacro cure (argument)
(list 'car argument))
So whenever you see (cure a) this results in (car a) at compile-time - the result of the macro describes the actual code to be used instead of the macro application.
(Please note that this example is given for illustrative purposes only. It is not a good example for making use of macros: "cure" would be better defined as a function.)
What you can see here is the fact that macros are Turing-complete. All of Common Lisp's power can be used at compile-time in order to produce new code within macros. Macros can contain other macro expansions and function applications in order to determine their outcome and, of course, they can even use iteration and/or recursion, conditional statements, and so on. Even the resulting code may include further applications of macros, which leads to repeated "macro expansion" until the resulting code consists purely of function applications.
So, effectively, macros in Common Lisp can be used for the same purposes that, for example, template meta-programming is used for in C++. Especially, they can be used for Generative Programming! (Generative Programming essentially deals with the definition of domain-specific languages whose compilers produce code for general-purpose programming languages.)
Macros are usually introduced in conjunction with the "backquote" syntax. I believe that this way of introducing macros makes things needlessly hard to understand. In fact, when you understand the previous note about Turing-completeness of macros then you already have a complete picture of what macros are at the conceptual level. However, the combination of macros and the backquote syntax can provide some valuable synergistic effects in practice. In the following I will introduce the backquote syntax, and later on show how it combines with macros.
As stated above, most forms in Lisp are function applications. So (car mylist) extracts the first element of the list referred to by mylist. Sometimes you don't want to evaluate a form (you don't want to have it interpreted as a function application). For example, in (car (a b c)), (a b c) is by default not seen as a list of three elements, but again as a function a with parameters b and c. If you want to avoid the evaluation of (a b c) you have to use the special form "quote" as follows: (car (quote (a b c))). Now, this whole expression evaluates to "a" (the first element of the list (a b c)). Note that (quote ...) is not a function application but a special form (a built-in feature of Lisp, see above).
Since quote is used quite often in Lisp, one can abbreviate it by using the quote syntax as follows: (car '(a b c)) means exactly the same as (car (quote (a b c))). By the way, quote can also be used to avoid the evaluation of a variable. Variables in Lisp are just names (as in other programming languages), so if you write down, for example, (car a) then the first element of the list referred to by the variable "a" is extracted. In order to avoid evaluation of variables you can write (quote a) or shorter, 'a, as you might have already guessed.
Now, sometimes you need to create lists by evaluating several expressions and/or variables and then concatenating them into a list. The simplest way to do this in Lisp is as follows: (list a (car b) c) evaluates a, (car b) and c, and then concatenates the results into a single list. Sometimes only some of the parameters to the "list" special form need to be evaluated. This can be easily accomplished by quoting some of the parameters as follows: (list a (car b) 'c) evaluates a and (car b) but not c. Evaluation of 'c returns the symbol c itself, so the last element of the resulting list is guaranteed to be c.
Sometimes the number of quoted (unevaluated) expressions is higher than the number of expressions that need evaluation. This is where the backquote syntax comes into play: it reverses the default behavior and regards everything as not to be evaluated. The exceptions need to be marked. So the following example: `(a b c ,(car d)) means exactly the same as (list 'a 'b 'c (car d)) - the symbols a, b and c are not evaluated, but the expression (car d) is. In the backquote syntax, the expressions/variables that need to be evaluated are marked with a comma. (Note that this can't be simulated with the quote syntax: if you type ' then the whole expression that follows is quoted and there is no syntax available to have parts of it evaluated as an exception.)
So essentially the backquote syntax is merely a convenience feature that reverses the default behavior of what is evaluated and what is not.
Now, let's recall the macro example given above.
(defmacro cure (argument)
(list 'car argument))
From the discussion about the backquote syntax you can derive that this is equivalent to the following definition.
(defmacro cure (argument)
`(car ,argument))
Now you should immediately realize why the backquote syntax is so useful in conjunction with macros: the body of the macro definition is now much closer to the code that it actually produces. (Remember that an application of (cure mylist) results in (car mylist) at compile-time.)
Macros are at the heart of another, more esoteric controversy between the Common Lisp and Scheme communities. Macros in Common Lisp might inadvertently capture variable names of the code that surrounds the places of their application. This sounds extremely dangerous but it isn't. Please see Paul Graham's "On Lisp" book for a detailed discussion of this topic - I won't go into the details here.
Scheme offers the concept of so-called "hygienic macros" (or shorter, "hygiene") that avoids variable capture. The problem I see is that it is too easy to describe hygienic macros as an improvement of macros. The standard argument is as follows: "Common Lisp macros may capture variable names; hygiene avoids that; so programming becomes safer." This argument is somehow supported by the fact that hygiene resembles the concept of lexical closures to a certain extent.
However, this is not the whole truth. Sometimes you actually need to capture variable names, and hygiene makes this very hard. It seems that hygiene has thrown the baby out with the bath water. Furthermore, it is really trivial to avoid name capture in Common Lisp as Paul Graham shows in his book so there is no real issue to solve here in the first place.
So my message is: hygiene is not necessary (and not provided) in Common Lisp because it's easy to write safe macros. Unless you're really interested in this issue at the theoretical level, you can safely ignore the concept of hygienic macros.
(The situation in Scheme is a little bit different because Scheme stores variables and function definitions in the same way - it is a Lisp-1, see above. This means that accidental capture of function definitions is more likely to occur than in Common Lisp. Therefore, there is a higher need to solve this issue in that community. This topic is also discussed in the following paper, already mentioned earlier on.
Richard P. Gabriel and Kent M. Pitman, Technical Issues of Separation in Function Cells and Value Cells, http://www.dreamsongs.com/Separation.html
For the sake of completeness I also have to point to the fact that obviously there have been some advances in this area more recently. As far as I understand, the programming language Dylan offers hygienic macros with easy ways to intentionally capture variable names. For example, Jonathan Bachrach has been working on improvements concerning this issue. See sections "The Java Syntactic Extender" and "Dylan Procedural Macros" at http://www.ai.mit.edu/~jrb/.)
The Common Lisp Object System (CLOS) is part of ANSI Common Lisp. However, since it was a late addition to the standard, some vendors have taken their time to implement it. For example, PowerLisp does not provide full coverage of CLOS. Still, almost all implementations offer support for CLOS by now. (Some vendors praise this as an outstanding feature of their respective Common Lisp implementation and I have found that confusing at some stages. Strictly speaking, a Common Lisp system cannot claim to implement the ANSI standard if it doesn't include a full implementation of CLOS.)
CLOS offers full object-orientation with classes, subclassing, multiple inheritance, multi-methods and before-, after- and around advices. This already goes very well beyond the features offered by other object-oriented programming languages.
Furthermore, a so-called "Meta-Object Protocol" (MOP) has been added that allows for inspection and manipulation of class hierarchies and method dispatch at runtime. The Meta-Object Protocol is not part of the ANSI Common Lisp standard, but has become a de-facto standard because most CLOS/MOP implementations were derived from the same sources. (CLOS and MOP are not separate entities, as for example the Java language and its reflection API, but the MOP can rather be understood as a proper superset of CLOS.)
An excellent overview of CLOS and the Meta-Object Protocol can be found in the following paper.
Linda G. DeMichiel and Richard Gabriel, The Common Lisp Object System: An Overview, ECOOP '87, Springer LNCS 276, freely downloadable at the bottom of http://www.dreamsongs.com/Essays.html (thanks to Richard Gabriel for making this available on my request)
A reference for MOP that is similar in style to the ANSI HyperSpec can be found at http://www.lisp.org/mop/. (Most of the external links on that page are broken but the reference itself seems to work.)
Barry Margolin has given a good rule of thumb on comp.lang.lisp:
"There are a couple of MOP-related functions that managed to sneak into ANSI CL, such as ADD-METHOD and ENSURE-GENERIC-FUNCTION, but not much of this.
A good way to tell whether something is part of basic CLOS or the MOP is whether any of its arguments are required to be class, method, or generic-function objects. Functions like MAKE-INSTANCE or CHANGE-CLASS don't fall into this category; even though they accept a class object, they also accept a class name in its place, and MOP functions generally don't provide this convenience layer."
An elaborate (and impressive) example of what you can do with the Meta-Object Protocol can be found in the following paper.
Andreas Paepcke, User-Level Language Crafting - Introducing the CLOS Metaobject Protocol, in: Object-Oriented Programming: The CLOS Perspective, MIT Press, 1993, freely downloadable somewhere at the bottom of http://www-diglib.stanford.edu/~paepcke/shared-documents/bibliography.html
Some additional notes:
Multiple inheritance is usually regarded by some people to be a bad thing because it potentially leads to name conflicts that are difficult to sort out. Most programming languages offer some default behavior for dealing with such name conflicts, but more often than not this leads to unsatisfactory solutions. Common Lisp also defines a default strategy for dealing with name conflicts by imposing a topological order based on the order of the superclasses given by the programmer. One can, of course, easily come up with pathological examples where this approach does not yield useful results. So it seems that Common Lisp has a solution that is equally bad as those of other programming languages.
However, this is compensated for in CLOS by means of user-defined "method combinations". A programmer can always explicitly define how methods are dispatched on a case-to-case basis. (Of course, this can even be modified at runtime.) So CLOS offers a very high level of flexibility to deal with naming conflicts. The topological sorting of superclasses should just be understood as a suggestion by the system.
Naming conflicts caused by multiple inheritance are hard to deal with in any language for conceptual reasons. Again, Common Lisp opts for giving the programmer all the expressive power that he/she needs to deal with any situation that might arise. Compare this, for example, to the situation in Java where multiple inheritance of interfaces might lead to naming conflicts that cannot be dealt with at all.
The standard method combinations and the Meta-Object Protocol allow for programming in an aspect-oriented style. That's no coincidence - Gregor Kiczales, one of the inventors of Aspect-Oriented Programming, also co-designed CLOS. Don't wait for next week's version of AspectJ - you can already do everything with Common Lisp right now! ;-)
The so-called LOOP facility is another standard feature of Common Lisp that allows you to express iterations in a style that is similar in spirit to Algol/Wirth languages. Here is an example of a loop that prints ten stars.
(loop for i from 1 to 10
do (format t "*"))
Again, CLtL2 is a good source of information. However, you will quickly notice that there is a vast number of possibilities and combinations of different styles. My impression is that it doesn't make sense to learn all the details because this would take far too long. The goal of the LOOP facility obviously was to allow for some kind of "natural" English-like expressions of iterations, and there are examples where this definitely eases the understanding of source code. (It is also a nice example of a domain-specific language embedded in Common Lisp for the domain of iteration.)
Here is another, more interesting example of the LOOP facility (due to Matthew Danish).
(defun fibonacci (n)
(loop for a = 1 then (+ a b)
and b = 0 then a ; stepping in parallel
repeat n
collect b))
Seemingly, the intended way to use the LOOP facility is to just "guess" a way to express an iteration and see if it works. If it doesn't you can either look up the specifics in CLtL2 or the ANSI specs, or go back to a more Lispy way of expressing iterations and recursions (with do, dotimes, mapcar, and the like).
However, this is just my guess, I don't know about the actual intentions of the LOOP facility's designers. (Some claim that the details can be learned very quicly, but I still have to check this on my own.)
Conditions are more or less what exceptions are for Java-like languages. However, conditions are much more powerful because, for example, condition handlers can direct the Lisp runtime system to just continue execution at the place where a condition was raised. This might seem strange at first when you are used to exception handling because, hey, an exception always means that there is a problem, doesn't it? Well, in Common Lisp conditions can also be signalled when there is no problem but, for example, multi-threaded code needs to be synchronized or other code needs to be notified of special events or whatever you might think of as useful. Furthermore, some conditions that signal real problems can be fixed on the fly, either interactively or programmatically.
So just as the models of most object-oriented programming languages turn out to be special cases of CLOS, Java's exception handling is again a special case of Common Lisp's far more general Condition system.
An excellent overiew of condition handling in Common Lisp, along with some interesting historical tidbits can be found in the following paper.
Kent Pitman: Condition Handling in the Lisp Language Family, in: Advances in Exception Handling Techniques, 2001, Springer LNCS 2022, http://world.std.com/~pitman/Papers/Condition-Handling-2001.html
Some notes:
Common Lisp offers catch- and throw constructs as features that allow for so-called non-local exits from functions. These are not the analogues to the try-catch-finally triple in Java! Instead you use something like handler-case, handler-bind, and so on, for try-catch blocks and unwind-protect for try-finally blocks. Please see the above paper and the specs for details.
By the way, the Condition system could also be a basis for Aspect-Oriented Programming, without the need to go meta!
I have already mentioned Paul Graham's book "On Lisp" in conjunction with macro programming. This book deals with many other advanced programming techniques for Common Lisp, like creation of utility functions, higher-order functions, database access, simulation of Scheme-like continuations, multiple processes, non-determinism, parsing, logic programming, and object-orientation. (However, the bulk of this book deals with macros.)
I repeat the link for convenience.
Paul Graham, On Lisp, http://www.paulgraham.com/onlisp.html
Java has a nice module system that allows you to bundle classes into packages that are to some degree shielded from each other. Furthermore, the package structure is reflected in a matching file- and directory structure, possibly hidden in zip- or jar files. This allows for simple but detailed handling of classpaths that define where to look for classes that can be loaded by the Java runtime on demand.
Common Lisp offers a package system that allows for bundling definitions (including, of course, classes) as well. However, there is no mapping whatsoever to some kind of file- and/or directory structure. The package system in Common Lisp only deals with how definitions are arranged at runtime inside the Lisp environment.
In fact, Common Lisp's package system is more powerful than that because it more generally allows you to group symbols into packages. Because definitions in Common Lips are always accessed via symbols, the ability to bundle definitions is just a special case of the more general package concept. Here is a nice example of what else you can do with packages in Common Lisp, given by Kent M. Pitman in comp.lang.lisp. (See http://makeashorterlink.com/?E3B823F91 for the complete message that includes the following example.)
"CL allows me, simply in the symbol data, to control whether I'm doing various kinds of sharing. For example, I can separate my spaces, as in:
(in-package "FLOWER")
(setf (get 'rose 'color) 'red)
(setf (get 'daisy 'color) 'white)
(defun colors-among (things)
(loop for thing in things
when (get thing 'color)
collect thing))
(in-package "PEOPLE")
(setf (get 'rose 'color) 'white)
(setf (get 'daisy 'color) 'black)
(defun colors-among (things)
(loop for thing in things
when (get thing 'color)
collect thing))
(in-package "OTHER")
(flower:colors-among '(flower:rose people:rose flower:daisy))
=> (FLOWER:RED FLOWER:WHITE)
(flower:colors-among '(flower:rose people:daisy people:rose))
=> (FLOWER:RED)
(people:colors-among '(flower:rose people:rose flower:daisy))
=> (PEOPLE:WHITE)
(people:colors-among '(flower:rose people:daisy people:rose))
=> (PEOPLE:BLACK PEOPLE:WHITE)
So Common Lisp's package system deals with bundling of symbols and, hence, allows for bundling definitions as a "side effect", but does not deal with searching and loading of system parts at all. In Common Lisp terminology, the latter task is called "system construction".
So please don't get confused in this regard. Java's packages and Common Lisp's packages just happen to have the same name but are used for different purposes (with some overlaps).
Common Lisp only defines rudimentary support for system construction (the functions "load" and "require" - see CLtL2 and the ANSI specs). Please see the next section for some more information on this issue.
The ANSI Common Lisp standard was finalized in 1994 and published in 1995. That was at a time when Java was just around the corner but hadn't yet been made publicly available. That was also before the commercial rise of the Internet. It's clear that in the last seven years programming support has progressed in several directions. Unfortunately, the "official" standardization of Common Lisp has not continued in the meantime. Many things are not included in ANSI Common Lisp that are by now taken for granted in other, more fashionable programming languages. Among these features are: a proper module ("system construction") facility, Unicode support, a platform-independent GUI library, sockets and TCP/IP, XML, WebServices, and so on (add your favorite feature here).
However, the Lisp world hasn't stood still in the meantime - just because something is not specified in the ANSI Common Lisp standard this doesn't mean that it doesn't exist.
The Common Lisp Open Code Collection (http://clocc.sourceforge.net/) offers a system construction facility as a free add-on for ANSI Common Lisp, among other things. Allegro Common Lisp and CLISP offer Unicode support. CLIM is a platform-independent GUI library that is supported by some vendors, including Franz, Xanalys and Digitool. Most serious Common Lisp implementations offer support for sockets and more advanced networking facilities. CL-XML is a library for dealing with XML stuff.
So the basic message is: if you need some library, do a little research with Google and you might find something that you can use. Also check out some of the links provided below.
Here is a list of useful and/or interesting links. Some, but not all of them have already been mentioned in the text above.
Richard Gabriel's homepage, http://www.dreamsongs.com - one of the fathers of Common Lisp, various interesting essays can be found at his homepage
Erann Gat's homepage, http://www.flownet.com/gat/ - he conducted the study that compared performance characteristics of Lisp, Java and C++
Paul Graham's homepage, http://www.paulgraham.com - author of the books "ANSI Common Lisp" and "On Lisp", also offers various interesting articles
David B. Lamkins' homepage, http://psg.com/~dlamkins/ - author of the freely available online tutorial "Successful Lisp"
John McCarthy's homepage, http://www-formal.stanford.edu/jmc/ - he invented Lisp in the 1950's
Peter Norvig's homepage, http://www.norvig.com
Andreas Paepcke's homepage, http://www-db.stanford.edu/~paepcke/
Kent Pitman's homepage, http://world.std.com/~pitman/ - he was the project lead for the Lisp condition system
ANSI Common Lisp (a hyperlinked version of the standard), http://www.franz.com/support/documentation/6.2/ansicl/ansicl.htm
The Common Lisp HyperSpec, http://www.lispworks.com/reference/HyperSpec/
Common Lisp the Language, second edition, http://www-2.cs.cmu.edu/afs/cs.cmu.edu/project/ai-repository/ai/html/cltl/cltl2.html
Common Lisp - Myths and Legends, http://www.lispworks.com/products/myths_and_legends.html
Continuations, http://c2.com/cgi/wiki?CallWithCurrentContinuation - an explanation on the WikiWikiWeb (http://c2.com/cgi/wiki)
FAQ for comp.lang.lisp, http://www-jcsu.jesus.cam.ac.uk/~csr21/lispfaq.html
The Feyerabend project, http://www.dreamsongs.com/Feyerabend/Feyerabend.html
The original 'lambda papers' by Guy Steele and Gerald Jay Sussman, http://library.readscheme.org/page1.html - these papers date back to the 1970's and discuss various aspects of and around Scheme; some of them are also of interest to Common Lispers
Lexical scope, dynamic scope, (lexical) closures, http://c2.com/cgi/wiki?ScopeAndClosures - an explanation on the WikiWikiWeb (http://c2.com/cgi/wiki)
Lisp Scheme Differences, http://c2.com/cgi/wiki?LispSchemeDifferences - a discussion that compares Common Lisp to Scheme on the WikiWikiWeb (http://c2.com/cgi/wiki)
Meta Object Protocol, http://c2.com/cgi/wiki?MetaObjectProtocol - a discussion about Common Lisp's MOP on the WikiWikiWeb (http://c2.com/cgi/wiki)
Tail calls, tail recursion, http://c2.com/cgi/wiki?TailCallOptimization - an explanation on the WikiWikiWeb (http://c2.com/cgi/wiki)
The Association of Lisp Users, http://www.lisp.org, the portal to the Lisp world
CLiki, http://ww.telent.net/cliki/index, a collection of links to and resources for free software implemented in Common Lisp
The Common Lisp Cookbook, http://cl-cookbook.sourceforge.net/
Common Lisp Hypermedia Server, http://www.ai.mit.edu/projects/iiip/doc/cl-http/home-page.html, is a full-featured, open source web server implemented in Common Lisp
Common Lisp Open Code Collection, http://clocc.sourceforge.net
The Common Lisp Open Source Center, http://opensource.franz.com/
JACOL, http://jacol.sourceforge.net, a framework for interaction between Java and Common Lisp via sockets
UFFI, http://uffi.med-info.com/, a package to interface Common Lisp programs with C-language compatible libraries
CLISP, http://clisp.cons.org - CLISP (open source)
CMUCL, http://www.cons.org/cmucl/ - CMUCL (open source)
Corman Technologies, http://www.cormanlisp.com - Corman Lisp and PowerLisp
Digitool, http://www.digitool.com - Macintosh Common Lisp
Franz, Inc., http://www.franz.com - Allegro Common Lisp
OpenMCL, http://openmcl.clozure.com/ - OpenMCL (open source)
Steel Bank Common Lisp, http://sbcl.sourceforge.net/ - Steel Bank Common Lisp (open source)
Xanalys, http://www.lispworks.com - LispWorks
DrScheme, http://www.drscheme.org, is an implementation of Scheme I have checked out and liked a lot. If I hadn't opted for Common Lisp, I would probably have stuck with this system.
Scheme FAQ, http://www.schemers.org/Documents/FAQ/
Copyright Quick-Start for Online Authors, by Gene Michael Stover, http://gms.freeshell.org/htdocs/copyright/copyright.html
Many thanks (in alphabetical order) to Tom Arbuckle (http://www.cs.uni-bonn.de/~arbuckle/), Joe Bergin (http://csis.pace.edu/~bergin/) and Richard Gabriel (http://www.dreamsongs.com/) for providing lots of useful feedback on the draft version.
Further thanks for even more feedback go to Paolo Amoroso, Marco Antoniotti, Tim Bradshaw, Christopher Browne, Thomas F. Burdick, Wolfhard Buß, Bill Clementson, Matthew Danish, Biep Durieux, Knut Aril Erstad, Frode Vatvedt Fjeld, John Foderaro, Paul Foley, Erann Gatt, Martti Halminen, Bruce Hoult, Arthur Lemmens, Barry Margolin, Erik Naggum, Nicolas Neuss, Duane Rettig, Dorai Sitaram, Aleksandr Skobelev, Thomas Stegen, Gene Michael Stover, sv0f, Raymond Toy and Espen Vestre. (Many of them are active participators in comp.lang.lisp.)