Concept Programming in XL The Art of Turning Ideas into Programs
TM
Concept Programming
1
2
Concept Programming
Chapter 1— Introduction . . . . . . . . . . . . . . . . . . . . . . . . . .
13
1.1.
Why another language? . . . . . . . . . . . . . . . . . . . . . . . 14
1.2.
Who should read this book? . . . . . . . . . . . . . . . . . . . . 16
1.3.
A quick tour of XL . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.4.
Contents Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Chapter 2— Simple Examples . . . . . . . . . . . . . . . . . . . . . .
19
2.1.
Hello World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.
Factorial Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3.
Computing a Maximum . . . . . . . . . . . . . . . . . . . . . . . 24
2.4.
Symbolic Differentiation . . . . . . . . . . . . . . . . . . . . . . 26
Part I — Concept Programming . . . . . . . . . . . . . . . . . . . . . . 31 Chapter 3— Concepts? . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
3.1.
Programming Philosophy. . . . . . . . . . . . . . . . . . . . . . 34
3.2.
Translating concepts. . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3.
Pseudo-Metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.4.
Concept Mismatch . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.5.
In Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Chapter 4— The Trouble with Programming . . . . . . . . . . .
53
4.1.
Scale Complexity and Moore s Law . . . . . . . . . . . . . 54
4.2.
Domain Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.3.
Artificial complexity . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.4.
Business Complexity . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.5.
The Grim State of Software Quality . . . . . . . . . . . . . 68
Chapter 5— From Concepts to Code . . . . . . . . . . . . . . . . . 5.1.
71
Turning Ideas into Code . . . . . . . . . . . . . . . . . . . . . . . 72
Concept Programming
3
5.2.
Abstractions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.3.
Abstractions in Programs . . . . . . . . . . . . . . . . . . . . . . 84
Part II —Core Language . . . . . . . . . . . . . . . . . . . . . . . . . . . .93 Chapter 6— Compiling XL . . . . . . . . . . . . . . . . . . . . . . . . . 6.1.
Representing Programs . . . . . . . . . . . . . . . . . . . . . . . 96
6.2.
Understanding Programs . . . . . . . . . . . . . . . . . . . . . . 98
6.3.
Compiling XL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Chapter 7— Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
105
7.1.
Source Text. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
7.2.
Tokens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
7.3.
Parse Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.4.
Practical Considerations. . . . . . . . . . . . . . . . . . . . . . 121
7.5.
Beyond the Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Chapter 8— Declarations . . . . . . . . . . . . . . . . . . . . . . . . .
127
8.1.
Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
8.2.
Subroutines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
8.3.
Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
8.4.
Constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Chapter 9— Control Flow . . . . . . . . . . . . . . . . . . . . . . . . .
145
9.1.
Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
9.2.
Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
9.3.
Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
Chapter 10— Generic Programming . . . . . . . . . . . . . . . . .
4
95
147
10.1.
Generic Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
10.2.
Generic Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 148 Concept Programming
10.3.
Defining Usable Notations . . . . . . . . . . . . . . . . . . . . 148
Chapter 11— Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
149
11.1.
Object-Oriented Programming. . . . . . . . . . . . . . . . . 149
11.2.
Encapsulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
11.3.
Construction and Destruction. . . . . . . . . . . . . . . . . . 150
11.4.
Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
11.5.
Dynamic Dispatch . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Chapter 12— Programming in the Large . . . . . . . . . . . . . .
151
12.1.
Design By Contract . . . . . . . . . . . . . . . . . . . . . . . . . 152
12.2.
Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
12.3.
Compiling Multiple Files . . . . . . . . . . . . . . . . . . . . . 152
Chapter 13— The Preprocessor. . . . . . . . . . . . . . . . . . . . .
153
Part III —Extensibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Chapter 14— Compiler Plug-ins. . . . . . . . . . . . . . . . . . . . .
157
14.1.
Declaration and Invokation . . . . . . . . . . . . . . . . . . . 158
14.2.
Tree Transformations . . . . . . . . . . . . . . . . . . . . . . . . 158
14.3.
Useful Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 158
14.4.
Active Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Chapter 15— Optimizations . . . . . . . . . . . . . . . . . . . . . . . .
161
15.1.
Why Add Optimizations? . . . . . . . . . . . . . . . . . . . . . 162
15.2.
When Should Optimizations Run?. . . . . . . . . . . . . . 162
Chapter 16— Syntax Configuration . . . . . . . . . . . . . . . . . .
163
16.1.
Why Alter the Syntax? . . . . . . . . . . . . . . . . . . . . . . . 164
16.2.
Syntax File Description . . . . . . . . . . . . . . . . . . . . . . 164
16.3.
Rendering Styles . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Concept Programming
5
16.4.
Impact of Syntax Changes . . . . . . . . . . . . . . . . . . . . 164
Chapter 17— Compiled Code Configuration . . . . . . . . . . .
167
17.1.
Why Configure the Bytecode? . . . . . . . . . . . . . . . . . 167
17.2.
Bytecode File Format . . . . . . . . . . . . . . . . . . . . . . . . 168
17.3.
Retargetting the Compiler . . . . . . . . . . . . . . . . . . . . 168
17.4.
Customizing the Output . . . . . . . . . . . . . . . . . . . . . . 168
17.5.
Generating Other Kinds of Output . . . . . . . . . . . . . . 168
Part IV —Standard Library . . . . . . . . . . . . . . . . . . . . . . . . 171 Chapter 18— Runtime Models and Portability . . . . . . . . . . 18.1.
Machine Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
18.2.
Object Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
18.3.
Error Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
18.4.
Environment Model . . . . . . . . . . . . . . . . . . . . . . . . . 175
18.5.
How Portable is Your Code? . . . . . . . . . . . . . . . . . . 176
Chapter 19— Core Functionality . . . . . . . . . . . . . . . . . . . .
177
19.1.
Integer Arithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . 177
19.2.
Floating-Point Arithmetic. . . . . . . . . . . . . . . . . . . . . 177
19.3.
Comparisons and Boolean Logic . . . . . . . . . . . . . . . 177
19.4.
Loop Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Chapter 20— Memory Management . . . . . . . . . . . . . . . . .
179
20.1.
Pointer Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
20.2.
Memory Allocation . . . . . . . . . . . . . . . . . . . . . . . . . 180
Chapter 21— Collections . . . . . . . . . . . . . . . . . . . . . . . . . .
6
173
181
21.1.
Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
21.2.
Strings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 Concept Programming
21.3.
Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
21.4.
Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
21.5.
Collection Algorithms . . . . . . . . . . . . . . . . . . . . . . . 182
Chapter 22— Mathematics. . . . . . . . . . . . . . . . . . . . . . . . .
183
22.1.
Numeric Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
22.2.
Transcendental Functions. . . . . . . . . . . . . . . . . . . . . 184
22.3.
Numeric Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . 184
22.4.
Symbolic Code Manipulations . . . . . . . . . . . . . . . . . 184
Chapter 23— External Storage . . . . . . . . . . . . . . . . . . . . .
185
23.1.
Streaming Objects . . . . . . . . . . . . . . . . . . . . . . . . . . 185
23.2.
Storage Management . . . . . . . . . . . . . . . . . . . . . . . . 186
23.3.
Database Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Chapter 24— Text Formatting . . . . . . . . . . . . . . . . . . . . . .
187
24.1.
Text Formatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
24.2.
Text Output. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
24.3.
Text Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
Chapter 25— Concurrency. . . . . . . . . . . . . . . . . . . . . . . . .
189
25.1.
Forms of Concurrency . . . . . . . . . . . . . . . . . . . . . . . 190
25.2.
Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
25.3.
Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
Chapter 26— Graphics. . . . . . . . . . . . . . . . . . . . . . . . . . . .
193
Chapter 27— Sound. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
195
Chapter 28— User Interfaces. . . . . . . . . . . . . . . . . . . . . . .
197
Chapter 29— Service Brokering. . . . . . . . . . . . . . . . . . . . .
199
29.1.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
Chapter 30— Error Handling . . . . . . . . . . . . . . . . . . . . . . . Concept Programming
201 7
30.1.
Annexes
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .203
References 30.2.
8
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 Quotes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Concept Programming
Listing 2-1:Hello World Program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Listing 2-2:Making the module scope implicitly visible . . . . . . . . . . . . . . . . . . . . . . . . 20 Listing 2-3:Factorial Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Listing 2-4:Iterative definition of the factorial. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Listing 2-5:Displaying factorial values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Listing 2-6:Output of the program in Listing˚2-5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Listing 2-7:Computing the maximum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Listing 2-8:Using the maximum function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Listing 2-9:Error reported when the type doesn t match . . . . . . . . . . . . . . . . . . . . . . . . 26 Listing 2-10:Defining a function using derivative notation . . . . . . . . . . . . . . . . . . . . . . 27 Listing 2-11:Code transformations when using the differentiation plug-in . . . . . . . . . . 27 Listing 2-12:Chaining plug-ins. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Listing 2-13:Defining a function using derivative notation . . . . . . . . . . . . . . . . . . . . . . 28 Listing 2-14:A plug-in implementing basic simplifications . . . . . . . . . . . . . . . . . . . . . . 28 Figure 3-1: Bitmap representation of 3 . . . . . . . . . . . . . . . . . . .35 Listing 3-2:Addition in XL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Listing 3-3:Output of the program in Listing˚3-2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Listing 3-4:ASCII encoding of the program in Listing˚3-2 . . . . . . . . . . . . . . . . . . . . . . 36 Figure 3-5: Pong, the first videogame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Figure 3-6: VirtuaTennis, a more recent version of Pong. . . . . . . . . . . . . . . . . . . . . . . . 39 Listing 3-7:Computing the maximum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Listing 3-8:Implementation of Maximum in C. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Listing 3-9:An object-oriented Maximum in Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Listing 3-10:C++ generic implementation of max . . . . . . . . . . . . . . .42 Listing 3-11:Maximum of a container in C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Listing 3-12:Displaying factorial values in C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Listing 3-13:Output of the program in Listing˚3-12. . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Listing 3-14:Overspecified Factorial function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Listing 3-15:Gross overabstraction of the factorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Listing 3-16:Subtle overabstraction of the factorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Listing 3-17:Underabstraction for the factorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Figure 4-1: Evolution of the size of the Linux kernel over time . . . . . . . . . . . . . . . . . . 55 Figure 4-2: Long-term evolution of source code size for different projects . . . . . . . . . 56 Listing 4-3:xauth man page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Listing 4-4:Conflicting syntax rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Listing 4-5:Syntax conflicts within a single language . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Listing 4-6:Ambiguous syntactic statements in C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Listing 4-7:Examples of ambiguous constructs in context . . . . . . . . . . . . . . . . . . . . . . 62 Listing 4-8:Error messages from g++ 2.95 on this code . . . . . . . . . . . . . . . . . . . . . . . . 63 Figure 5-1: Evolution of Programming Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Concept Programming
9
Listing 5-2:Combinations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Listing 5-3:Computerese and mathematical notation. . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Figure 5-4: Visual Basic non-textual editing of program code . . . . . . . . . . . . . . . . . . . 85 Figure 5-5: NeXT Interface Builder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 Listing 5-6:Poor choice of names. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 Listing 5-7:Factorial with documented limitations on input . . . . . . . . . . . . . . . . . . . . . 88 Listing 5-8:Permutations function without composition . . . . . . . . . . . . . . . . . . . . . . . . 89 Listing 5-9:Breaking down composition rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Listing 6-1:Factorial source code decomposition in tokens . . . . . . . . . . . . . . . . . . . . . . 97 Listing 6-2:Command to generate a graphical parse-tree representation. . . . . . . . . . . . 97 Figure 6-3: Color-coded representation of an XL parse tree . . . . . . . . . . . . . . . . . . . . . 98 Listing 6-4:Example of scanner definition using lex . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Listing 6-5:Simple calculator using bison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Listing 6-6:Declaration and Definition in C. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Figure 6-7: XL compiler components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Listing 7-1:Using the integer notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Listing 7-2:Adding long text delimiters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Listing 7-3:Infrequent condition where capitalization is significant in XL . . . . . . . . . 110 Listing 7-4:Sample code being fed to the XL parser . . . . . . . . . . . . . . . . . . . . . . . . . . 112 Listing 7-5:Generating a parse tree dump from the sample code. . . . . . . . . . . . . . . . . 112 Listing 7-6:Dumping the sample code in graphical form. . . . . . . . . . . . . . . . . . . . . . . 113 Figure 7-7: Color-coded representation of an XL parse tree . . . . . . . . . . . . . . . . . . . . 114 Listing 7-9:Default xl.syntax file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Listing 7-10:Inconsistency between statements and functions. . . . . . . . . . . . . . . . . . . 120 Listing 7-11:Using XL0 to represent a document . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 Listing 7-12:Incorrect precedence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Listing 7-13:Selecting the correct precedence using a different operator name . . . . . 123 Listing 7-14:Selecting the correct precedence using parentheses . . . . . . . . . . . . . . . . 123 Listing 7-15:Non-intuitive parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Listing 7-16:A parse tree in the code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Listing 7-17:A more clumsy way to create the same parse tree . . . . . . . . . . . . . . . . . . 124 Listing 7-18:Escaping a name in a syntax expression . . . . . . . . . . . . . . . . . . . . . . . . . 124 Listing 7-19:Code generated for the parse tree expression in Listing˚7-18 . . . . . . . . . 124 Listing 7-20:Selecting between statements and expressions . . . . . . . . . . . . . . . . . . . . 125 Listing 7-21:Comparing statement and expression parse trees . . . . . . . . . . . . . . . . . . 125 Listing 7-22:Syntax expression to recognize a parse tree . . . . . . . . . . . . . . . . . . . . . . 125 Listing 7-23:Commutativity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Listing 8-1:Variables required for a calculator program . . . . . . . . . . . . . . . . . . . . . . . 128 Listing 8-2:Variables with actual types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Listing 8-3:Simple calculator program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
10
Concept Programming
Listing 8-4:Adding minimum and maximum computations . . . . . . . . . . . . . . . . . . . . 130 Listing 8-5:Company/employee relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Listing 8-6:Type inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 Listing 8-7:Displaying pseudo-random numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 Listing 8-8:Subroutines named after verbs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Listing 8-9:Use of procedure . . . . . . . . . . . . . . . . . . . . . 135 Listing 8-10:A simplified interface for ReadLn . . . . . . . . . . . . . . . 136 Listing 8-11:Incrementing a value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 Listing 8-12:Simplified interface of WriteLn . . . . . . . . . . . . . . . 137 Listing 8-13:Overloading based on return type is invalid . . . . . . . . . . . . . . . . . . . . . . 137 Listing 8-14:Writing to the console in Pascal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Listing 8-15:Writing to the console in C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Listing 8-16:Writing to the console in C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Listing 8-17:Writing to the console in Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Listing 8-18:Calculator reading complete expressions . . . . . . . . . . . . . . . . . . . . . . . . 140 Listing 8-19:Simplified interface for WriteLn . . . . . . . . . . . . . . . 141 Listing 8-20:Expression reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Listing 8-21:Ambiguous expression reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Listing 8-22:Error message for an ambiguous expression reduction . . . . . . . . . . . . . . 142
Concept Programming
11
12
Concept Programming
Chapter 1 — Introduction
Ce que l’on conçoit bien s’énonce clairement. (What we understand clearly we express clearly) Nicolas Boileau
XL stands for eXtensible Language. It is a programming language designed from the ground up to solve problems in modern programming that old tools can t address. This is a bold claim. Extraordinary claims require extraordinary proof. This book is an attempt at giving such a proof, by explaining what XL is, why it was necessary to invent a new language, what problems it solves (or doesn t). Understanding begins with knowledge, and for that reason, this book will attempt to teach you how to program in XL. But more importantly, I hope that this book will give you the desire to program in XL. In my very biased opinion, XL is more fun to program with than most other programming languages. It results in dense, clean code that is easy to write and maintain. I hope to share some of my enthusiasm for XL with you. An XL compiler is available as a free software project [1].
Concept Programming in XL
13
Introduction
1.1. Why another language? There are dozens of programming languages today. So why create yet another language? What kind of problems are not solved by today s languages? If these problems are real, and worth solving, why can t they be solved by fixing or improving existing languages? Is XL simply reinventing the wheel? What problem does it solve? XL attempts to solve a large number of problems, as this books will hopefully demonstrate. But there is one key problem, one that by itself justified the extra effort of putting the X (extensibility) in XL: how can a language allow programmers to comfortably represent arbitrary concepts in their code?
This problem is worth solving. Over and over, programmers find themselves unable to translate an important idea in their code and, as a result, need to workaround the limitation or invent new programming tools. For example, object oriented programming required new programming languages, because the old ones could not represent the concepts involved such as inheritance or polymorphism, in a sufficiently usable manner. On a more modest scale, programmers hit this limit in most programs of a reasonable complexity, and solve the problem either with verbose coding, representing an implementation of the concept rather than the concept itself, or by developing custom tools such as domain specific languages or specifically designed preprocessors. The operative word in this problem statement is arbitrary. Existing programming tools are designed around a limited set of constructs, such as objects or functions. As a consequence, there are concepts that they will not support gracefully. C did not support object-oriented programming well, and it could not be made to support it well without a major twist. Massive redesign of the language was required to that end, in the form of Objective C or C++. By contrast, the ideas behind XL are intended to allow the language to grow on demand, without major redesign, and under control of programmers rather than language designers. This new approach focusing on the representation of concepts as code is called concept programming, and it is the main topic of Chapter 3. XL attempts to achieve concept programming objectives by being extensible at various levels, as shown in Part III: • The syntax can be altered in a controlled way. • The semantics can be extended; programmers can add new constructs. • The output of the compiler can be customized. This extensibility allows the programmer to focus first on the concepts to represent in the code, and then writes code on his or her own terms, using familiar notations. Why not fix another language? There are millions of lines of code written in other lan-
guages. Some form of compatibility with an installed base clearly facilitates the adoption of a new languages1. There needs to be a powerful reason to break from the legacy.
14
Concept Programming in XL
Why another language?
In the case of XL, the reason to create a new language is the impossibility to achieve the primary objective, extensibility, without starting over from scratch. It is not by lack of trying. One of my previous project was a Java-to-Java extensible compiler called Moka [3]. It did not work well for reasons that are beyond the scope of this introduction, but which can be summarized as follows: a single programmer may be able to add simple language extensions to Java, but if twenty programmers attempt that, the result is chaos, conflicts and confusion. Java was not designed from day one to be extended, and as a result, attempting to extend it breaks it. C++, with its vastly more complex set of syntactic and semantic rules, would be much worse. From a technology point of view, there is another way to understand this issue. One key mechanism to allow language extensions is the ability to manipulate internal representations of the program, notably a form called a parse tree (see Part III - Extensibility). Internal representation for languages like Java or C++ typically uses dozens of node types. There will be an node type for if statements, a node type for function declarations, and so on. Extensions would need to add their own node types. This makes it difficult for one extension to correctly deal with all node types, and to avoid conflicts with other extensions. By contrast, the XL parse tree only requires 7 node types (see Section 7.3 on page 112), and is designed so that extensions do not need to (and cannot) add new types of nodes. Reinventing the Wheel? Initially, one might think that there will still be a fair amount of
useless reinvention, if only for concepts that are already widely used. How different can XL integer types be, compared to C or Java integer types? It turns out that starting with a clean sheet gives a lot of freedom for considering what can be done differently today than over 30 years ago, when C was designed. This is particularly true with a new guiding principle such as concept programming. As suprising as it sounds, even the most mundane programming abstractions like integer types or integer arithmetic are subtly different in XL than in languages like C or C++. Think about it: if XL is extensible, programmers should be able to add something like an integer type or an integer addition themselves, shouldn t they? Conversely, is there a better proof that XL s extensibility mechanisms work than to use them to implement such basic concepts? This is, indeed, exactly the approach taken in XL. There is no keyword like int naming any built-in type. There are no built-in arithmetic operators either. The integer types are defined in the standard library, as are arithmetic operators like addition. Chapter 19 and following will describe how this is achieved. Naturally, this should not come at the price of usability. Using these types in XL works just like in C or Java. In the end, everything looks the same, everything behaves the same but it is now under your control, rather than a guarded temple of the sacred priesthood of language designers and compiler writers.
1. Compatibility with a legacy is not essential for a new language to be successful, however. Perl or Python are relatively recent languages that were not obviously derived from another language.
Concept Programming in XL
15
Introduction Having Fun. There is one last reason for creating a new language: I want programming to
be fun. Is there any C programmer who never heard a complaint about this or that aspect of C, C++ or Java? For C, C++ or Java, programmers have very little influence. But XL is under their control. XL certainly will have flaws and limitations initially. But the good news is that programmers can fix them over time, and that the most widely accepted improvements will soon become standard. This is, in a sense, very similar to Linux. Linux started as a deeply flawed and limited Unix copycat. But over time, these flaws and limitations were fixed, thanks to its free software nature. Linux is now a fully grown, mature system, and it s a lot more fun to use than most original variants of Unix. My hope is that over time, XL will mature into an environment that is undisputably more fun to work with than other languages.
1.2. Who should read this book? This book was written by a programmer for programmers. While it doesn t really require experience with other programming languages, it often uses them as a point of comparison. This is based on the assumption that most programmers will be famililar with at least one of the languages used as an example, and that it is important to highlight differences between XL and what they are used to. As I wrote in the introduction, I would like this book to give you a desire to program in XL. But I have no illusion that the choice of a programming language for a specific project depends on a number of factors, including legacy, development environment, business constraints. As a result, maybe you have no plans to use XL in the short term. Is it useful to read about a language even if you don t intend to use it immediately? I believe that the answer in the case of XL is yes. XL, due to its concept programming roots, may teach you a trick or two, leading your mind towards paths that are not natural in your usual programming world. For that reason, I want to believe that most programmers will benefit from learning about XL, whether they plan to use it or not.
1.3. A quick tour of XL In Chapter 2, we will give a few practical examples of XL, to get an idea of what the language looks like. Before we look at this code, I can highlight some of the features that define XL s identity, in the hope that this will make you curious. XL has no keywords, and its syntax is totally context free (Chapter 7). It has no built-in types, no built-in operators, these are all defined in the library and can be redefined 16
Concept Programming in XL
Contents Overview
(Chapter 19). The XL parse tree is standardized, and there are tools to match or generate parse trees easily (Chapter 6). The compiler accepts plug-ins that extend the language by recognizing parse trees containing the extension and transforming it into a lower-level form (Chapter 14). A set of plug-ins implement a relatively usual structured imperative language, wich supports generic programming (Chapter 10), and large-scale programming techniques including objects (Chapter 12). XL has a very powerful mechanism for defining operators, called expression reduction (Chapter 10). A rich set of libraries provide a large number of features (Chapter 20 and following), including concurrency, exception handling, garbage-collected or manually-controlled memory management, file and database storage, graphical user interfaces, and more.
1.4. Contents Overview Chapter 2 gives a few simple examples of XL programs, and concludes this introduction. Part I - Concept Programming explains the fundamental ideas on which XL was built. Part II - Core Language defines basic language features used in any XL program. Part III - Extensibility explains how XL can be extended, and how to use that capability. Part IV - Standard Library describes the environment in which XL programs execute, including the functionality provided by the standard XL library.
Concept Programming in XL
17
Introduction
18
Concept Programming in XL
Contents Overview
Chapter 2 — Simple Examples
Let’s not make things worse by guessing Gene Kranz, Ground control, Apollo Mission
It is possible to define a language from the bottom up, but this approach makes it difficult to have a global vision of what the language looks like, and what it is about. Trying to make up the missing information would lead you to attempt to guess what it should look like, and for XL, the guess would all too often be wrong. For that reason, it appears necessary to start with a few simple programs. I will also use these carefully chosen programs to highlight some of the design choices that might not appear obvious at first sight, but may help you quickly grasp the XL design philosophy. The programs that we will look at are the following: • A simple program displaying Hello World on the output console. • Programs computing and displaying factorials. • Programs computing and displaying minimums and maximums • A program demonstrating a domain-specific extension to the core language Despite being very short and very simple, all these programs will demonstrate some interesting properties of XL for programmers.
Concept Programming in XL
19
Simple Examples
2.1. Hello World Little could seem more desperately boring than a program displaying Hello World on the output console. Here is one such program in XL: LISTING 2-1: Hello World Program
1 import IO = XL.UI.CONSOLE 2 IO.WriteLn "Hello World"
Can anything interesting be told about such a program? As a matter of fact, yes. Let s begin with the first line. The import statement is used to import a module. Modules are one of the constructs supporting large-scale programs (see Chapter 12, Programming in the Large ). A module declares (exports) a number of related entities, which another part of the program can then use. In this example, the module XL.UI.CONSOLE exports a subroutine called WriteLn, which writes the text given as an argument to the console. Each module creates a scope, meaning that WriteLn is not directly visible in the program that imports the module. Rather, a scope resolution is necessary to indicate where WriteLn comes from. In large programs, this helps locate which module a given subroutine comes from. The complete name of the WriteLn subroutine is XL.UI.CONSOLE.WriteLn. There may be other subroutines with the same name in other modules. For instance, there is also an XL.TEXT_IO.WriteLn, which writes to a stream (such as a file or a network connection), and in most implementations forms the basis for XL.UI.CONSOLE.WriteLn. It would not be very convenient to use the full names all the time, so there are two ways to shorten the reference to a specific module. In Listing˚2-1, we used the first approach and gave a short name, IO, to the module in the import statement. This allows us to refer to XL.UI.CONSOLE.WriteLn with the shorter name IO.WriteLn. Another approach is shown in Listing˚2-2. The use statement makes a module scope, implicitly visible in the current context. In that case, it is no longer necessary to use a scope resolution to refer to the WriteLn subroutine: the compiler finds it through the use statement. Note that the scope is made visible, but it remains distinct from the current scope. LISTING 2-2: Making the module scope implicitly visible
1 use XL.UI.CONSOLE 2 WriteLn "Hello World"
The XL.UI.CONSOLE module is one of many standard modules. These modules together form the XL standard library, which defines a very large fraction of the capabilities of the language. The library is organized in a hierarchical way, the top-level module being XL. Application modules can also be defined, as well as technology specific modules. A specific implementation of XL can feature a MICROSOFT.WINDOWS.GDI module, a POSIX.STDIO module, and so on. The XL modules offer one additional portability guarantee compared to application modules: the features they contain may not always be there (in which case a program using them would not compile), but if they are there, their interface and semantics is precisely defined.
20
Concept Programming in XL
Factorial Function
For instance, it is possible that you compile for a tiny embedded system, like a toaster, that obviously doesn t have console capabilities. If you try to build the programs in Listing˚2-1 or Listing˚2-2 for that toaster, they will fail to compile, because there is no console and the module XL.UI.CONSOLE will not be present. However, if the module is present and contains a WriteLn subroutine, then the subroutine has to output the Hello World text to the console in response to these programs. The standard library provides a module called XL_BUILTIN. This module defines operations that are always available: everything behaves as if a use of XL_BUILTIN was made before any other statement. Basic numeric types and essential operations like addition are made available by this module. The portability guarantee also applies to this module. For instance, consider a 16-bit machine with no floating-point support. The XL compiler for this machine may not feature 64-bit integer types and operations, nor any floating-point arithmetic. An attempt to use these features will result in a compilation error. Since the standard library is implemented using only normal XL capabilities, it is possible to add to it. Imagine that on your 16-bit microcontroller, you really need a floating-point addition, and you are willing to pay the price of a software implementation. This software implementation can be added to your programming environment in such a way that it is not distinguishable from one that would have shipped with the compiler in the first place. A last thing to note about this short program is that there is no main function like in C or C++. Instead, the code of the program is executed directly.
2.2. Factorial Function The factorial function is used notably in probabilities. The factorial of an integer N is written as N! and defined as follows: 0! = 1 N! = N (N-1)! for any integer N > 0 This definition translates in XL into the code in Listing˚2-3. LISTING 2-3: Factorial Function
1 function Factorial(N : integer) return integer written N! is 2 if N = 0 then 3 return 1 4 else 5 return N * (N-1)!
Concept Programming in XL
21
Simple Examples Function Declaration. The Factorial function declaration in line 1 describes essential properties of this first XL implementation of the factorial concept1: • A function in XL denotes a subroutine that returns a value. The value of mathematical functions is usually considered to depend solely on their arguments, but XL functions may depend on program state, like global variables. • The name Factorial was chosen after the name of the concept we try to represent in the code. The compiler itself doesn t care about the name. The program would work just as well if the code contained the name BlueLagoon, but the relationship between the code and the original concept would be somewhat obscured. The point is that the name is really intended only for the programmer, as a way to refer to the code entity. • Enclosed in parentheses is a parameter list, which in this case declares a single parameter, called N. This parameter has the type integer, a computerized approximation of mathematical integers. This parameter list is intended to match the parameters required for the mathematical factorial:2 the mathematical function also takes a single parameter. The name N is just as arbitrary as Factorial, and we could replace all instances of N with another name without changing the meaning of the program. • The return integer part specifies that the function Factorial returns a value of type integer. In our mathematical description, we did not specify this information, since it could be deduced from the values used to compute the function. One could imagine that the compiler could similarly deduce the return type. But there are several software engineering reasons to provide this additional element of information. For example, an useful technique in programming is encapsulation: in some cases, we might want to hide the implementation from Factorial users, while still specifying what value it returns. This will be explained starting with Chapter 8. • Finally, the written N! part declares a notation for the function, called a written form. Again, this notation is chosen to match the corresponding mathematical notation. The compiler doesn t care about the notation, and the function could be declared and used without any written form (as in many other programming languages). Written forms play the role in XL that operator overloading plays in programming languages like C++ or Ada, but it is more general and more flexible. Given this written form, the expression (N-1)! on line 5 is equivalent to a function call like Factorial(N-1). Function Implementation. The implementation of the function follows after the word is, and has a few noticeable properties: • XL uses indentation to define the structure of programs, such as blocks of instructions. The Factorial implementation is the indented code located between line 2 and line 5 included. Indentation gives the code an graphical shape matching the high-level code
1. The reason why I was careful to take the the precaution of describing the code in Listing˚2-3 as an implementation of the factorial concept will soon become apparent. 2. This choice of type is intentionally flawed. We will discuss why in Section 3.4 on page 48.
22
Concept Programming in XL
Factorial Function
structure. That makes it easy for the eyes to locate structure boundaries, and minimizes the risk of conflict between the way the brain and the compiler scan the code. • The if statement is used to make decisions based on a test. If the condition N=0 is true, the indented code following the then is executed, in that case the return statement on line 3, which returns the value 1. Conversely, if the condition is false and there is an else part as in our example, the indented code following else will be executed, which in that case attempts to compute the factorial using recursion. Iterative Definition. Listing˚2-3 is only one possible implementation of the factorial concept. It uses a recursive definition, where N! is computed as a function of (N-1)!, and so on until 0! is computed. There are other possible ways to define a factorial. An alternative, iterative definition of the factorial can be written as follows:
N! = N x (N-1) x (N-2) ... x 2 x 1 This definition leads to a different implementation of the Factorial function, called an iterative implementation of the factorial, shown in Listing˚2-4. LISTING 2-4: Iterative definition of the factorial
1 function Factorial(N : integer) return integer written N! is 2 result := 1 3 for I in 2..N loop 4 result *= I
It is noteworthy that the interface of the function defined in Listing˚2-4 is identical to the interface in Listing˚2-3. However, the implementations differ significantly, with respect to their performance characteristics in particular. On most platforms, Listing˚2-4 will generate faster and more efficient code than Listing˚2-3, because a function call is generally more expensive than a loop. Let us just note for now that even if there is a single mathematical concept, multiple implementations can be written. There is a general rule that multiple valid code representations exist for the same concept. These implementations may still differ in important ways for the programmer: performance, memory usage, implementation limits. Note that the variable I in the for loop is not explicitly declared. Much like in languages such as Ada, XL for loops often declare the iterating variable implicitly. However, while usage and syntax are similar to Ada or Pascal equivalents for the simplest cases, the implementation is very different. As you might have guessed, XL for loops are extensible, using a mechanism described in Section 9.2.4, For Loops on page 146. This mechanism also controls what variable may be declared implicitly and what type it will have. Displaying Factorials. A program to display the factorials of numbers 0 to 10 is shown in Listing˚2-5. In addition to the same function Factorial as in Listing˚2-3, the program
now contains code that is executed when the program begins, much like the code displaying Hello World in Listing˚2-1.
Concept Programming in XL
23
Simple Examples LISTING 2-5: Displaying factorial values
1 import IO = XL.UI.CONSOLE 2 3 function Factorial(N : integer) return integer written N! is 4 if N = 0 then 5 return 1 6 else 7 return N * (N-1)! 8 9 for I in 0..10 loop 10 IO.WriteLn I, "!=", I!
The code iterates over all values of I between 0 and 10. For each value, it uses WriteLn to display the value of I and the value of I! The output of that program is shown in Listing˚2-6. LISTING 2-6: Output of the program in Listing 2-5. 0!=1 1!=1 2!=2 3!=6 4!=24 5!=120 6!=720 7!=5040 8!=40320 9!=362880 10!=3628800
2.3. Computing a Maximum The minimum and maximum concepts are well understood. There are subtle mathematical variants, but for the time being, I will focus on the following definition for the maximum of N values a1, a2, , a N: max(a 1 ) = a 1 $ " max(a 1, a 2, …, a N ) = # " !
a1
if max(a 2, …, a N ) < a 1
max(a 2, …, a N )
otherwise
This definition assumes the existence of an order relation between individual values, that we wrote with the less-than sign < and that we use to compare two values. A type with such an order relation is said to be ordered1. I chose the recursive definition above because it is an operational algorithm to find the maximum of a set of values. This makes it easy to translate into code. The corresponding code is shown in Listing˚2-7.
1. For simplification, we assume a total order. The more general case is beyond the scope of this introduction.
24
Concept Programming in XL
Computing a Maximum LISTING 2-7: Computing the maximum
1 generic type ordered where 2 A1, A2 : ordered 3 Test : boolean := A1 < A2 4 5 function Max(A1 : ordered) return ordered is 6 return A1 7 8 function Max(A1 : ordered; ...) return ordered is 9 result := Max(...) 10 if result < A1 then 11 result := A1
The code in Listing˚2-8 shows how this Max function can be used. LISTING 2-8: Using the maximum function
1 X : integer := Max(1, 3, -1, 17) 2 Y : real := Max(1.0, 0.7, -12.21)
Generic Types. The first part of the code in Listing˚2-7, between line 1 and line 3, tells the XL compiler what an ordered type is: a type is ordered if two values A1 and A2 of that type can be compared with a < operator returning a boolean (truth) value. We need a boolean value because this is what the if statement expects as a condition, and we will use the if statement in the implementation of the algorithm. The word generic on line 1 indicates that the type ordered is not a specific implementation type, but rather a name that can be used to represent any type, such as integer or real.
For instance, in Listing˚2-8, we use Max with integer or real values. This is valid, since integer and real values can be compared with a < operator, and so they are ordered types. When Max is applied to integer values, the code in Listing˚2-7 is equivalent to code where all instances of ordered had been replaced with integer. The first parameter of function Max has type integer, and the function returns an integer type. Similarly, when Max is applied to real values, the definition of Max is equivalent to code where all instances of ordered were replaced with real. This very powerful technique is called generic programming, and I will discuss it in more details in Chapter 10. Overloaded Functions. The definition of Max requires two function declarations, one on
line 5, and one on line 8. These two declarations differ in their parameter list. The compiler choses what function declaration to use based on the arguments given to the function. For instance, Max(1) will use the declaration on line 5, whereas Max(1,3) will use the declaration on line 8. This is called an overloaded function. We have already seen overloaded subroutine with WriteLn, since it received different types of arguments in Listing˚2-1 and in Listing˚2-5. Overloaded functions can be used to represent the various facets of a same concept, as we did here, or to represent different concepts that happen to share a name in the usual vocabulary. For instance, one might have an Add function corresponding to integer addition, and
Concept Programming in XL
25
Simple Examples
an Add function corresponding to the addition of a new device to the system, two conceptually very different uses of the word add . Variable Argument Lists. The declaration on line 8 makes use of the symbol .... This
indicates that the number and type of the additional parameters is unknown. This variable argument list is the XL formalism corresponding to the notation that we used in the original definition. These ... parameters can only be used as a whole to call another subroutine, as we do with the expression Max(...) on line 9. If we initially wrote Max(1,2), the compiler notices that the declaration on line 5 doesn t work, so it uses the declaration on line 8, with ... being the argument 2. The Max call on line 9 becomes equivalent to Max(2). This time, the declaration on line 5 applies. What would happen if we wrote Max(1,2.5)? Like before, the declaration on line 8 is chosen, with ordered being integer, and the expression on line 9 becomes Max(2.5). This now uses the declaration on line 5, but with the subtle difference that ordered becomes real. This, in itself, is valid. However, assigning a real to an integer is not valid, since the type is not the same. So the compiler detects a type error, as shown on Listing˚2-9. LISTING 2-9: Error reported when the type doesn’t match test.xl:17: Errors while instantiating 'Max': | Test_bad : integer := Max(1, 2.5) |_____________________________^ test.xl:17: with '...' set to '2.5' | Test_bad : integer := Max(1, 2.5) |____________________________________^ test.xl:12: No translation found for 'result := Max ( ... )' | result := Max(...) |_____________^
Please note that the above error message relates to the implementation of Max. It is possible to write code using variable argument lists (...) where different arguments have different types. We have already used such a subroutine: WriteLn.
2.4. Symbolic Differentiation As the name implies, computers are often used to perform computations. For that reason, most programming languages since Fortran feature support for mathematical expressions. At the very least, arithmetic operations can easily be performed by using notations that are direct transpositions of their mathematical counterpart. Where the mathematician thinks of a sum as a+b, the code contains a+b. Domain-specific notations. There are a few application domains, such as economics or
physics, where equations and concepts are expressed or defined not just in terms of functions and basic algebra, but in terms of mathematical derivatives1. There are simple formu-
26
Concept Programming in XL
Symbolic Differentiation
las to compute the derivative of most functions. It is simple enough that even pocket calculators such as the Hewlett-Packard HP-48 series can do it. Consider the computation of the rate of change of a dampening oscillator. This rate of change is expressed by an equation that looks like the following: f &t) =
t d( sin & %t ) exp (– ---- )) & t 0 '' dt&
Except if you have been a programmer for too long, it is tempting to use code that looks like Listing˚2-10: this is a natural extension of the kind of notation we use for algebra. LISTING 2-10: Defining a function using derivative notation
1 function f(t, omega, t0 : real) return real is 2 return d(sin(omega * t) * exp(-t/t0))/dt
But if you have been a programmer for long enough, you know that most compilers do not know how to compute derivatives. It is not a technological problem, since pocket calculators can do it. It is simply that the derivative notation, while widely used by a few, is still not used by enough people to be incorporated in computer languages. Compiler plug-ins. The XL compiler doesn t know any more about symbolic differentia-
tion than a C, Java or Ada compiler does. However, unlike these languages, XL supports compiler plug-ins. Plug-ins are pieces of code that interact with the compiler to perform source code transformations. With the appropriate differentiation plug-in, the code in Listing˚2-10 becomes valid. Symbolic differentiation is a good example of domain-specific notation where the use of a plug-in makes perfect sense: the notation is not needed by most, but those who need it find the ability to use the notation very handy. Instead of having the manually differentiate the expression (or using a pocket calculator and copying the result), rocket scientists can now write code in their programs that more closely reflects the concept it represents. Command-line invokation. The first way to invoke a plug-in is to use a compiler option to
apply the plug-in to the whole source code. In the free software implementation of XL, such options begin with a + followed by the name of the plug-in. The effect of the plug-in when the compiler is invoked that way is shown in Listing˚2-11. LISTING 2-11: Code transformations when using the differentiation plug-in $ xl -parse test.xl -show function f (t , omega , t0 : real) return real is return (d (sin (omega * t) * exp ( - t / t0) ) / dt) $ xl -parse test.xl +differentiation -show function f (t , omega , t0 : real) return real is
1. The derivative of a function f & x ) is usually denoted as
Concept Programming in XL
df . Refer to algebra books for details. dx
27
Simple Examples return ((0 * t + omega * 1) * cos (omega * t) * exp ( - t / t0) + sin (omega * t) * (( - 1 * t0 - - t * 0) / (t0 ^ 2) * exp ( - t / t0))
The -show option renders the code that the compiler parsed. The -parse option requests that the source code be simply parsed, without attempt to analyze it or generate code. Finally, the +differentiation option is what caused the plug-in to be invoked. Plug-ins can be chained on the command-line, in which case the output of the first plug-in is fed as input to the second one. Listing˚2-12 shows how the constantfold plug-in, which eliminates useless operations like multiplications by 0, can cleanup the output of the differentiation plug-in. LISTING 2-12: Chaining plug-ins $ xl -parse test.xl +differentiation +constantfold -show function f (t , omega , t0 : real) return real is return (omega * cos (omega * t) * exp (- t / t0) + sin (omega * t) * (- t0 / t0 ^ 2 * exp (- t / t0)))
Pragmas. The drawback of command-line invokation is that the plug-in applies to the entire source code. This increases the chances that two plug-ins will conflict with one another, and makes it difficult to locate where in the source code a given plug-in is used.
A more local invokation of a plug-in uses a notation known as pragma, which is the name of the plug-in (and optional parameters) between curly braces. In our case, we will use the {differentiation} pragma to invoke the differentiation plug-in. This is illustrated in Listing˚2-13, which differs from Listing˚2-10 only by its use of the pragma. LISTING 2-13: Defining a function using derivative notation
1 function f(t, omega, t0 : real) return real is 2 return {differentiation} d(sin(omega * t) * exp(-t/t0)) / dt
The effect, internally, is to apply the plug-in only to the section of the code marked by the pragma. What section exactly is being processed will become clearer after reading Chapter 7. Writing Plug-ins. Plug-ins are metaprograms, that is programs that manipulate programs.
Chapter 14 explains in details how they work. A complete listing for the differentiation plug-in is a bit beyond the scope of the present introduction, but by the end of the book, you will fully understand how it works. Listing˚2-14 implements a much simpler plug-in, which is a very reduced version of the ConstantFold plug-in called Simplify, which removes additions of 0, multiplications by 1 and multiplication by 0. LISTING 2-14: A plug-in implementing basic simplifications
1 translation Simplify 2 when ('X'+0) then 3 return X 4 when ('X'*1) then 5 return X 6 when ('X'*0) then 7 return parse_tree(0)
28
Concept Programming in XL
Symbolic Differentiation
The first line declares the plug-in called Simplify using the translation statement. That statement is itself provided by a plug-in: most programs would not have any use for translation statements. That translation plug-in is normally used only to write plugins or other forms of XL meta-programs. The translation statement contains a number of when clauses, which specify source code forms to recognize. Within a when clause, quotes are used to name elements and implicitly declare variables of the same name in the code following then. For instance, when(’X’+0) will recognize forms like 1+0 or A*3+0. In the first case, X will be set to the internal compiler representation of 1, which is equivalent to parse_tree(1); In the latter, it will be set to the internal representation of A*3, which could also be generated using parse_tree(A*3). Quotes can be used in parse_tree as well, so that parse_tree('X'+0) when X is the internal representation of 2*A[1] will return the internal representation for 2*A[1]+0. All these extension mechanisms will be explained in much more details in Part III - Extensibility.
Concept Programming in XL
29
Simple Examples
30
Concept Programming in XL
Symbolic Differentiation
Part I — Concept Programming
Concept Programming in XL
31
Simple Examples
32
Concept Programming in XL
Symbolic Differentiation
Chapter 3 — Concepts?
There is no spoon The Matrix
Concept programming is a programming methodology focusing on the process of translating concepts, that reside in the programmer s head, into code, that lives in the machine. This may sound very simplistic, but experience has shown that many errors are caused by a lack of understanding of the differences between a concept and its code representation. Just like the character Neo in the movie The Matrix has trouble understanding how a young boy can bend a spoon using his mind alone, we sometimes tend to forget that our code does not behave like the concepts it represents. There is no spoon. The difference with The Matrix is that all programmers know about this boundary, they simply forget it. By drawing attention to the gap between code and concepts, concept programming makes it possible to discover and address flaws that otherwise would not be obvious even to experienced programmers. It offers new techniques to evaluate the quality of code.and suggest innovative ways to attack well-known computing problems. Hopefully, the Max function in Listing˚2-7 should have illustrated some of these benefits to those of my readers who are experienced in other programming languages: as written, Max is almost impossible to write in programming languages predating XL, even if it represents a very common and widely used concept. Concept programming is really the secret sauce behind XL. It is the core design philosophy. This is why it is necessary to understand a little better what it really means before we go deeper into the finer technical details of XL.
Concept Programming in XL
33
Concepts?
3.1. Programming Philosophy Concept programming focuses on the most fundamental process of programming, the process of translating concepts, which reside in our brain, into code, which is ultimately intended to materialize on some kind of computing device. For a good understanding of the translation process, one must: • Clearly delineate the concept space and the code space (Section˚3.1.1), • Define an equivalence between concepts and their code representation (Section˚3.1.3),
3.1.1. Concept Space and Code Space In order to make the gap between code and concepts more explicit, I will often refer to two abstract locations, concept space and code space. This is obviously only a figure of speech, but it often helps clarify what I am referring to. I use a typographical convention where a blue, fixed-size font denotes what belongs to the code space. For instance, I may talk about the window in the code space inheriting some of the properties of a window in the concept space. In that particular case, this conceptual window may itself be some abstraction of the common properties of the physical objects we call windows. Some concepts do not even have a physical representation to refer to. For instance, a function only has a mathematical definition, but no physical reality1. In the code space, this distinction between physical and conceptual entities does not exist. Both a function and a window will ultimately boil down to some particular binary encoding. It is impossible to tell, by looking at a piece of code, if the concept it represents has a physical reality or not. For that reason, to the extent where we are ultimately only interested in the code representation, we can safely ignore the distinction in the concept space as well. In other words, we assume that the conceptual window accurately represents the physical ones. Programmers usually refer to physical models when they describe this simplification. Programmers make another distinction between data and code. This is a distinction I will generally avoid when considering the code space: program code can be seen as input data for some execution engine, something that is quite explicit in the case of a Java virtual machine or of a PC emulator. Conversely, a Microsoft Word document can be seen as a program intended for the Microsoft Word execution engine. For that reason, whenever I refer to code space, this covers equally any form of program data. There are gray areas in this distinction: when the program is forming in the programmer s mind, it usually exists in a conceptual form, yet something that is already very close to the final code, is it in concept space or in code space? By convention, I will choose to call it 1. We may postulate some specific encoding using neuron connections or neurochemicals in our brains, but in the current state of medical science, we are unable to describe the physical materialization of a function.
34
Concept Programming in XL
Programming Philosophy
concept space until it actually can be materialized in one way or another. For instance, a listing in a book is in code space, but the understanding one gets by reading the listing is in concept space1.
3.1.2. Abstractions The reason we tend to forget about the distinction between code space and concept space is that we built a lot of software that attempts to mask the difference. When we refer to the number 3, its representation in the code is generally written as 3. From there on, it becomes hard for us not to think of 3 as the actual number 3. Nothing could be farther from the truth. If you look carefully, 3 is really, depending how you look at it: • A bitmap on a graphic screen that may look something like Figure 3-1
FIGURE 3-1: Bitmap representation of 3
• A binary representation in memory of the ASCII code for the character 3 (its ASCII code is a 7-bit or 8-bit value with decimal value 51) • A binary representation in memory of the value 3, namely 00000011 if the value is stored on 8 bits. There is a very large number of other possible representations for the number 3, but none is remotely close to the mathematical entity we call 3. How is it, then, that we can become so confused as to no longer distinguish 3 and 3, or even forget how different they are?
3.1.3. Principle of Equivalence The reason we forget about the difference between a concept its representation in the code space is that layers upon layers of hardware, software and conventions in our computers are designed specifically to create and maintain this illusion. For instance, the XL program in Listing˚3-2 displays the an output looking like what is shown in Listing˚3-3.
1. This distinction will actually prove useful in some cases. See Listing˚5-6 for an example.
Concept Programming in XL
35
Concepts? LISTING 3-2: Addition in XL
1 import IO = XL.UI.CONSOLE 2 IO.WriteLn ”3+4=”, 3+4
LISTING 3-3: Output of the program in Listing 3-2 3+4=7
Naively, one might be tempted to say that the computer simply computed 3+4, and since the result is 7, that s what it displays. After all, computers are supposed to be good at math, aren t they? But in reality, the process is much more complex. Let s try to re-create every step, just to understand how much we ignored with our naive interpretation. 1. The program in Listing˚3-2 exists in the code space in the form of some sequence of bits, using a conventional character encoding such as ASCII or Unicode. For instance, what the computer displays as 3 is really stored as an 8-bit (byte) value 51 using the ASCII encoding. Inside a Unix machine using the ASCII encoding, the program in Listing˚3-2 is really a sequence of bytes with the values shown in Listing˚3-4 (each byte being itself a sequence of 8 bits). Near the end, you will recognize a byte with value 51, and that s our 3 (there s another one in the middle that is part of the text literal ”3+4=”). LISTING 3-4: ASCII encoding of the program in Listing 3-2 105 109 112 111 114 116 32 73 79 32 61 32 88 76 46 85 73 46 67 79 78 83 79 76 69 10 73 79 46 87 114 105 116 101 76 110 32 34 51 43 52 61 34 44 32 51 43 52 10 2.
3. 4.
5. 6. 7. 8. 9.
36
Another program, the XL compiler, reads this sequence of bytes, and interprets it as an XL program. As a consequence, it understands the byte with value 51 as the character 3, and according to the XL language rules, considers the first one in the program as being part of the text literal ”3+4=”, and the second one as the integer literal 3. The integer literal 3 is converted to an internal binary representation. The same happens to the integer literal 4. Part of the language rules specify that 3+4 will result in a binary addition of the binary representations for 3 and 4. Since these are constants, this addition could happen during the compilation, but let s assume that the compiler decides to perform it only when the program actually runs. The compiler generates another sequence of bits, the output program, suitable to be interpreted as an executable program on the target platform. The program is executed. An execution engine, such as a microprocessor or virtual machine, reads the sequence of bits generated by the compiler. Part of this execution is to execute an instruction that adds the binary pattern for 3 and the binary pattern for 4, resulting in the binary pattern for 7. The code for WriteLn is executed, the last argument is the binary pattern for 7. The code for WriteLn is designed to transform its input into a sequence of characters representing the numerical value. In that case, WriteLn emit a single character, and on an ASCII machine, that character would have encoding 55.
Concept Programming in XL
Translating concepts 10. The
character is interpreted by the execution environment. On modern systems, it will typically be rendered graphically in some window with a bit pattern similar to Figure 31 (except that this time, it is a rendering of 7 instead of 3). 11. Our brain interprets the bit pattern on the screen as an image of the number 7 and think: Wow! This computer sure knows about math! . In the whole process, only step 7 performed anything remotely close to our addition (and then, it was really a binary addition). Yet all these layers of software conspire to give us the illusion that the computer knows about mathematics, integers and addition. The same is true for any entity we manipulate through a computer: we believe that we adjust the color of a picture in Adobe Photoshop, but we really apply carefully designed algorithms on a binary representation of the picture, creating another binary representation that the computer renders as a picture with different colors. The important takeway is that the code space and the concept space are not even remotely similar. However, a lot of software in the computer is there with the only purpose to give us computerized approximations of concepts we are familiar with. The existence of a large body of software designed to give us the illusion that we are dealing with familiar concepts creates a very useful relationship between the concept space and the code space, which I call the principle of equivalence: a concept and its code representation are equivalent. Within limits which I will discuss in Section 3.2.3, Domains on page 43, they have the same meaning, and behave similarly, to the point where it is possible and useful to ignore the distinction. The principle of equivalence is what we rely on when we assume that 3 is just the same as 3, or that Photoshop just changed the colors of the picture. More importantly for programmers the principle of equivalence is the only reason we can write programs at all. Every program begins with a conceptual description. Without the principle of equivalence, we wouldn t have any reason to believe that the code representation would behave like this conceptual description. With it, however, we can write an addition as 3+4 as in Listing˚3-2, and trust that the computer code will behave like the mathematical addition 3+4. We verified this step by step in the previous section, so we can now justify our belief. But the same reasoning applies to other numbers, other operations, to pictures, and to practically any concept we may want to manipulate with a computer.
3.2. Translating concepts In Section˚3.1, we defined the landscape in which the transformation of concepts into code occurs. Having defined the context, it is now possible to define the process itself. At the highest level, this translation process can then be summarized as follows: • Create a model, which includes only the relevant concepts (Section˚3.2.1),
Concept Programming in XL
37
Concepts?
• Materialize this model as code (Section˚3.2.2), • Identify and document the limits of the model (Section˚3.2.3).
3.2.1. Modeling The problems really start in the concept space. With very few exceptions, programmers do not attempt to put the whole universe in their computer. They simplify. They identify the relevant concepts and their key properties, and remove or ignore irrelevant ones. This all important simplification, which I call modeling, is the very first step in building a program in the concept programming methodology. The resulting set of concepts, and the relationships between these concepts, form the model of the program. The model is the first description of what the program attempts to do. How do we identify relevant concepts and properties? Consider for instance the first commercial videogame, Pong, shown on Figure 3-5. Most people identify this crude set of pixels to a simplified game of tennis.
FIGURE 3-5: Pong, the first videogame
Such an identification is possible because the most essential concepts of the game are present in Pong: • There are two players, each in a half of the terrain, the net being represented by a dotted marking splitting the screen in halves. • A ball bounces back and forth along the terrain, with different trajectories. • Each player controls the position of some kind of racket that can be used to return the ball if it touches it. • If the ball passes behind the racket, the player on that side loses. • There is a scoreboard. It is difficult to think of a simpler model of the game of tennis. Remove any of the bullets and the game becomes much less attractive, and much more difficult to identify as a game. This makes it very interesting to consider what was removed from the game: • The play area is two-dimensional,
38
Concept Programming in XL
Translating concepts
• The ball does not spin. It does not have a texture, nor color, and its movements are very simplified • The rackets can only be controlled along a single axis. • Other ways to lose a point, such as throwing an object to the head of the opponent, are not available. • The complicated scoring system of real tennis was vastly simplified • There are no ads on display alongside the play area, and no public. More recent videogame simulations of tennis, such as the one shown in Figure 3-5, removed most of these simplifications. They vastly expanded on the original model. This shows the amount of flexibility that we have in modeling.
FIGURE 3-6: VirtuaTennis, a more recent version of Pong
It also demonstrate the vast capabilities of the human brain at bridging the gap : for all the added details in Figure 3-5, the resulting picture would still fool no one in believing that this is a real game of tennis. To some extent, both games are summarized as implementations of the following concept: a computer simulation of the tennis game
3.2.2. Materialization A model of what the program must do is the first necessary step, but at some point, programmers need to translate that model into actual code. I call this materialization. Materialization is very dependent on the actual programming environment being used. The materialization of Pong, for instance, involved some discrete logic. This is something programmers no longer have to do very often. They resort to significantly higher level of
Concept Programming in XL
39
Concepts?
abstraction. Depending on the tools and abstractions that are available to you, specific concepts may or may not represent easily. It is important to use tools that can materialize the concepts you care about in a form that preserves some kind of equivalence, as defined in Section˚3.1.3. In practice, tools are very often a significant limiter. To illustrate that, let s consider the concept of maximum for which we gave a simple XL implementation in Listing˚2-7, shown again in Listing˚3-7. This concept is one of the simplest imaginable concept, and it has a number of uses in computing. So you could expect that it is easy to materialize in any programming language. LISTING 3-7: Computing the maximum
1 generic type ordered where 2 A1, A2 : ordered 3 Test : boolean := A1 < A2 4 5 function Max(A1 : ordered) return ordered is 6 return A1 7 8 function Max(A1 : ordered; ...) return ordered is 9 result := Max(...) 10 if result < A1 then 11 result := A1
Interestingly, it is not easy at all. As a matter of fact, it is quite difficult to design a really good implementation of the concept of maximum with widely available programming tools or techniques, ranging from object-oriented programming to generic programming. In that respect, all development tools and methodologies are the programming equivalents of George Orwell s Newspeak in the novel 1984 : without the capability to represent a concept in your programs, it becomes much more difficult to think about that concept or even to try and use it at all. Conversely, the few concepts that are available to you rapidly become overused. To a man with a hammer, everything looks like a nail; to a woman programming in Smalltalk, everything looks like an object. In this section, I will merely illustrate some of the problems programmers face materializing concepts with inadequate tools. I will analyze these problems more systematically in Section 3.3, Pseudo-Metrics and Section 3.4, Concept Mismatch . The ideal “maximum” concept: In its most general interpretation, the very simple concept called maximum is not limited to any number of items, nor does it apply to any specific type of item either. It applies to any number of ordered values, where you can define ordered in a number of ways, for instance by using a less-than relational operator.
The most natural program representation of this concept would therefore be something that computes the maximum of a set of values (not just two), provided they are ordered . In computer programs, something that computes can be a function, or an object with methods triggering the computation, or a program receiving data as input, or even a hardware component which interacts in some way with the program to produce the result. Any of these programming models might be best suited to a particular class of problems.
40
Concept Programming in XL
Translating concepts Notation. Regardless of how it s implemented, the something that computes will be
referred to using a particular program notation. A natural notation for a maximum concept is something like Max(1,5,3). This choice of notation is an important step in materialization. Natural here needs to be taken with a grain of salt, because this is a highly subjective choice, but one can argue that for the majority of people not exposed to programming, this notation would seem closer to the normal function notation they learn in basic mathematic classes1 than, say, (max 1 5 3) or 1 max: 5 max: 3, which happen to be the notations used by Lisp and Smalltalk respectively. In the case of programs, this same notation can, once again, cover a number of different implementations: it can be a function call in most languages, or a method call, or the use of a preprocessor macro, or in a language like C++, the construction of an object, or the invokation of a specific method of a function object. In practice, how it s implemented doesn t matter much, as long as the notation remains easy to use and to understand and is reasonably efficiently to execute. Simplified implementation. Actual implementations of the maximum concept are not always close to this ideal. For instance, a common implementation of the maximum concept in C is shown in Listing˚3-8. LISTING 3-8: Implementation of “Maximum” in C
1 int max(int a, int b) { if (a < b) return b; else return a; }
This implementation is obviously quite limited compared to our ideal definition. It covers only integer values, and accepts only two arguments. In practice, this implementation is still quite useful. Following a common rule of programming, less than 20% of the effort has solved over 80% of the problem. The definition in Listing˚3-8 is actually good enough that most programmers will not even realize its limitations. For example, when they need to compare three values, most C programmers will naturally write max(A,max(B,C)), not necessarily noticing that in doing so they expose a knowledge of the implementation of max and break encapsulation. This is easy enough for max, but the same attitude can prove risky in more complicated scenarios. For instance, there is no direct equivalent of this strategy for the average concept, since the average of N values cannot be rewritten easily using only the average of two values. An object-based approach. A particularly nasty kind of materialization problem occurs
with inadequate application of a respected methodology. Programmers may consider the methodology authoritative, and refuse the notion that it may not apply. In the case of the maximum concept, we can illustrate this using the object-oriented method. A pure object-oriented programming model recommends using objects to which you send messages . This turns out to be a rather complicated and unnatural approach for implementing the maximum concept. For example, an object-oriented Java implementa1. This is the notation I used when defining the maximum concept in Section˚2.3.
Concept Programming in XL
41
Concepts?
tion of maximum could be something like Listing˚3-9. To compute the maximum, you create an instance of the Maximum class, and you invoke the Data method to supply the data on which the maximum needs to be computed. LISTING 3-9: An object-oriented Maximum in Java
1 class Maximum { 2 private int maxInt; 3 4 public Maximum(int X) { maxInt = X; } 5 public Maximum Data(int X) { 6 if (X > maxInt) 7 maxInt = X; 8 return this; 9 } 10 public int Result() { 11 return maxInt; 12 } 13 14 public static void main(String[] args) { 15 int max = new Maximum(2).Data(8).Data(5).Data(7).Result(); 16 System.out.println("The maximum is: " + max); 17 } 18 }
The natural notation, Max(2,8,5,7), is nowhere in sight. Instead, we find something like new Maximum(2).Data(8).Data(5).Data(7).Result(), which is arguably quite ugly. If only for this practical reason, this implementation doesn t appear as the best way to deal with the problem at hand. But the notation also exposes a number of disturbing issues. What is this Maximum object after all? What is its equivalent in the concept space? Generics and templates: As we noted above, our implementations so far only works with
one data type, integers1. To deal with the common class of algorithmic problems which are identical except for the type, languages such as Ada, C++ or Eiffel offer genericity (or templates in C++ jargon). Genericity is the ability to describe parameterized data types or functions, where parameters can themselves be data types. For instance, you can describe a Max function template which indicates how to compute a maximum, and then instantiate that template to create a Max function for integers, a Max function for floating-point values, and so on. A generic version of the max function in C++ is shown in Listing˚3-10. LISTING 3-10: C++ generic implementation of max
1 template 2 T max(T a, T b) { if (a < b) return b; else return a; }
This implementation doesn t share Listing˚3-8 s restrictions on types. However, it still has the restriction on the number of arguments. One way to workaround the issue in C++ is to use another concept that is more easily materialized: what if we replaced the maximum of N entities with the maximum of a container containing N entities? C++, like many other programming languages, has better support for containers than for variable argument lists. Listing˚3-11 illustrates this. 1. Trying to solve that issue in Java using polymorphism is left as an exercise to the reader. This should illustrate even better how inadequate object-oriented programming is for materializing this concept.
42
Concept Programming in XL
Translating concepts LISTING 3-11: Maximum of a container in C++
1 template 2 T max(std::vector v) { 3 T result = v[0]; 4 for (int i = 1; i < v.size(); i++) 5 if (result < v[i]) 6 result = v[i]; 7 }
Replacing the original concept with another introduces unwanted side effects. For example, the user needs to create a vector to pass as an argument. Corner cases behave differently: the maximum of no element is not defined, but Listing˚3-11 will not allow the compiler to detect the case, resulting in a program that will fail in sometimes spectacular ways, that is, by throwing an exception, which might be spectacular enough if this code happens to control the guidance system of an Ariane 5 rocket. Concept-oriented language. The examples we gave so far were not intended to denigrate other languages, but to illustrate how difficult it can be to materialize even the simplest concept. One important objective of a concept-oriented language like XL is to remove as many barriers as possible to the materialization of a wide range of concepts. This implies that the representation of concept has to be useful, easy to use, and have other programmer-friendly properties. This is easier said than done, and it will take most of this book to explain how XL achieves this objective.
3.2.3. Domains Materialization relies on the principle of equivalence. Once we have established an equivalence between 3 and its code representation 3, we can almost ignore the difference. But each concept representation is only an approximation of some kind, usually limited by the implementation and the tools being used. As a result, the principle of equivalence is valid only on a limited domain called the domain of equivalence. This is illustrated by the program in Listing˚3-12, which is a C equivalent to the program in Listing˚2-5, except that it displays factorials up to 35. At least, that is what a reading of the program suggests at first sight. Why? Because that particular reading relies on the principle of equivalence holding true
Concept Programming in XL
43
Concepts?
1 2 3 4 5 6 7 8 9 10 11 12 13 14
LISTING 3-12: Displaying factorial values in C #include int fact(int N) { if (N == 0) return 1; return N * fact(N-1); } int main(int argc, char **argv[]) { int i; for (i = 0; i < 36; i++) { printf("%d!=%d\n", i, fact(i)); } }
Unfortunately, in that specific case, the trust in the principle of equivalence is misplaced. Listing˚3-13 is one possible output of the program1. LISTING 3-13: Output of the program in Listing 3-12. 0!=1 1!=1 2!=2 3!=6 4!=24 5!=120 6!=720 7!=5040 8!=40320 9!=362880 10!=3628800 11!=39916800 12!=479001600 13!=1932053504 14!=1278945280 15!=2004310016 16!=2004189184 17!=-288522240 18!=-898433024 19!=109641728 20!=-2102132736 21!=-1195114496 22!=-522715136 23!=862453760 24!=-775946240 25!=2076180480 26!=-1853882368 27!=1484783616 28!=-1375731712 29!=-1241513984 30!=1409286144 31!=738197504 32!=-2147483648 33!=-2147483648 34!=0 35!=0
1. This particular output corresponds to a machine where int is represented using 32 bits.
44
Concept Programming in XL
Pseudo-Metrics
It is easy to demonstrate that the mathematical factorial, being the product of non-zero positive integers, is always a non-zero positive integer. Why, then, is the output in Listing˚2-6 showing negative or null values? Many programmers will give you the following answer: integer overflow. In other words, the program fell outside of the domain of equivalence between the mathematical concept of integer and the C code representation called int. I have been very careful to distinguish between the concept and the implementation, that is the representation of the concept as code. I actually went as far as writing in regular font the value 0 to distinguish it from the value 0 as used in the program. The code is only a representation of the concept, not the concept itself. The integer type is not strictly equivalent to mathematical integer concept. In particular, there is no such thing as the largest integer, but there is a largest integer. When this maximum value is reached or exceeded in a computation, the behavior of the code representation (integer or int) will diverge from the original concept. At this point, the equivalence is broken. We fell outside of the domain of equivalence. Robustness Domain. In "Object-Oriented Software Construction" [4], Bertrand Meyer
defines correctness as the ability of programs to behave correctly under normal conditions, and robustness as the ability of programs to behave correctly even under abnormal conditions. The domain of equivalence is the domain where the program is correct. There is a larger domain where the program still performs correctly, and this is the robusness domain. The implementation of integer operations in XL normally detects overflow situations and reports them as runtime errors. The output in Listing˚3-13 could be produced by an XL program only if that mechanism was disabled1. The domain of equivalence of XL integer arithmetic is limited to operations that do not overflow, but thanks to overflow detection, its robustness domain covers the whole range of possible inputs to arithmetic operations. If possible, an implementation should try to detect if it falls outside of the domain of equivalence. This does not expand the domain of equivalence, nor does it help staying within it. That ultimately remains the responsibility of the programmer. But detecting if equivalence is broken avoids undetected incorrect behaviors in programs. In other words, it improves the robustness of programs by expanding the robustness domain.
3.3. Pseudo-Metrics All the steps involved in turning ideas into code are difficult. In order to guide the process and make sure that we know how much the principle of equivalence holds, we need ways
1. This may be desirable in some cases, for instance to improve performance.
Concept Programming in XL
45
Concepts?
to evaluate what we are doing, some sort of measurement, some sort of metric to evaluate the distance between concept and code. The problem is that the concept space cannot be formalized well enough for any kind of real metric to be defined precisely. Instead, we have to resort to some kind of handwaving, pseudo-metrics that relate the concept space and the code space. The most important pseudo-metrics for concept programming are the following: • Syntactic noise evaluates the difference in appearance between concept and code. • Semantic noise evaluates the difference in meaning between concept and code. • Bandwidth measures how much of the concept space is covered by the code. • Signal/noise ratio measures what fraction of the code space is used for concept representation as opposed to code-centric concerns. Even if they are only approximately defined, these pseudo-metrics are quite useful in identifying a number of potential and actual issues in the code. But since they can t be defined in a formal way, programmers may disagree on the actual value of a given measure, or even on the relative ranking between two implementations. Just as with music, what is noise to one person may be music to another. And just as with engineering, reducing noise is a worthy objective, but it is never possible to completely eliminate it. This is worth keeping in mind as we start studying pseudo-metrics. Syntactic Noise. Syntactic noise measures the presence in the code of notations and other
elements that do not represent any concept. These elements introduce a difference in appearance between the code representations and normal notations for the concept. This is noise because these differences need to be filtered out to access the actual concept-related information. Often, syntactic noise is the consequence of limitations of the tools being used. A good example is the semicolon ; at the end of statements in C. This character has no equivalent in the concept space. It is necessary in C only to parse C source programs. Strictly speaking, it still exists in the concept space of the C compiler: we can describe it as the character delimiting the end of statements in C . But this is a concept related to C itself, not anything useful to the majority of programs written in C. Syntactic noise introduces elements in the code that have no equivalent in the concept space. It makes the code more difficult to write, since tool-specific syntactic elements need to be learned. It makes the code more difficult to read, since the unwanted elements must be filtered out. Unfortunately, there will always be a minimum amount of syntax that needs to be learned in any case, and consequently there will always be a minimum amount of syntactic noise. It remains a worthy goal to reduce it as much as as possible. Syntactic noise is not always caused by limitations in the language. Improper use of the tools can also introduce noise. For instance, programmers who are not too confident in their
46
Concept Programming in XL
Pseudo-Metrics
recollection of operator precedence in C such as myself sometimes add useless parentheses. This kind of syntactic noise is often not too difficult to avoid or reduce. Semantic Noise. Whereas syntactic noise measures the difference in appearance between
the concept and its code representation, semantic noise measures the difference in meaning. It indicates the presence of elements in the code that do not behave like the concept they represent. This is another form of noise, because these differences also need to be filtered out before getting to the original meaning of the concept. Like syntactic noise, semantic noise can be caused by limitations in the tools, or by some unfortunate choice made when materializing concepts into code. An integer overflow is a good example of semantic noise caused by a limitation in the tools. There is no such thing as an integer overflow in the concept space. So the use of integer introduces some semantic noise. The programmer needs to pay attention to a characteristic of the implementation that does not exist at all in the concept space. If you have, like some languages, an integer type that grows as needed, then the overflow limit will be much higher, but there is still semantic noise: programmers need to consider memory usage and performance of large integers in their design and algorithms. Signal/Noise Ratio. In concept programming, the signal is all the information about the concepts that the code can carry, as opposed to overhead in the code such as syntactic noise or semantic noise. The signal to noise ratio measures how much signal there is in the code as opposed to noise, how much of the code is actually dedicated to representing actual concepts rather than code-centric concerns that are irrelevant in the concept space.
It is desirable to increase this ratio, as doing so indicates that a higher fraction of the code is dedicated to useful information. In general, higher levels of abstraction tend to increase the signal to noise ratio significantly. The foreach loop of C# has a better signal-to-noise ratio than the normal for loop as it was available to Java developers originally. Bandwidth. Two representation of the same concept do not necessarily have the same domain of equivalence. The bandwidth is a measure of what fraction of the concept space is accurately represented in the code. A concept representation is said to have higher bandwidth if its domain of equivalence is larger .
For instance, the XL Maximum implementation in Listing˚2-7 has arguably a larger bandwdith than the C implementation in Listing˚3-8: the XL code deals with any number of arguments of any type with a less-than comparison, whereas the C code deals only with two arguments of the int type. Impact on Language Design. The design of XL attempts to optimize the concept pro-
gramming pseudo-metrics in a large number of situations: reduce the amount of syntactic and semantic noise, increase the signal to noise ratio and increase bandwidth. Rather than attempt to list examples here, which probably would not make much sense yet, I will use the pseudo-metrics just defined when explaining the rationale behind each feature of XL.
Concept Programming in XL
47
Concepts?
3.4. Concept Mismatch Pseudo-metrics are useful to evaluate how closely and how extensively the code represents the concepts. But it is sometimes useful to classify cases where the concept and the code no longer match. In general, this is called a concept mismatch, but there are a number of interesting special cases worth knowing about. Concept Casts. A concept cast happens when, instead of implementing the original concept, the programmer implements a variant, a derived concept, or some concept that can be translated to the original concept one way or another. Concept casts typically occur because the original concept cannot easily be translated into code using the development tool of choice, and so an alternate representation is chosen instead, which materializes more easily.
Listing˚3-11, Maximum of a container in C++ , showed a very typical example of concept cast. Since C++ does not have support for type-safe variable argument lists, but has good support for containers, it is not possible to directly code an equivalent of Listing˚2-7 implementing the maximum of any number of elements, but it is easy to write code that computes the maximum of the elements in a container. A concept cast in itself is not bad. One could argue that selecting the right concept casts is an essential skill when materializing concepts. For instance, when we use an integer, we essentially do a concept cast between the mathematical integer type, which has no upper nor lower value, and the simpler to implement concept of integer in a limited range . This kind of conscious concept cast is a necessary evil when materializing concepts. But a concept cast always comes with a price, primarily in the form of semantic noise, since there is a difference in meaning between what is written in the code and what the programmer intended. The real problem is when a programmer uses a concept cast without realizing it, or worse yet, get somehow convinced that the concept finally implemented was the originally intended one. In the rest of this book, when we talk about concept casts, we will only refer to the unconscious or unwanted kind. In our example, if the C++ programmer starts to believe that the original concept is the maximum of a container, it will be impossible to detect all the semantic noise introduced by the concept cast, like the need to create containers, or the appearance of artificial boundary conditions such as empty containers. Domain Errors. A domain error happens when code starts executing outside the domain of
equivalence, in particular if this is not detected. An integer overflow is a good example of domain error. A domain error, like a concept cast, introduces semantic noise, since the code starts behaving in a way that no longer matches the original concept. The difference with a concept cast is that the concept being represented is the correct one, at least within the domain of equivalence, and that the problem arises only outside this domain.
48
Concept Programming in XL
Concept Mismatch
A domain errors does not necessarily mean that the input is out of range. For example, a rounding error in floating-point arithmetic is a domain error. The equivalence is broken in a detectable way, even if the input to floating-point operators is within range. Overspecification. Overspecification happens when the code adds constraints on its behavior that are not imposed by the concept space structure, nor by known limits of the implementation. Overspecification might seem innocuous, but it introduces semantic noise in the form of arbitrary limitations or unnecessary complexity. In practice, it can cause software failure when none should have happened.
Overspecification may happen at the interface level. For instance, the domain of equivalence of the Factorial function in Listing˚3-12 is integers between 0 and 12 inclusive. If we rewrite the function as shown in Listing˚3-14, indicating that the input is in range 1..3, then the function is overspecified. LISTING 3-14: Overspecified Factorial function
1 function Factorial(N : integer range 1..3) return integer written N! is 2 if N = 0 then 3 return 1 4 else 5 return N * (N-1)!
The upper bound means that the input value 4 will not be accepted, even if the code would compute the correct result 12. The lower bound is even worse, because it prevents the algorithm from working properly: in order to compute the correct result, at least one call to Factorial(0) is necessary. This is an example of unnecessary failure caused by overspecification. Overabstraction and Underabstraction. There are other kinds of concept mismatch where the limit is not the implementation, but our capability to understand it. Overabstraction is a representation in the code space that hides or ignores useful information, making it difficult to reconstruct or understand the original concept. On the contrary, underabstraction is a representation in the code space that shows too much information, overwhelming us to the point where we can t recognize the concept.
Listing˚3-15 shows an example of gross overabstraction for the factorial. This case is not very subtle, since it never works. The concept being represented is the right one, it is just not implemented correctly because a relevant part of it is not taken into account. Code that forgets to take an important limit case into account is unfortunately all too common. LISTING 3-15: Gross overabstraction of the factorial
1 function Factorial(N : integer) return integer written N! is 2 return N * (N-1)!
It is so common that the code we showed so far for the factorial, as shown again in Listing˚3-16, has a more subtle variant of overabstraction. What happens if the function is given a negative integer? In order to simplify the concept, we ignored how the function behave in a good half of the concept space!
Concept Programming in XL
49
Concepts? LISTING 3-16: Subtle overabstraction of the factorial
1 function Factorial(N : integer) return integer written N! is 2 if N = 0 then 3 return 1 4 else 5 return N * (N-1)!
The code in Listing˚3-17 is an example of underabstraction for the factorial. Technically, it works, but the concept is obscured. LISTING 3-17: Underabstraction for the factorial return integer is 1 1 2 6 12 60
1 function F(N : integer) 2 if N = 0 then return 3 if N = 1 then return 4 if N = 2 then return 5 if N = 3 then return 6 if N = 4 then return 7 if N = 5 then return 8 ...
Extreme examples like this are infrequent, but more subtle variants are commonplace. Underabstraction causes syntactic or semantic noise: the represented concept is not visible in the code, buried in implementation details. Mixed Metaphors. It is usual and useful to mix metaphors in software. We all use a desktop where we manipulate icons, files, menus and windows using a mouse. Outside of a computing environment, this description would not make sense at all. Mixing metaphors is somewhat necessary: each metaphor has a limited domain of validity, since the code space can only represent a subset of the concept space. One could imagine an alternate jargon for our common computing environment where in a room we manipulate coverpages and documents using a hand. That would be superficially more consistent, but finding desktop-oriented equivalents to windows and menus seem a little more difficult.
In program code, mixing metaphors is also necessary. When we create a payroll program, we have at the very least three concept categories to care about: employees, accounts and printouts. It simplifies our life as programmers quite a bit to talk about printing an employee rather than printing the employee record. It is not hard to find other examples of this kind of mixed metaphors: drawing a window (which most programmers understand as drawing the contents inside the window), killing an editor (which describes the termination of a process running a text-editing program, not some murderous intent against a member of the press), and so on. Like concept casts, mixed metaphors introduce semantic noise, and so they must be introduced carefully and deliberately in the code. The risk of mixed metaphors is one of the primary arguments used against function overloading. In "Overloading vs. Object Technology" [5], Bertrand Meyer states that different things should have different names . That s the whole problem with this argumentation: with respect to Listing˚2-7, Computing the maximum , I pointed out that the two function declarations for Maximum are really two aspects of the same function. Similarly, two Add functions implementing addition for integer and for real numbers are arguably two aspects of the same thing .
50
Concept Programming in XL
In Conclusion
On the other hand, a Sub function subtracting two values and another returning a submarine object are not the same concept at all. If they coexist in the same body of code, there is a high risk of confusion. The problem is not with overloading, which is quite useful and necessary from a concept-programming point of view. Without overloading, we would be forced to add syntactic noise by renaming some aspects of a concept. The problem is really with mixed metaphors used inappropriately.
3.5. In Conclusion In this chapter, I have given an overview the methodology behind XL, concept programming. I have defined some useful terminology, such as domain of equivalence, concept cast or semantic noise, that I will keep using regularly in the next chapters. I have also given an outline of the process used to write a program in the concept programming methodology, which I may sometimes refer to when explaining the rationale behind this or that design choice of XL. In the next chapter, we will define more precisely the problem that concept programming attempts to solve.
Concept Programming in XL
51
Concepts?
52
Concept Programming in XL
In Conclusion
Chapter 4 — The Trouble with Programming
(= (D, E)) // Then increment a // and regardless of the test result, // set b to 0 T = 0;
Notice how, in this code, lines that look extremely similar have vastly different meanings. This makes the code quite difficult to understand for the average C++ programmer. The confusion results from a severe case of mixed metaphors: the same code structure doesn t mean the same thing at all from one place to the next. As an aside, if you ever apply any of these techniques, please do not mention that you learned them from this book You can check for yourself that this code is indeed valid C++ using the on-line version of the Comeau C++ compiler [11]. This compiler is based on the high-quality Edison Design
62
Concept Programming in XL
Artificial complexity
Group C++ front end [12], arguably the only compiler around that really understands all of C++. As a result, it compiles the code with a single warning (d value being unused). On the other hand, this same code fails to compile correctly on most compilers. The compilers are at fault, since the program is valid C++ (even if it is an extremely bad coding style). But for the programmer, it s all the same: the program doesn t compile. The error message emitted by the version 2.95 of g++ is particularly enlightening, as shown on Listing˚4-8. Confused it is, indeed. LISTING 4-8: Error messages from g++ 2.95 on this code c++_for_dummies.cpp: In function `void f()': c++_for_dummies.cpp:12: parse error before `)' c++_for_dummies.cpp:14: confused by earlier errors, bailing out
4.3.4. Standards Aren’t Well outside of the control of the typical programmer, external standards also define the behavior of your software. One might think at first that standards are better defined than the typical program. This would be ignoring the massive power of destruction of what is often called Design by Committee . Design by committee is what happens when a few talented individuals get together to design something complex. Ego and politics rapidly take over. Standards are generally designed by committee, and this leads to a few interesting properties: • Standards are built as aggregates, by individuals with often conflicting interests. If there are two solutions to a problem, then the two solutions often become part of the standard. • As an extension, most standards offer at least two solutions for every single problem. In other words, standards ensure that there is no standard way to do things. • While pieces of the standard have generally been tested ahead of time, the standard as an aggregate is generally well ahead of its implementations. Sometimes, implementing the standard reveals interesting problems that nobody had foreseen. • Thus, standards need to evolve, and to answer the questions raised by previous versions of the standard. This evolution maintains the standard in a state of flux, making it a moving target, in direct conflict with the expected uniformization. Interestingly, many of these points remain valid even for de facto standards. Politics and egos do exist inside corporations as well. Microsoft Windows, arguably the most important de-facto standard in the computing industry today, is a very good example of an API that aggregates numerous methods to do the same thing (GDI vs. DirectX, Win32 vs. Win16), had key elements dropped after they were found to be impractical (the old Program Manager, or the NT 3.51 protected graphic drivers architecture), and is a perpetually moving target (Microsoft admitted that only about 60% of Windows NT4 applications should run well on Windows 2003).
Concept Programming in XL
63
The Trouble with Programming
4.3.5. Lies, Damn Lies and Documentation I talked earlier about documentation. While writing documentation for users is generally an ordeal, imagine what it is to read documentation intended for programmers! The Unix manual page for xauth shown in Listing˚4-3 is by no means an exception. One important source of documentation for programmers are the comments in the source code. The problem with comments is that, using today s technology, compilers can t verify that they tell the truth. As a matter of fact, comments tend to get neglected over time, and to lie more and more as the code gets modified while the comments aren t. There is even a school of programming that tells that comments are bad, because they always get out of sync eventually. Their motto: The source code is the documentation . On the surface, this appears logically consistent, much like suggesting that the best way to prevent a patient from getting ill is to kill him. Naturally, the good solution is to keep the comments consistent with what the code does. But that is not easy, and it is rarely done well. So experienced programmers read the documentation, but they take it with a grain of salt. It is not uncommon to find design documents that are 5 years old for software that is modified daily. All sources of documentation can lie, including the comments right above the instruction which you are reading in the source code, or the on-line help of the program, or even its error messages. A simple rule: trust no one!
4.4. Business Complexity A last and important source of complexity in day-to-day programming work is the business environment in which the programs are being developed. This environment defines deadlines, budgets and other sources of constraints and pressure for developers.
4.4.1. Requirements: Everything, Twice, a Lot, with Sauce A friend of mine, when asked what he wants to eat, invariably responds: everything, twice, a lot, with sauce . This turns out to be the stance adopted by most users with respect to the requirements for their software. This used to be true of the military scientists 50 years ago who wanted to fully exploit the (then) mighty power of their appartment-sized pocket calculators. This is true today of young gamers who insist on running their favorite games at 270 frames per second, ten times the amount their eyes can actually deal with. Many programmers see users as the only serious trouble with programming. Users are this bunch of idiots who can t even read the manuals, let alone the source code. If you doubt how much programmers despise users, listen to the jargon of operating-system kernel developers, who are generally regarded as ber-programmers among their peers. If you lis-
64
Concept Programming in XL
Business Complexity
ten to these alpha programmers, they spend their days talking about user-land, user-level demotion, non-privileged user, or user access control. And they don t even think about it, it s simply the basic jargon of their community. But users are good for us. Really. They keep the pressure on us programmers. They define how and when progress happens. For instance, programming languages evolved to meet the ever increasing needs of users, not programmers. The driver for change was not programmers themselves. Contrary to their own belief, programmers are a lazy bunch of strongly conservative people. Changing their habits is extremely difficult. Think about it: when you heard first about this new incompatible programming language named XL, what was your very first thought? Yeah! I ll use that right away , or instead Why the heck would we need a new language? Today, the most common language is C, which was invented over 30 years ago. And the next most popular languages, notably Java and C++, are almost always C derivatives. Believe me, programmers did not push the change in programming languages. Users did. How did that happen? First came the new computer power, in application of Moore s law. Users immedidately wanted to use that new power and maximize their bang for the buck . So new application domains emerged one after the other: mathematical computations for all sorts of military uses, business and payroll computations for large corporations, varied computations for smaller teams, user interface for home users, and finally the Internet. Each of these application domains introduced a new kind of complexity, and successively bred languages adapted to the new kind of applications (namely Fortran, Cobol, C, C++ and Java). Such radical changes in the kind of applications are called paradigm shifts . Hold that thought: I will get back to paradigm shifts a few times. But these radical changes are all but one example of the kind of pressure that users put on programmers. On a more constant basis, they simply ask for the best of what can be done with the hardware that they have, and in the business environment where they evolve. Everything, twice, a lot, with sauce .
4.4.2. Mission: Impossible Naturally, when users ask for the impossible, they won t get it, right? Wrong. Experienced programmers know that impossible missions are part of their daily life. Indeed, impossible missions are exactly the kind of experience that makes them experienced in the first place. Often, programmers believe that this should not be the case. They may blame the marketing development for overhyping a future product. They may blame the sales rep for promising a yet-to-be-invented feature. They may blame the project manager for setting impossible deadlines. They may blame the contract department for signing something without checking with them first. In the end, they often get the feeling that they keep fixing up problems that could have been avoided.
Concept Programming in XL
65
The Trouble with Programming
But any serious programming is first and foremost a research job. Every team needs to chart new territory that has rarely, if ever, been explored before. If the product that the users want was available on the shelves, chances are that they would buy it. If you need to develop something new, it often means that it has not been done yet. And charting this new territory is exactly what makes programming so interesting and so fun. But the consequences for programmers are simple: they need to predict the future, and that s a difficult thing to get right. We are asked to accurately predict the schedule, budget, resource requirements, and various other key characteristics of a project, although the specifications are unclear, the target market still nonexistent, the necessary technologies only nascent in some remote laboratory, and your team a bunch of total strangers. Yet we do give answers, based on our best guesstimates. Sometimes, we get it reasonably right, at least considering a reasonably broad definition of reasonably . More often than not, however, the result is pathetically laughable in retrospect at least. I invite all programmers to retrieve some of the early e-mails exchanged about a project they recently completed. In general, it makes for some good fun to see how stupid we were when we were younger. Didn t we know any better? Did we really agree to do that?
4.4.3. Costs and Delays: The Really Mythical Man-Month Of all important characteristics of a software project, the most important to managers and other business makers are costs and delays. The difficulty of evaluating this is legendary among programmers. The illusion than one can predict software delays and costs even has a name: "The Mythical Man-Month" [13]. The name man-month itself implies a difficulty: while the month is a well-defined unit of time, the man is not a well-defined unit of work-per-month. The quantity of quality of the work depend enormously on the person itself, and on the work to be done. It varies over time for a single individual. It depends on totally unpredictable factors such as the person s familial environment or health. So when we give the guesstimates to management, we base them on a set of implicit assumptions such as the kind of programmers we will get, or sometimes even which programmers precisely will be working on the project. We assume, hope or pray that no unplanned event will put any of these individuals off trajectory with respect to the project. We make evaluations based on what we know of the project, sandbag a little to give a little margin for errors, and there you have it.
4.4.4. Programmers: Interchangeable or Indispensable? There are two important variant of the man-month myth that causes untold damage: the myth of interchangeability of programmers, and the myth of their indispensability.
66
Concept Programming in XL
Business Complexity
The myth of interchangeability exists mostly among managers. The man unit, as we said, is not a very precise unit of work per month. Yet managers often do as if any programmer could replace any other on any project at any time. Sometimes, they just have no choice, maybe because one member left the team. But there are cases where this causes sometimes fatal self-inflicted wounds to otherwise healthy teams. This occurs more frequently in large corporations, where bureaucracy and policies from above tend to be more blind to the needs of individual teams. In some cases, this attitude even becomes a conscious business decision. Many startups burn out their programmers quite deliberately. In complete contrast, programmers tend to believe in their own indispensability. And sometimes, they orchestrate things so as to be indispensable, an activity that we half-jokingly call preserving job security . Such programmers are sometimes referred to as divas . Divas simply believe that the knowledge they hold the keys to somehow gives them some special rights. They tend to avoid sharing that precious key knowledge with anybody, by fear of losing their power. This is why a diva, regardless of her technical merit, is not necessarily a very good programmer: divas do not realize that they need the team as much as the team needs them, that shared knowledge fructifies and empowers more in the long term. At the opposite end of the spectrum, the culmination of the ideal of sharing ( share and share alike ) is the Free Software movement. Nobody is indispensable. Yet nobody is replaceable, a startup CEO once told me when I resigned from his company. We left in very good terms, and his approach seems very wise to me. But then, he s not nearly as rich as another startup CEO I met when I was young, who ate programmers for breakfast, and whose company went on to become a dominant player in the videogames industry.
4.4.5. Looming Deadlines and Death March After having given their best guesstimates, programmers have a serious problem: their management holds them to it. In very infrequent cases, success just happens. Normally, however, all sorts of events and forces of nature immediately conspire to delay the project left and right, and otherwise doom it to failure. In the initial phase, everybody in the team works at a good, but relaxed peace. As pressure mounts however, the pace becomes more sustained, and the programmers begin to tire, make mistakes, and sometimes even look for other jobs. The project changes a lot compared to initial projections, and many unexpected problems arise. All too often, the project finally slips past deadline after deadline, and exhausted programmers work frantically to finish what has slowly become an impending failure. When this happens, the project has turned into a "Death March" [14]. This is not a pretty sight. When programmers think about pressure on the job, they most often think about a death march.
Concept Programming in XL
67
The Trouble with Programming
4.5. The Grim State of Software Quality All the complexity would not be such an issue if we were still capable of producing highquality software. But we are not. Mediocre software quality is a direct consequence of our inability to deal with the complexity of the task.
4.5.1. Software Quality: An Oxymoron? The software industry has an abysmal track record in quality control. One of the oldest jokes in the industry is that if automobiles had followed the same development cycle as the computer, a Rolls-Royce would cost $100 today, get a million miles per gallon, and explode once a year, killing everyone inside1. In no other industry is there such a tacit understanding between the vendor and the buyer of a product that the product doesn t actually work. The software industry, on the other hand, not only routinely delivers defective products, it has built a whole business on the foggy notion of software upgrades , which all too often add bugs and unwanted functionality without fixing the existing problems. Beta-testing, the practice of having customer test products that are still in development, is no longer an exception. It has become so commonplace that many customers are now willing to pay for beta quality software. Maybe they simply have little choice: so many software products were released with beta quality that you might as well save time and money by accepting the fact. This is a very important issue, say if you travel in a fly-by-wire plane, or if your food and pharmaceuticals are produced in computer-controlled plants, or if the gas mileage and emission levels of your car depend on the proper operation of its software, or if you are about to score an all-times world record at Pac-Man. In any of these cases, you really don t want the software to misbehave, your life depends on it. Many people believe that software problems plague only off-the-shelf software, but not software used for life-critical situations. This is just not true. The first Ariane-5 european rocket, a half-billion dollars project, had to be destroyed in flight following the failure of onboard software due to a very routine out-of-range error. But stories like this are quite frequent for who wants to listen: fighter planes flipping upside-down at the equator because of a sign error in the fly-by-wire software; computer-controlled X-ray machines overdosing their patients; delays in airports while the arrival and departure screens showed nothing but a Microsoft Windows crash screen
1. The original quote is generally attributed to Robert X. Cringely.
68
Concept Programming in XL
The Grim State of Software Quality
4.5.2. Software Licenses: Shameless Self-Absolution When faced with the prospect of being held liable for delivering shoddy products, many companies in other business fields adopt a reactive approach: they take the hits as they come, and adjust the best they can. The software manufacturers, on the other hand, took a very proactive and innovative approach early on: software license agreements. Software license agreements are a unique kind of one-way contract, giving the vendor all sorts of rights, while waiving most, if not all responsibilities. Customers have the unfortunate habit to not read them, but they should. The kind of rights that software vendors grant themselves grows at an alarming rate, while the liabilities that they recognize decrease almost as fast. Some recent licenses contain gems like: • they give the vendor the right to use your computer (at no charge, naturally), • they deny you the right to use the software you purchased (most commonly, after you purchased a new computer), • they forbid you to use the software to criticize the vendor, • they forbid you to publish benchmarks and other measurements made with the software, • they grant the vendor rights to personal data recorded by the software, • they force you to pay multiple times for the same product or service, • and the list goes on. Software license agreements do not improve software quality. But in a sense, they fix the problem, or at least hide it conveniently under the carpet. They are one of the key forces explaining why users were so-far unable to get decent quality from software.
4.5.3. Boiling Frogs If there are blatant flaws in the products that the software industry delivers, why don t we react more strongly? As we trust more and more aspects of our daily lives to computers, we should be very alarmed by this situation. Yet, we have come to accept it as normal. There are three main reasons: • The first reason is the boiling frog syndrom. When you put a frog in boiling water, it jumps right out of it. But if you gradually bring the water to a boil, the frog will not realize what s happening 1. The same has been true with software quality. The situation is indeed intolerable in more than one way, but we have come to accept it because it has been imposed on us gradually. • The second reason is the perception that software engineers are poor saps trying hard to fix the bugs. Anybody who has been in a software development team knows that there is 1. I cannot testify to this fact myself. It might as well be an urban legend, for all I know. Like most of my compatriots, I prefer my frogs sauteed in a light butter-garlic sauce. The idea of boiling frogs must have occured on the other side of the channel.
Concept Programming in XL
69
The Trouble with Programming
nothing farther from the truth. Programmers often don t care about fixing bugs because it s not as exciting as adding new features. Managers don t care about fixing bugs because it doesn t improve the bottom line as much as new features. • The third reason is a sense of inevitability. You can t do anything about software quality. Every software has bugs. So you just have to live with it. But while it is true that there will always be minor defects, like in any manufactured product, there is really nothing inevitable to major design or security flaws in software. The current state of software quality is something that customers don t like. And when customers don t like some aspect of a family of products, there is an opportunity for newcomers to jump in and fill the gap.
4.5.4. Cars Don’t Rust Anymore To a large extent, the situation of the software industry is similar to the situation of the car industry in the 1970s. Both industries started with a boom, and everybody wanted to buy the products. Simply renewing the models from year to year was enough to bring a continuous stream of cash. People regarded their old 1995 personal computer as totally outdated as soon as they laid their eyes on the shiny new 1996 model of the same manufacturer (hey, look, it has a Pentium Pro, and 16MB of RAM too!), just like forty years earlier thay regarded the Corvette 1955 as obsoleted by the shiny new 1956 vintage (hey, look, the V8 is standard now, and 210HP too!) It worked for software as well, going as far as branding Microsoft Windows based on the manufacturing year! But we know what happened next to the automotive industry. The quality of cars was not too good. People thought then that car rust was part of the natural world order, just like people think today that software without bugs cannot exist. Cars then, like software now, competed on price, features and looks, not on quality. In the end, this situation became a perfect opening for some manufacturers, notably the Japanese car industry, to establish first some beachheads, then strongholds, and to finally bring even the largest behemoths of the american automotive industry to their knees. It is too early to tell if history will repeat itself, but it is worth noting that the main competitors to today s software giants, notably the Open Source and Free Software movements, compete at least as much on quality as they do on other factors such as price.
70
Concept Programming in XL
The Grim State of Software Quality
Chapter 5 — From Concepts to Code
If I have seen further, it is by standing on the shoulders of Giants (Isaac Newton)
Understanding the problems programmers face is futile if it doesn t help us solve them. Before we start exploring the particular answers that XL and concept programming have to offer, we can start with more general considerations and observations that are valid with practically any programming language. Concept programming was not born out of a vacuum. I did not wake up one day, thinking Why don t I invent a new programming paradigm today? Instead, concept programming slowly matured as a formalization of various techniques I had learned and developed over time. It was my attempt to address issues that were in my opinion not solved correctly by object-oriented programming, or C++ templates, problems that typically required the use of multiple programming languages addressing different facets of the solution. But it builds on all these existing techniques. XL stands on the shoulders of giants. The first step is to explain what abstractions are, how programming paradigms were built around them, and how they address the various forms of complexity we discussed. Finally, I will discuss a few practical considerations when turning abstractions into code.
Concept Programming in XL
71
From Concepts to Code
5.1. Turning Ideas into Code As I explained in Chapter 3, programmers need to to remember that there is a gap between the concepts and the code. This gap will never go away, no matter how sophisticated your development tools and programming techniques would be. The concepts remain in the concept space, which is held inside your brains, in a way that today s medical science cannot fully explain yet. The code resides in the code space, which begins in our heads as well, but is intended to ultimately be materialized in a computer. This distinction may seem really trivial, but it is not. There are countless errors and bugs that are rooted in some incorrect assumption that the code is equivalent to the concept it represents. Many of us know the old joke about the customer closing the window of the room he was sitting in after the phone support technician asked him to close the window . To us computer folks, this is funny because the support technician clearly thought about closing the frontmost application window. But this really exposes the kind of disconnect that can exist between the concept space and code space. How do programmers bridge the gap? They use computerized abstractions for common concepts. Development tools are the first and most important source of abstraction for programmers. They offer us convenient, symbolic representations of the zeroes and ones that ultimately every program boils down to.
5.1.1. Seven Levels of Abstraction Development tools did not happen all at once. Rather, the history of programming tools advanced by leaps and bounds. In that, they only followed the increasing needs of users and exploding complexity of computing problems. Correspondingly, increasing levels of abstractions had to be invented to write software dealing with these problems. Very roughly, one can identify seven successive different levels of abstraction: 0. Initially, there was no abstraction at all. Programmers were toggling switches which directly manipulated the state of the machine. This can be called the level 0 of abstraction. 1. Assembly language: Remembering the position of switches was easily overwhelming, and did not allow good communication among programmers. The first level of abstraction was assembly language, where a binary pattern such as 000010 was replaced with a mnemonic command such as ADD. 2. High-level languages: As computers became more widespread, the absurdity of recoding the same algorithms again and again for different machines became quite obvious. This spawned the creation of a second level of abstraction with FORTRAN (FORmula TRANslator), a nifty representation of programs that was largely independant of the underlying machine.
72
Concept Programming in XL
Turning Ideas into Code 3.
4.
5.
6.
7.
Structured programming: As programs grew, the structure of instruction sequences and data layouts became difficult to understand and to maintain. A new generation of structured programming languages, such as Pascal or C, abstracted the control flow and data structures. Modular programming: Interactions between software components became complicated enough that they needed to be abstracted as well. The result was called modular programming, and is well illustrated by Ada packages or the Modula series of languages. Modular programming generalized the notion of encapsulation, in other words hiding private implementation details behind a public interface. Object-oriented programming: In turn, data structures themselves soon became so complex that minor changes would cause widespread effects in programs. Object-oriented programming, as found in Smalltalk or Simula, added a level of abstraction around pieces of data, turning them into objects with well-defined interfaces, and hiding the actual implementation of the data structures. Most people now think of encapsulation as applying to objects. Generic programming: It became important to avoid recoding common algorithms, in particular when the only change was the type of data being processed. Languages with support for generic data types and subroutines such as Ada and C++ gave the programmers a way to abstract the description of the algorithms from the actual types. Generic programming enabled software components for common algorithms, as illustrated by the C++ Standard Template Library (STL), later folded into the C++ language itself. Metaprogramming: The ultimate step is the ability to abstract the program itself, to write programs about programs. This capability can derive from self-representation and self-evaluation as in Lisp1. More limited variants exist, like virtual machines that create an abstract representation of the program execution environment, reflection and introspection as a way to explore the data types in a uniform way, or C++ template metaprograms as a way to perform compile-time computations and optimizations.
5.1.2. Meta-abstractions Concept programming, as presented in this book, is methodology guiding the creation of new forms of abstraction. It applies to any of the abstraction levels listed above. In that sense, it can be thought of as a form of meta-abstraction, focusing on the generation of abstractions much like meta-programming focuses on the generation of programs. Whereas previous abstractions deal only with constructs in the code space, concept programming deals with the relationship between these code constructs and the concept space.
1. Interestingly, this makes the seventh level of abstraction one of the most ancient ones. It still remains the highest level abstraction, as illustrated by the fact that Lisp used its metaprogramming capabilities to incorporate new forms of absstraction as they were invented. For instance, Lisp became the first normalized object-oriented language. This was achieved using standard metaprograms (macros), rather than through fundamental changes to the core language as happened when moving from C to C++.
Concept Programming in XL
73
From Concepts to Code
As a consequence, concept programming languages such as XL do not highlight specific code constructs such as keywords, syntactic elements, objects, high-order functions. Instead, they are constructed around the more general mechanisms making it possible to build these code constructs from the original concepts. Naturally, the code representation will still make use of techniques from any of the known abstraction levels. But the principle guiding every other abstraction technique will be: does it look and feel like the application domain concept? In practice, this leads to the implementation of the Maximum concept we gave in Listing˚2-7, Computing the maximum . The existence of variable argument lists and a ... syntax doesn t really matter, what matters is how the code relates to a maximum concept as a mathematician understands it. This example gives us an early understanding of how concept programming can lead us to write markedly better code, even for very mundane and common algorithms. Other Kinds of Abstractions. Programming languages and development tools are by no
means the only kind of abstractions which software developers rely on: • Operating systems abstract memory, processes or input/output devices. • Software libraries and other software components hide the implementation details of most modern software technologies. • Standards and protocols hide the particular details of operation of differing implementations and allow them to coexist and cooperate. However, since concept programming is a form of meta-abstraction, we will discover that it can help with these other forms of abstraction as well. This will become clearer as we dig deeper into the implementation of XL.
5.1.3. Shifting Paradigms As was suggested in Section 4.4 on page 64, users define the need for better software. It is relatively straightforward to map the seven levels of abstractions to changes in the usage model of computers. When the new development tools were invented, they suddenly enabled a new class of applications. Each time, computers became capable of doing something that was not possible or practical before, and quite often, the new usage rapidly became prevalent. These major changes in computer usage are called paradigm shifts, and they map quite naturally onto our levels of abstraction: 1. Assembly language derived from the desire to use computers as practical tools, rather than mere laboratory experiments. 2. Formula translation followed the transformation of computers into reusable computing devices, rather than machines built for and dedicated to a single task.
74
Concept Programming in XL
Turning Ideas into Code 3. 4. 5. 6. 7.
Structured programming corresponded to the introduction of computers into the mainstream, first for large corporations, and later for individuals. Modular programming gained steam as software grew too large for any single individual developer to know all the details of every component in the system. Object-oriented programming became popular with graphical user interfaces, which were very difficult to program with earlier tools. Generic programming is a consequence of software turning into a mature engineering discipline more interested in software reuse. Finally, virtual machines and reflection were boosted by the rise of the Internet and the explosion of distributed applications across heterogeneous machines1.
An important observation at this point is that existing programming tools advanced by leaps and bounds. Whereas Moore s law induces a relatively steady growth in software complexity, development tools, on the other hands, progress discontinuously. This is illustrated on Figure 5-1. Paradigms and productivity. Each new paradigm introduces a significant boost in pro-
ductivity. For a period of time, programmers are ahead of the wave. They feel comfortable experimenting with new kinds of applications. As time goes by, however, software complexity keeps increasing and eventually catches up. When this happens, programmers can no longer deal with the complexity comfortably. Instead, they tend to take more time than planned and create programs with growing error ratges. Increased complexity that is not yet managed results in tedious processes and coding. Some programmers feel that something ought to be automated, but generally don t find the time. If they do, it is using ad-hoc techniques such as domain-specific languages (DSL). Ultimately, this results in software that becomes expensive. At some point, the cost becomes prohibitive and a new paradigm emerges. Predicting the future is a difficult task best left to psychics. However, it is reasonable to think that, at some point in our future, a new kind of paradigm will emerge. Paradigms in the making might disrupt the way we program. Here are a few examples of things that might happen in a near future because they are already beginning to happen: distributed computing; ubiquitous and mobile computing; speech-based user interfaces Concept programming is designed to facilitate the adoption of these new paradigms, by enabling the creation of new kinds of abstractions to incorporate each new paradigm as soon as it emerges. XL, in particular, is designed as an extensible language precisely so that paradigm changes can happen within the language, rather than require a complete change in the toolchain. Demonstrating that the approach works begins by using it to implement existing paradigms.
1. However, true metaprogramming remains rarely used even today. One possible reason is that the languages that really support it, like Lisp, are often shunned by programmers for a variety of reasons.
Concept Programming in XL
75
XL Concept programming
Soft (Mooware co m re’s Law) plexity
Complexity
From Concepts to Code
Problems
Tools
Tedious Slow Expensive
We are here
*
Java, .NET Virtual Machines
Comfortable Fast Cheap
C++, Objective Pascal Object-oriented programming
Pascal, C Structured programming
BASIC Fortran Interactive programming High-level languages Mainframe Microcomputers
Visicalc
GUI
Internet
Time
FIGURE 5-1: Evolution of Programming Tools
The Rise and Fall of Programming Paradigms. Paradigm shifts in the use of computers
cause corresponding shifts in development tools. However, it would be wrong to think that this is all natural. Like most people, programmers tend to stick to what they know. It takes a good deal of effort to migrate them to a new way of doing things. Therefore, they won t change tools simply because their users started doing something original with the newfound computer power at their disposal. Programmers change tools because they have to. We change ours habits because the old tools and techniques don t cut it anymore. We migrate only after it becomes painfully obvi-
76
Concept Programming in XL
Abstractions
ous that it is too difficult to use the old tools to satisfy the users needs. We adapt when procrastinating gets more painful than moving ahead. Two recent examples illustrate this quite well: • Java s success can largely be attributed to a failure of C and C++ to address Internetspecific needs. Yet you may remember people trying to create distributed applications using tools such as RPC or Corba. It s not impossible, but it s definitely painful by any measure. By contrast, Java offers a lot of ways to perform remote method invocations, to serialize objects, and more, that make it a better tool, qualitatively, than C or C++ for Internet applications. • Earlier, C++ had similarly displaced C for the development of graphical user interfaces, simply because C could not address the needs of user-interface developers satisfactorily. A fair number of programmers, however, still tried to write and rewrite event loops and menu command callbacks in non object-oriented languages such as C. This was painful, but the learning curve was smoother, since you did not have to understand all these fancy object-oriented design rules. It is important to stress that programmers don t switch to new techniques because it is impossible to develop the new applications using the old model, but because it becomes inconvenient. It is not impossible to develop an Internet application using Corba, it is simply difficult, and generally not competitive compared to alternatives. Similarly, many fine applications featuring a graphical user interface were developed in good old C, with the help of inordinate amounts of high-caffeine beverages. However, when a new category of user problems appears, ultimately the corresponding programming abstraction emerges, which makes it easier to deal with the original problem. It is reasonable to infer that, at some point, this will happen again. This is why it is important to understand what makes a good abstraction, and how we create them.
5.2. Abstractions An abstraction is a simplification of a concept, for instance a name given to a mathematical function. The abstraction is simpler to manipulate, it allows higher-level reasoning. Such abstractions are useful to programmers, because various mechanisms allow to replicate conceptual abstractions in the code space. For instance, our definition of Factorial in Listing˚2-3 began with a mathematical definition of the factorial in the concept space. We will call an abstraction mechanism any process turning a particular concept form into one of the possible abstractions of that concept. While Factorial is the representation of the corresponding factorial concept, and an abstraction of the factorial definition, it is based on a more general abstraction mechanism called functions . Functions are only one
Concept Programming in XL
77
From Concepts to Code
of many abstraction mechanisms available to programmers. Others include: types, macros, files, literals, and so on. Abstractions are useful in that they allow us to create a better, higher-level connection between the concept space and the code space. They help addressing the problems we explored in Chapter 4 more manageable, notably by reducing complexity: • they simplify the problem by hiding some secondary aspects from view, • they enable reuse by defining classes of equivalent problems with a single solution, • they foster diversity through the composition of small entities into larger ones, and • they improve scalability and power through layering.
5.2.1. The Secret of Simplicity: Hide to See More Abstractions are often not defined as much by what they offer as by what they hide. This is not surprising, since the main objective of abstractions is to simplify the representation of the underlying concepts. Consider for instance the problem of traversing a list to find a field with a given value, and how increased levels of abstraction hide more and more details: • Assembly language programmers must pay attention to the offet of the link in memory, the link representation, the size and alignment of the access to the link, the best instruction to access the link, which register the link will reside into while traversing the list, memory latencies, code scheduling, and a few thousands others mundane details • C or Pascal programmers only need to know the name of the link field and of the structure in which it resides, and how to adapt some boilerplate code to traverse the lists. C programmers often hide this boilerplate code behind macros. • Java programmers can borrow from a few pre-existing classes implementing the List interface, knowing little more than the class of the objects they want to put in the list. • C++ programmers take advantage of generic algorithms, which work identically for various container type. The same code can work with an array, a list, a vector. • XL programmers can write generic code that only refers to a container, and let the library select the best container type based on actual use. Increased levels of abstraction, by hiding irrelevant details, allow programmers to focus on the task at hand at a higher level and, consequently, be more productive. Once the necessary concepts have been learned, code with a higher level of abstraction is easier to design, to write, to read, and to maintain.
5.2.2. The Secret of Reuse: The Principle of Equivalence Programmers can use abstractions to represent complicated concepts because of the principle of equivalence I introduced in Section˚3.1.3. A code representation is considered equiv-
78
Concept Programming in XL
Abstractions
alent to the concept it represents. But interestingly, an abstraction is also considered to be equivalent to what it represents. The noteworthy consequence is that multiple representations of the same concept can also be considered to be equivalent to one another, since they are all equivalent to abstractions of the same concept. For example, a function call is equivalent to the expanded body of code executed by the call to that function. That makes it possible to use the function abstraction, and still maintain the equivalence between the code space and the concept space. The benefit is that the function representation is more manageable. The principle of equivalence is the basis for reuse of code in software. Programmers experience it firsthand when they start typing some code, then realize that they coded the same thing already somewhere else, and make a function with that shared code1. The programmer relies on the principle of equivalence to call the function instead of using the original expanded code, both representing the same concept.
5.2.3. The Secret of Efficiency: Implementation Properties The equivalence in the code only implies functional equivalence to an extent defined by the programmer or the language construct being used. Functionally equivalent variants may have vastly different properties or attributes for the programmers. A function call, for instance, is much shorter than the expanded code, and therefore simpler to manipulate. The function parameters make it easy to change an argument, while replacing the argument in the expanded code would be very difficult. Programmers should select an equivalent form with otherwise interesting properties that make it preferable. To most programmers, using the function call in place of the same expanded code repeated over and over will be considered beneficial2. Two code representations that are functionally equivalent may also differ with respect to implementation characteristics. Different implementations of the same function may have various performance properties, or various constraints on the valid range of their inputs. Listing˚2-4, Iterative definition of the factorial showed an implementation of the factorial, which is functionally equivalent to that of Listing˚2-3, but uses iteration instead of recursion. In general, recursive functions generate slower code than iteration, unless the compiler applies specific optimizations. On the other hand, many algorithms are simpler or only possible in recursive form. Both example implementations here shared many other characteristics, notably the same serious limitation on the range of inputs before the computation overflows the set of allowed integer values.
1. Concept programming would recommend to identify relevant concepts long before the coding phase. 2. To the proponents of the alternate programming style often called copy-paste inheritance , I recommend looking for another job where they don t make the life of their colleagues miserable
Concept Programming in XL
79
From Concepts to Code
Programmers need to find a representation that is not only functionally equivalent to the concept within the framework of the program, but also has good properties: mnemonic, efficient, maintenable, consistent, and so on. Only then will a powerful and useful equivalence exist in the mind of the programmers between the actual program entity and the related concept. The programmer can now trustfully use the program entity in place of the original concept, since the concept is conveniently, yet efficiently represented by that piece of code.
5.2.4. The Secret of Variety: Composition The transposition of a concept into an abstract program entity paves the way for the next step: composing abstractions. Abstract representations would not be very useful if they could only be reused within the exact same context. Instead, all programming languages offer a variety of composition rules that allow entities to be combined into a larger result. Listing˚5-2 shows how the factorial function defined in Listing˚2-3 can be used to compute the number of combinations and the number of permutations in a set. The two functions use the Factorial function in different ways, passing different arguments. The functions defined in Listing˚5-2 can now be used just like the leaf Factorial function, and represent different mathematical concepts. It becomes irrelevant that they both internally call the same Factorial function to perform their duty. Composition allows us to represent new concepts by combining the same elements in different ways. This is a way to increase our portfolio of software components. LISTING 5-2: Combinations
1 function Combinations(N, K : integer) return integer is 2 return N! / (K! * (N-K)!) 3 4 function Permutations(N, K : integer) return integer is 5 return N! / (N-K)!
In Listing˚5-2, the expressions in the code are written in a form that looks largely similar to the traditional notation used when describing the equivalent concepts. What a mathematician would write as (N-K)! is written as (N-K)! in the code. This is by design. It is more practical for composition rules to be written the same in the concept space and in code, and so most programming languages take this approach at least for arithmetic expressions. Not all languages do. Lisp, for instance, uses a prefix polish notation, and Lisp programmers would write the same expression as (! (- N K))1. When they associate a function and its definition, mathematicians and Haskell [16] programmers typically use an equation. XL, in order to carry additional concept information such as a description of the domain of equivalence, combines the various elements differently, using the function syntax we have seen a few times2.
1. However, even if that notation is much less frequently used than the traditional infix notation, it had originally been introduced in the concept space by a polish mathematician named Jan Lukasiewicz.
80
Concept Programming in XL
Abstractions
Most languages feature many ways to combine entities: expressions, compound type creation, data declarations, sequencing instructions, argument passing, and a few others. In addition, these kinds of combinations can generally be performed in any order and in any number. This large number of possibilities can be used to create the whole variety of entities in programs from a small number of basic components.
5.2.5. The Secret of Power: Layering The power of abstraction is vastly increased by a particular form of composition: layering. Layering is the process by which a combination of program entities are themselves turned into a new, higher-level abstraction for a more complicated concept. Layering can be repeated multiple times, which allows the progressive abstraction of concepts of virtually arbitrary complexity. Through layering, a complicated problem can be reduced into simpler components, and these simpler components reduced again, until each of the individual components becomes simple enough to be represented using some primitive entities of the programming tool being used. For instance, a very simple expression like Combinations(4,2) can successively be reduced as follows: • a call to the function Combinations of Listing˚5-2 with parameters 4 and 2, meaning: • the execution of the statements in the Combinations in sequence, meaning: • the computation of the expression in the return statement, meaning: • the invokation of the Factorial function 3 times with arguments 4, 2 and 2 respectively, which each times means: • the execution of the body of the Factorial function from Listing˚2-3 (we could also have chosen the function from Listing˚2-4), successively with values 4, 2 and 2 for N, which means, each time: • the initialization of the variable result with value 1, followed by successive multiplications, which lead to result being 2 when N=2 initially, and result=24 when N=4, at which point • execution resumes in the caller (Combinations in that case), which can compute: • the combined expression 24/(2*2) using integer operations1, which results in • Combinations returning the value 6, which is • the value of the expression Combinations(4,2).
2. XL extensibility features could be used to implement an equation-based notation if desired. Haskell can dispense with much of the additional notation thanks to a very interesting type system. However, when the type needs to be given, a different notation is required even in Haskell. 1. These operations are built-in entities in most languages, but for XL there is here an additional level of abstraction (a written form) that we will ignore for the sake of simplicity.
Concept Programming in XL
81
From Concepts to Code
This exercise in futility illustrates exactly just how much abstraction resides in the average program. Note that, since we stayed within the implementation limits all along, the principle of equivalence has been preserved throughout this entire chain of events. The value of the mathematical expression 4C2 is indeed the mathematical integer 6. Therefore, the expression Combinations(4,2) is a valid representation of the corresponding mathematical concept. Once again, we think: Wow, computers sure know about math
5.2.6. Abstractions against Complexity Abstractions are useful because they allow programmers to address the problems listed in Chapter 4, The Trouble with Programming . I will start by analyzing how abstractions help managing the various forms of complexity. Scale Complexity: Hiding irrelevant implementation details eliminates all the associated complexity. In other words, creating an abstraction can cancel out most of the complexity below a given level. Someone referring to a function no longer needs to think about the implementation details. This is true both in the concept space and in the code space. A mathematician can think about a probabilities problem in terms of number of permuations , without needing to remember each time that this is defined using a factorial. Similarly, a programmer can write a call to Permutations without needing to remember how it is implemented.
This would not be enough to fight complexity that grows exponentially over time, but then layering allows us to repeat the process as necessary, until the highest level abstractions become manageable again. It is not more complex for a programmer to deal with a button on a modern graphical user interface than it was for the previous generation to deal with the kind of simple keyboard-driven commands that existed then. However, there are thousands or millions of lines of code behind that simple button that did not exist for the text commands. The scale complexity increased by orders of magnitude, but thanks to new layers of abstraction, the complexity for the programmer did not. Domain Complexity: The process of abstraction creates representations in the code that we can reliably associate with the original concept, using the principle of equivalence. The complexity of the application domain unsurprisingly transposes into the program domain. However, the code representation does not need to be as complex as the original domain, thanks to appropriate modeling.
For example, if you are creating a car-racing videogame, chances are that your car model will not include a detailed description of the cigarette lighter and passenger lamp. Those complexities exist in the application domain: somebody needs to design and build these two pieces in the real car. But in the representation that has been created for the program application, their presence is unnecessary. Even if the abstract model contains a description of the car movements, it is probably very simplified compared to the kind of mechanical equations that engineers had to resolve when the real car was designed. 82
Concept Programming in XL
Abstractions
The domain complexity transposes to the program, but to a limited, controlled extent. Proper modeling and careful choice of appropriate abstractions allow programmers to focus on what is relevant to the program. Artificial Complexity: Manipulating any abstraction introduces a small dose of artificial complexity: the programmer at the very least needs to learn how the concept is represented in the program source. There is a big return, however, since it is expected that the abstract form is simpler and easier to manipulate than the original form. The investment is made once, but the benefits are reaped on every use. For instance, the C++ standard library introduced lists and algorithms operating on lists. C programmers need to learn how to use these abstractions, which may take a little while. But then, each use is simpler compared to the hand-coding of list manipulations.
However, creating a new abstraction doesn t automatically transform code that did not previously use it. In order to benefit from a new abstract concept representation, programmers may have to retrofit or refactor their code. This is a relatively painful operation. The benefits may be worth it if the resulting code is vastly simpler. But quite often, old artificial complexity is carried over from a previous iteration of the design where the necessary abstractions were not invented yet. Once you decide to get rid of the baggage, the artificial complexity can be significantly reduced, but the decision to refactor is a difficult one. Refactoring is especially painful in the area of language design. XL being designed from scratch eliminated many minor annoyances found in many previous languages. However, using it means losing a lot of the existing code. The step is more easily taken for new code, where the benefits pay off immediately, than for old code that needs to be ported to a completely new language and design model. Business Complexity: Good abstractions have several positive effects with respect to
business complexity: • At the start of a project, they make program code easier to produce, by offering programmers efficient representations of the real world context that business cares about. They also help designers understand how to put the individual components together to achieve the project objectives. Abstractions increase productivity. • As the program matures, good abstractions can ensure that the various components in the program interact the way they are expected to. This avoids problems at the time the components are integrated together. Abstractions reduce risks. • Once the program has been delivered, good abstractions facilitate its maintenance, by offering descriptive views of what should happen. If a piece of code doesn t work, the principle of equivalence generally helps understanding what it should actually do. Abstractions augment the longevity of the code. None of these observations is particularly new. These same benefits (productivity increase, lesser risk, better longevity and maintenability) have been touted as attributes of previous paradigm shifts, notably object-oriented programming and structured programming. But in
Concept Programming in XL
83
From Concepts to Code
reality the benefits are not a property of any particular method. They are only a property of the new abstractions that the method enables. Another interesting aspect of abstractions is that they lead to a separation of concerns. If a program uses the Combinations function and gets a wrong result, it is possible to attribute the problem to that specific function. Separation of concerns is important, since it allows to assign responsibilities to pieces of code and, ultimately, to individual programmers. Quality: Everything else being equal, good abstractions increase the quality of software.
When they understand what they are dealing with, programmers introduce less bugs, and they spend less time finding the remaining problems. They have a point of reference: the high-level concepts that the abstract constructs in the program represent. Good concepts, when properly implemented by the development tools, also increase quality by automatically identifying errors and problems before they become critical. For instance, a programming language that has a proper notion of array like Ada or XL can identify and warn if the array is being accessed outside of its legal bounds. If, instead, the concept of array is simulated using a lower-level of abstraction, like pointers displacement, as in C, then the compiler cannot detect common errors. Concept programming languages like XL extends this idea to library software components (not just built-in entities). Active libraries (see Section˚14.4) can document their own requirements to the compiler, for instance to detect and report their own errors.
5.3. Abstractions in Programs Programming languages offer various facilities to implement the abstractions discussed in the previous section. These facilities include: • Naming entities or giving them symbolic representation, to hide implementation details, • Defining some abstract machine model, to preserve the principle of equivalence, • Featuring composition rules, to allow the combination of program entities, and • Allowing the nesting of program constructs to create layers. All these methods have limits worth knowing about, and can break the ideal relationship between the concepts and the underlying machine representation.
5.3.1. Names and Symbols One of the key attributes of program entities is their name or, more generally, the symbol representing the corresponding concept in a mnemonic way.
84
Concept Programming in XL
Abstractions in Programs
Depending on the capabilities of the programming language that you use, several kinds of symbolic representations may exist. For instance, Listing˚2-3, Factorial Function uses an operator to represent the invocation of the factorial function, as in N! Operators are symbolic representations, but not names. In that case, the computer symbol is derived from a mathematical notation. This is not always true. Using *A or A^ to access the element pointed to by a pointer is a symbolic notation which has no direct counterpart outside of computer science. It is, in that sense, a pure computing concept, which is unique to programmers. There are all sort of intermediate cases. For instance, a computer file has little to do with the real-life equivalent, except that it contains data. Existing programming languages generally represent concepts using a text-based form. There has also been some research on using other non-textual forms to represent concepts [15]. It could make sense, for example, to display a mathematical formula using a nicely rendered version of the mathematical notation. Even if such systems do see the light of day in widely accepted programming tools, they probably won t sensibly change the kind of abstractions that can be represented nor how they are represented. This is the reason why, at least in its initial form, XL remains a text-oriented language1. LISTING 5-3: Computerese and mathematical notation Gamma := 1/sqrt(1-V^2/C^2)
1 * = --------------------v2 1 – ----c2
Graphical User Interface Editing. A few commercial development tools actually use such
non-textual representations extensively. A well known example is Visual Basic from Microsoft, where editing the graphical user interface is done directly by manipulating a graphical representation of the windows and other widgets. This is illustrated in Figure 54, which shows a form containing a command button, a couple of option buttons, a text box and its label.
FIGURE 5-4: Visual Basic non-textual editing of program code
The inspiration for this approach was found in NeXT s InterfaceBuilder software, shown in Figure 5-5. In this environment, it is possible to edit the layout and even some properties of user interface objects graphically. 1. However, since XL is really defined by an abstract syntax tree (Chapter 7), an advanced editor using nontextual representations can in theory be developed, and I hope that this will happen one day.
Concept Programming in XL
85
From Concepts to Code
FIGURE 5-5: NeXT Interface Builder
Obfuscation with Bad Names: Experienced programmers know how difficult it is to select good names for functions, variables or classes. The difficulty stems from the fact that a complicated concept, such as a complete sequence of instructions, has to be accurately described in a concise way. All other possible symbolic notations, such as operators, must also be selected with the same care as names. They need to remind the reader easily of what they represent.
Selecting good names is one area where compilers and languages can t help. Only experience and judgement allow programmers to select names that click . The computer cannot read your mind to figure out if the name you are using is misleading, improperly used or offensive. From the programming language semantics point of view, the choice of name or symbol is purely arbitrary. But for the programmers, it makes a world of difference. LISTING 5-6: Poor choice of names
1 function MultiplyByTwo(NotACat:integer) return integer written N-- is 2 SquareRoot : integer 3 result := 1 4 for SquareRoot in 2..NotACat loop 5 result *= SquareRoot
86
Concept Programming in XL
Abstractions in Programs
The code in Listing˚5-6 illustrates the problem. While the generated code is practically equivalent to that of the Factorial function in Listing˚2-4, it is obviously more difficult to understand that it relates to a factorial concept, simply because there is no match between the concepts being represented and the names or symbols in the program. This absurd example illustrates a contrario the fundamental principle of concept programming that code is a representation of application concepts. In Listing˚5-6, you don t see this function doing a multiply by two as the name suggests. The code doesn t lie to the compiler, but it s lying to the programmers reading it. Even if the NotACat name itself is not a lie, since the variable is indeed not a cat, it doesn t help either. It actually goes in the way, since you spend some time wondering OK, if it s not a cat, then what is it? This trivial example may look silly, but variants of this creep again and again in virtually every piece of code. One frequent occurence is that a particular piece of code may change function over time, but the name is never updated to reflect the new function. As a result, the name becomes an obstacle to correct understanding of the program. Another common issue is to reuse variables, confusing their meaning. This is more frequent among users of languages such as C which do not allow a variable to be declared anywhere. C++ or XL makes it easy to declare a variable right where you need it. Encoding Implementation in Names: Names are intended to stand as a placeholder for
what they represent. In that, they hide what the entity really is from an implementation point of view, and replace it with a concept that is equivalent in the programmer s mind. Some older programming languages such as BASIC had some additional implementation information embedded in the name. For instance, M$ would be a string, while M% would be an integer, and M a floating-point number. This turned out to be a bad idea, since it carried with the name some implementation-related information which the name was designed to hide. Unfortunately, a variant of this bad idea is still in widespread use today. It is called the hungarian notation, and it s a naming convention where variable names begin with implementation information (generally type-related). As a result, you find code littered with names such as lpszStr, where lpsz means Long pointer to zero-terminated string , and Str means string. Not only is the notion that it is a string present twice in the name, but one of them details the implementation. Worse yet, the name carries over obsolete information, since on today s machines, there is no such thing as a long pointer : long pointers are an outdated notion from the 16-bit and Windows˚3.1 era. There are more subtle variants of this problem in code that evolved past an implementation detail which remains encoded in the name. For instance, you find in the Windows API functions like CreateWindow and CreateWindowEx, where the Ex suffix stands for extended . The name simply carries over an implementation consideration, namely that the function is the version 2 of an interface for which preserving backward compatibility was required1. 1. This particular case would not have been necessary if C supported function overloading
Concept Programming in XL
87
From Concepts to Code
In conclusion, programming languages offer many ways to name entities or represent them using symbols. This allows the actual details of the entity to remaing hidden. But since the compiler doesn t know what you are talking about, you need to make sure that the name and symbols are meaningful, that they do not confuse the programmers, and that they don t carry unnecessary or incorrect implementation details.
5.3.2. Damage Control Regardless of the precise semantics, it is important to keep in mind that the program concepts are only an approximation to the real concept. The first and foremost rule of concept programming remains to remember that code is only a translation and not the real thing. Remember: there is no spoon. Trying to make a distinction between the original concept and its computer representation may sound like a pedantic exercise in futility, and a big deal of complication for something completely trivial. But just as for name and symbolic representations, the fact that it is actually nontrivial is best illustrated when it doesn t work. This is shown by the code in Listing˚3-12, Displaying factorial values in C and the results in Listing˚3-13, where we saw the computer display null or negative factorial values. Documenting Implementation Limits. Is the program in Listing˚3-12 wrong? If we stick
to the definition of a mathematical factorial, then the program is indeed wrong. In a crippling show of bad faith, we could finesse that statement by saying that the program computes the correct factorial modulo a specific power of two (231 in that example). A more honest assertion is that the implemented factorial is a good approximation of the original concept for values that are small enough. As someone said before me, there are no bugs, there are just perfect programs with invalid input. To increase the robustness of our code, we can document this limitation as part of the interface of the function, as shown in Listing˚5-7 (the last value computed without overflow being the factorial of 12). The require keyword introduces a precondition, in other words a condition that is require for correct operation of the Factorial function. Preconditions are fundamental elements of a design methodology known as design by contract, initially popularized by the Eiffel language (see Section 12.1, Design By Contract on page 152). LISTING 5-7: Factorial with documented limitations on input
1 function Factorial(N : integer) return integer written N! 2 require N >= 0 and N output.html
Like other program representations, parse trees are very dependent on the particular language and compiler technology. The details of the XL parse tree representation, called XL0, will be specified in Section˚7.3. In Figure 6-3, green is used for XL0 prefix nodes, blue for XL0 infix nodes, and pink for XL0 block nodes. The good news is that you will practically never need to know the details, as XL features high-level constructs to deal with parse trees.
Concept Programming in XL
97
Compiling XL
FIGURE 6-3: Color-coded representation of an XL parse tree
6.2. Understanding Programs The reason compilers generate an internal representation of the program is to be able to make sense of it, and transform the source code into a final executable program. The first level of understanding is syntax analysis, which checks if the program structure looks right. This phase essentially transforms text into tokens and tokens into a parse tree. The second level is semantic analysis, which verifies that the program actually means something. This second phase annotates the parse tree with additional information such as symbol tables (to find what definition might be associated with a particular name at a particular point in the program). After these two phases, it is possible to perform various operations on the parse tree, but the most common is to expand it and feed it to a code generator in order to generate some kind of machine code.
98
Concept Programming in XL
Understanding Programs
6.2.1. Syntax: What a Program Looks Like Syntax analysis determines if a program structure follows the language rules. Tokens. The first step is to make sure that the program is composed of valid tokens. The
scanner (also called the lexical analyzer) is the compiler component that reads source program text, character by character, and builds a sequence of tokens. It is during this stage that the compiler will for instance detect if a number is malformed, or if a text literal is too long. In many compilers, the scanner is written using a domain-specific language designed to write scanners, such as lex [17] or its free software equivalent, flex [18]. Listing˚6-4 shows a simple scanner definition written using lex, that defines the tokens required for a simple calculator. 1 2 3 4 5 6 7 8 9 10
LISTING 6-4: Num %% [Num]+ ”+” ”-” ”*” ”/” clr sto rcl
Example of scanner definition using lex [0-9] { { { { { { { {
yylval return return return return return return return
= atoi(yytext); return TOKEN_NUMBER; } TOKEN_ADD; } TOKEN_SUB; } TOKEN_MUL; } TOKEN_DIV; } TOKEN_CLEAR; } TOKEN_STORE; } TOKEN_RECALL; }
It is not necessary to use lex to write a scanner. As a matter of fact, the XL compiler cannot use lex because XL is configured at runtime by an external syntax definition file (see Section 7.3.6, Syntax Configuration file on page 118), whereas lex only supports statically-defined scanning rules. Parse Tree. The second step is to transform the sequence of token into a parse tree using
grammatical rules. The parser (also called the syntactical analyzer) is a compiler component that takes the sequence of tokens as input and generates a parse tree as output. In many compilers, the parser is defined using another domain-specific language, such as yacc [19] or its free software equivalent, bison [20]. Listing˚6-5 shows a simple calculator definition written using bison. For such a simple scenario, the parse tree can be implicit, being actually recorder in the execution state of the parser.
Concept Programming in XL
99
Compiling XL
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
LISTING 6-5: Simple calculator using bison %{ #define YYSTYPE double #include double calc_mem; %} %token TOKEN_NUMBER TOKEN_ADD TOKEN_SUB TOKEN_MUL TOKEN_DIV %token TOKEN_CLEAR TOKEN_STORE TOKEN_RECALL %% program: /* Nothing */ | program statement ; statement: ’\n’ | expr ’\n’ { printf (”%f\n”, $1); } | TOKEN_CLEAR ’\n’ { calc_mem = 0.0; } ; expr: TOKEN_NUMBER { $$ = $1; } | expr TOKEN_ADD expr { $$ = $1 + $3; } | expr TOKEN_SUB expr { $$ = $1 - $3; } | expr TOKEN_MUL expr { $$ = $1 * $3; } | expr TOKEN_DIV expr { $$ = $1 / $3; } | TOKEN_STORE expr { calc_mem = $1; } | TOKEN_RECALL { $$ = calc_mem; } ;
Like lex, bison is useful but not necessary to write parsers. Another frequently used technique is to write a recursive descent parser [21]. A recursive descent parser is a top-down parser built from mutually recursive subroutines. Such parsers can be slow for some grammars, but they can also be quite fast if the input language is designed in such a way that the parser never needs to backtrack, in other words, it never need to change its mind due to some ambiguity like the ones we discussed in Section˚4.3.2 and Section˚4.3.3. XL cannot use bison, for the same reason it cannot use lex: the grammar is defined by an external syntax definition file, but a tool like bison is designed for languages with a statically defined order of precedence. The XL parser generates an explicit parse tree, using a recursive descent parser parameterized at runtime.
6.2.2. Semantics: What a Program Means Semantic analysis determines the meaning of a program. In particular, this phase associates programmer-defined identifiers to their definitions, checks type constraints according to the language s type system, and verifies that constructs are used in the proper context. Symbol Table. Programming languages allow programmers to name entities (see Section˚5.3.1) and give it some additional properties such as a type.
A declaration is a statement that tells the compiler what these properties are for a given entity. A definition is a statement that tells the compiler what value or implementation to associate with a declaration. Listing˚6-6 shows a declaration and a definition of a function called add in C.
100
Concept Programming in XL
Understanding Programs LISTING 6-6: Declaration and Definition in C
1 extern int add(int a, int b); 2 int add(int a, int b) { return a+b; }
The compiler associates a name such as add to a declaration or a definition using a symbol table. A symbol table may record the specific scope in which a given symbol is defined. For example, in Listing˚6-6, the add symbol is declared and later defined in global scope, whereas a and b are declared in a local scope (in that case, the parameter list of the function). Scoping rules indicate the extent of the source code where a given symbol is visible. For example, in Listing˚6-6, the scoping rules of C ensure that a and b are not visible outside of the add function implementation. The rules regarding the number of possible declarations and definitions with the same name vary from language to language. For instance, in C, there can be only one function with a given name, but in C++ or XL, functions can be overloaded. Even in C, a structure, a function and a variable can share a name: the Unix function stat takes a struct stat as its argument, and a function could still call a local variable stat. This simple form of overloading can be implemented by having different namespaces in the symbol table. The more general case found in C++ of XL allows an arbitrary number of declarations in the same scope to share the same name, and the symbol table needs to accomodate that requirement. The precise rules have an impact on how the symbol table is represented. But typically, a the symbol table for a given scope is represented using some sort of hash table, map or dictionary. In order to address scoping rules, symbol tables for different scopes may be linked in a hierarchical way. For instance, the symbol table for a function will contain a pointer to the symbol table for the global scope, so that a symbol not found in the function scope can be looked up in global scope. Type System. The type system is also very different from language to language.
Many programming languages, like C, Pascal or Ada, declare the type of an entity along with its declaration, and enforce a number of type checks when the declaration is used. This is called static typing or sometimes strong typing (the latter being so overloaded that its meaning is very difficult to define precisely). Even within static type systems, there are a number of variations in how strict the rules are. For example, in C, you can freely assign between numeric types even if they have different size or signedness. By contrast, Ada has much stricter conversion rules. Ada also adds constraint checks at runtime, so that some type violations that cannot be easily enforced at compile time are enforced at runtime. For example, an integer overflow remains undetected in C, but raises an exception in Ada. Instead of static typing, languages such as Lisp use a more dynamic type system. In Lisp, type information can be given to the compiler, but it is seen mostly as an optimization hint. It is easier to declare variables that can hold a value of any type, including instructions forming a Lisp program. This is very powerful, but requires a rather complex runtime to operate compared to languages like Ada or C. It also requires more effort on the part of the Concept Programming in XL
101
Compiling XL
compiler to achieve comparable runtime efficiency, so that in numerous cases, languages with dynamic typing will be less efficient at runtime. There are all sorts of special scenarios. The Haskell type system, for instance, can infer the type of variables based on how they are used. Type declarations in Haskell tend to be much less frequent than in languages like C. Languages like Prolog attempt to infer not just the type, but also the value of the variable. In such languages, references to a variable form its definition, and it is legal to reference variables whose definition is not fully known. A type system is generally implemented by having an internal representation of types stored in a type table. Named types can be referenced from the symbol table, and the internal representation for a declaration or definition will refer to an entry in the type table.
6.3. Compiling XL The XL compiler has some rather traditional aspects, and a few less traditional ones covering the extensible nature of XL. Figure 6-7 shows how the various component interact in the free-software XL compiler [1].
Scanner Text
Semantics Plug-Ins Expansion Optimizations
Parser
Tokens
Parse Tree (XL0)
Renderer Bytecode Text
FIGURE 6-7: XL compiler components
The scanner takes text as input and generates a sequence of tokens. These tokens are then analyzed by the parser to generate a parse tree. The XL parse tree is a public data structure, that is available to XL programmers throught the XL standard library. A number of components access the same parse tree representation, generating a modified parse tree as output. These components include the compiler semantics phase, compiler plug-ins that implement language extensions, an expansion phase, and optimization components. Finally, the resulting parse tree is turned into the output of the compiler. There are two main output components. A renderer transforms parse trees into a text format that represents the program. In a sense, the renderer is a reverse parser. The renderer is used when the parse
102
Concept Programming in XL
Compiling XL
tree needs to be presented in human-readable form, for instance when emitting error messages. The second output component is the XL bytecode module. It takes specially formatted parse trees containing, and generate text output based on a bytecode configuration file. The bytecode module is used when generating machine code (which can be in a variety of languages). It is essentially similar to a stripped-down rendered: the same effect could be achieved using the renderer, but much less efficiently.
Concept Programming in XL
103
Compiling XL
104
Concept Programming in XL
Compiling XL
Chapter 7 — Syntax
Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. Antoine de Saint Exupery
The syntax of XL is configurable and extensible. It was designed to accomodate a very large number of needs. Obviously, the syntax can be used to represent XL program text, where XL is considered as a traditional programming language. But since XL can be extended, the syntax must also allow other uses, to represent concepts that are not part of the core language. This principle extends further: in XL, no language construct deserves special treatment. Whatever mechanism is used by programmers to extend the language must also be sufficient to implement basic constructs like functions or control statements. As a result, the XL syntax is very flexible and very versatile. It can for instance be used to represent data structure, much as XML or similar data-oriented languages. But XL did not achieve this objective by adding complicated syntactic constructs. Instead, was achieved it by reducing the core syntax of the language to its simplest form. Whereas other programming languages may have dozen of keywords, XL has none; whereas simply parsing programs in other languages may require deep knowledge of language semantics, the structure of an XL program depends solely on its shape and not at all on its meaning.
Concept Programming in XL
105
Syntax
7.1. Source Text Since an objective of XL is to represent arbitrary concepts, the language should not restrict itself to a particular human language such as English. However, in order to achieve portability, the design should also take into account that the only character set guaranteed to be present on most platforms is, for all intent and purpose, the american character set, as available in ASCII or EBCDIC encodings. To address this problem, the meaning of an XL program is defined solely based on the program s appearance to the programmer, irrespective of the encoding used to represent the program in the machine. In other words, the meaning of an identifier like Hello does not depend on the underlying encoding of the program source, which might be ASCII, Unicode, EBCDIC or any other standard. The same is also true for an identifier like Résultat, with an important caveat: that identifier may not be representable on a number of platforms which do not have a representation for accented letters like é. For instance, a program containing this identifier cannot be entered in a machine that uses pure 7-bit ASCII. Character supported on the platform where the program is compiled are classified as one of: alphabetic, numeric, line terminators, whitespace, underscore, punctuation, and control characters. All characters but control characters can be used in XL program. The meaning of the program is based on this classification, but the precise implementation of the classification falls outside of the scope of the language definition. Character encoding standards, on the other hand, may specify how this classification is implemented to a large extent. Alphabetic Characters. Alphabetic characters include at least the 26 letters of the English
alphabet. If the character set contains letters in multiple case or accented letters, these characters are treated as alphabetic. If the character set includes other characters used to express a human language, such as ideograms or mathematical value symbols such as !, ∞, or Ω, these characters are also considered as alphabetic. Some human languages might not traditionally make use of whitespace. Consecutive ideograms of that language not separated with whitespace will be treated as a single identifier, much like consecutive english words in identifiers such as HelloWorld. Conversely, whitespace will split the ideograms into multiple identifiers, much like it does in Hello World. Numeric Characters. Numeric characters include at least the 10 digits used in most of the western world. If the character set contains digits in multiple case, or characters that would be traditionally given a numerical value between 0 and 9 inclusive but are not alphabetic, these characters are also treated as numeric characters.
In some languages such as Hebrew, the same characters can be considered as alphabetic or numeric depending on context. In the case of XL, such characters will be considered as alphabetic.
106
Concept Programming in XL
Tokens
XL has support for based numbers, and uses letters for bases above 10. Only the nonaccented letters A through Z can be used for that purpose, either in upper or lower case, and they represent decimal values 10 through 35 respectively. Other alphabetic characters are illegal in based numbers. Line Terminators. The XL text is scanned as if there was a special character at the end of
each line of input, called a line terminator. This is true even if the internal representation of lines does not use line terminators, for instance if each line is a record in a database. Whitespace. There must be at least one whitespace character in the character set. If there
are tabulation characters, they are treated as whitespace and not as control characters. If there are multiple whitespace characters (including tabulation characters), the indentation must be indicated with a single one of them in a given source file. Underscore Character. There can be one or several underscore characters. They are not indispensable for writing XL programs, being used merely to increase readability in identifiers or numbers. Except when they appear in text literals, underscore characters can be removed from XL programs without changing their meaning. Punctuation. The following punctuation characters are necessary to write XL programs: +-*/=.,?()[]{}:;!#%^&\'"|. Additional punctuation characters may be available. Symbols that are relational in nature, such as ≤ or ≈ should be marked as punctuation if
possible, so that they can be used to define additional operators.
7.2. Tokens XL distinguishes the following kinds of tokens: integer, real, text, name, symbol, newline, block delimiters and indentation delimiters. Keyword tokens are notably missing compared to other programming language: keywords are represented by normal name tokens.
7.2.1. Numbers Integer and real numbers are represented respectively by integer literals, such as 100, and real literals, such as 3.1415926. The scanner identifies the kind of literal, and returns either an integer token or a real token. The decimal separator in XL is the dot . as is customary to most english-speaking people. A number always begins with a numeric character. Notably, -3 is not a number but a unary operator applied to a number. The difference is academic in most languages, but it matters in XL, because the two representations have different parse trees.
Concept Programming in XL
107
Syntax
The largest possible consecutive sequence of numeric characters is scanned and interpreted as an integer in base 10. Two consecutive digits can be separated by a single underscore character. For example, 256_000 is the number 256 000. If the next character is a #, then the base-10 number just read represents a base, and must be between 2 and 36 inclusive. The following characters are interpreted as digits of the base, with letters betwee A and Z representing digit values above 9. Two such digits can be separated by an underscore as well. For instance, 2#1001 represents 9, 16#FF represents 255 and 3#2222_1111_0000 represents 528120. If the next character is a dot, it is interpreted as a decimal separator, and the scanner will return a real token. For example, 1.3 represents 1.3, 2#1_000.111_000 represents 8.875. If the next character is a # it is ignored. For example, both 16#33 and 16#33# represent the number 51. One reason for this is that Ada programmers are used to the second # sign. The other reason is that a separator is needed before the exponent for all bases above 15, where E is a digit of the base. If the next character is the character E or e, followed by an optional + or - sign, followed by a sequence of decimal digit, then this represents an exponent. If the exponent is negative, then the scanner will return a real number, but positive exponents are allowed for integer numbers. For example, 2E3 is the integer 2000, whereas 1E-3 is the real 0.001. The exponent, like the base, is expressed in base 10, but it represent a power of the base. In other words, 2#11#e3 is 3x23=24. Each one of the elements described above can be used to improve readability in specific cases, or convey the intent of the programmer better. Listing˚7-1 illustrates this. But if misused, these notations can also make the code more difficult to read. However, it normally takes more effort (more characters) to make the code difficult to read. 1 2 3 4
LISTING 7-1: Using the integer notations bitmask := 2#1111_0000_1111_0000 pi := 3.14159_26535_89793 billion := 1E9 billionth := 1E-9
7.2.2. Text Text is represented using text literals, such as ”Hello World”. A particular form using single quotes is used in general to represent single characters, such as ’A’, but the distinction is done during semantics (when assigning a type to these entities), not during scanning. In particular, multi-character character literals are valid from the parser s point of view. Text literals can begin with a single or double quote character, and terminate when the same character that began the character literal is found again, unless that character is doubled. A text literal starting with a single or double quote character cannot span multiple lines.
108
Concept Programming in XL
Tokens
Doubling the quoting character that began the text allows that character to be placed in the text. For example, ”Hello” ”World” is a sequence of two text literals, one containing Hello and one containing World. On the other hand, ”He said: ””Hello World””” is a single text literal containing He said: ”Hello World”. The syntax definition file1 can also define long text delimiters, which can be used to create text terminals that span multiple lines. The text delimiters must be names or symbols (see Section˚7.2.3). This is useful for parts of a program where long text literals are required. Listing˚7-2 illustrates this usage, assuming that > delimiters are marked as long text delimiters in the syntax file. Line delimiters and indentation are preserved in the text returned by the scanner. LISTING 7-2: Adding long text delimiters
1 IO.WriteLn >
The characters allowed in text literals are in most cases the same as the characters allowed in the program itself (see Section˚7.1). There are rare cases where this is not true, when the program is compiled on one machine and intended for another, a processus called cross compilation. In this case, it is possible that the machine on which the compilation happens (the host) has a wider character set than the machine for which the program is to be generated (the target). A cross compiler needs to perform appropriate character set translations, and if necessary, emit error messages if characters are used that do not exist on the target. Many other languages feature so-called escape codes that can be placed in text literals. For instance, in C, \n is a newline character, and \t is a tab character. XL does not have such a mechanism integrated in the scanner, because escapes codes are really dependant on the text representation, more so, in general, than printable characters, and also depend on how the text is used. For intance, HTML represents newline using
instead of \n. In XL, escape codes, when necessary, are interpreted by appropriate plug-ins or runtime scanning of the text literals. They are not defined by the scanner itself.
7.2.3. Names and Symbols Names and symbols are used to denote entities in the XL program. Names begin with an alphabetic character, followed by a possibly empty sequence of alphabetic or numeric characters. Two such characters can be separated by a single underscore for readability purpose. For instance, Mozart or Aleph_0 are names. Leading, trailing or double underscores are not allowed, so __ is not a valid name in XL.
1. The format of this file is explained in Section 7.3.6, Syntax Configuration file on page 118.
Concept Programming in XL
109
Syntax
Symbol are sequences of punctuation characters, at the exclusion of characters identified as block delimiters in the syntax configuration file (Section˚7.3.6). For example, * and are valid XL symbols. It is worth noting that an expression like A&~B corresponds to two operators in C (it reads as A and not B ), but to only one &~ operator in XL. To get two separate operators, one would need to use additional whitespace, for example A & ~B. Capitalization. The scanner and parser preserve case and separating underscores in names,
but the XL semantics ignores them in general. For example, the same entity can be referenced as John_Doe, JohnDoe, or JOHN_DOE. Ignoring style is similar to the way we name things in real life. When you talk about Mr. John Doe, you do not ask whether it s John Doe with a big D, or John Doe with a small D . Many other languages are case-sensitive, mostly because it simplifies the compiler. However, this also makes talking about the entities in the programs more difficult, since spoken language does not convey case information. So programmers need to refer to variables using phrases like big V or small V . In XL, this problem does not exist. Another issue eliminated by style-insensitivity relates to the style choices of library code. Most programs today use several libraries from various source. It is not infrequent that these libraries use different styles. For example, in C on Unix, the operating system is accessed using functions that are spelled in lowercase with no word separations, for example fopen. On the other hand, the X11 library uses functions that mix case, for example XOpenDisplay. With C, you have no choice but to import these inconsistencies in your own code. With XL, on the other hand, you could make your own code use a single style irrespective of the choices made by library designers. However, the original presentation of names is preserved in the parse tree, which is useful for example in emitting error messages that are consistent with the style and capitalization actually used in the source code. Preserving capitalization is also useful for the rare situations where the symbol refers to an entity that is not governed by XL semantics, notably when interfacing with other languages. This is shown in Listing˚7-3, which imports the C function glClear from OpenGL into XL. In that example, the XL name Clear can also be written as clear or using any other style. However, the capitalization of glClear matters, because it refers to a C function. LISTING 7-3: Infrequent condition where capitalization is significant in XL
1 procedure Clear(mask : bitfield) is C.glClear
Text and Comment Separators. A name or symbol may be marked in the syntax config-
uration file as a text or comment separator. In that case, the scanner does not return it. For comments, it skips it and looks ahead for the text sequence closing the comment. For long text, it scans the text ahead until the matching closing text delimiter is found, and returns the text in between.
110
Concept Programming in XL
Tokens
7.2.4. Blocks Punctuation characters can be marked as block delimiters in the syntax configuration file. Such block delimiters always work in pair, one character opening the block and one closing it. By default, the block delimiters are parentheses, brackets and curly braces: (){}[]. The reason the block delimiters are treated specially is to allow nested blocks to be written easily. For instance, the expression ((A+1)*2)begins with two consecutive block opening characters. For other punctuation, this would form a single symbol. Block characters, on the other hand, are returned individually by the scanner.
7.2.5. Indentation, Whitespace and Line Terminators If the syntax configuration file specifies indentation to play the role of a block delimiter, then indentation is also returned by the scanner as block opening and block closing tokens. When multiple levels of indentation are closed in a single line, then the correponding number of unindentation tokens will be returned. Indentation can be marked using any whitespace character, but only one of them in any given source file. In other words, you may indent with tabs or space depending on your preference, but you may not have one part of the same source file indented with tabs and the other with spaces. This restriction only applies to indentation: all whitespace characters can be used interchangeably outside of indentation. Whitesepace is meaningful only in text literals and indentation, otherwise whitespace is simply ignored, although it may sometimes be necessary to separate names or symbols. Line terminators are virtual characters that mark the end of a line. Some operating systems, notably Unix derivatives, use an actual control character as a line terminator in the encoding of the program text. Others, like Windows or DOS, use a sequence of characters (carriagereturn followed by line-feed). There are other ways of storing text that do not rely on anactual character terminating lines. For instance, the text could be stored in a database, or there could be an encoding of the length of each line. These implementation details are hidden from the scanner: as far as the scanner is concerned, the program text contains line terminating characters. Sequences of one or more line-terminating characters are treated like a special symbol. Like any symbol, it can be used for example to specify the termination of a comment in the syntax configuration file. The default syntax configuration uses this line-termination symbol to terminate comments that begin with the symbol //. The line-terminating symbol is also normally identified as an infix operator, for reasons that will become apparent in Section 7.3.4, Prefix and Infix Nodes .
Concept Programming in XL
111
Syntax
7.3. Parse Tree Empirical evidence suggests that only a minuscule fraction of the XL programs worth mentioning are composed of a single token. This is the reason why we need a parser to combine a sequence of tokens into a parse tree. The XL parse tree consists of seven types of nodes, four leaf nodes and three internal ones: • The leaf nodes types are integer, real, text and symbol/name nodes, like 0, 1.5, ”ABC”, Name or *=. • Infix trees represent infix operators like A+B or A xor B. They have a left and a right child, and a name or symbol to denote the infix operator. Infix nodes are used in particular to represent line separators. • Prefix trees represent two trees placed immediately one after the other, like not A, A! or Write ”Hello”. They have a left and a right child. • Block trees represent a tree surrounded by block delimiters, like (A). They have a single child. The parse tree is generated according to a set of precedence rules, which are defined in the syntax configuration file. To most programmers, the precise rules won t matter much, because XL has specific programming constructs dealing with parse trees (see Section 7.4.3, Syntax Expressions on page 124). Consequently, even code manipulating parse trees can usually rely on the XL parser to generate them. We saw a very short example of how this works in Listing˚2-14, A plug-in implementing basic simplifications .
7.3.1. Displaying the Parse Tree The XL compiler has options to display the parse tree in debug or HTML formats. Listing˚7-4 contains a code sample, and Listing˚7-5 shows the command used to dump the parse tree in debug format for that code. This is useful to understand how a given text is parsed. The -parse option indicates that the XL compiler is only to parse the input, without applying any semantics to it. The -style option selects a style sheet . In that case, the style defined in the debug.stylesheet file is used. Finally, The -show option sends the parse tree to the standard output of the compiler. LISTING 7-4: Sample code being fed to the XL parser
1 function Max(A1 : ordered; ...) return ordered is 2 result := Max(...) 3 if result < A1 then 4 result := A1
LISTING 7-5: Generating a parse tree dump from the sample code $ xl -parse sample.xl -style debug -show (prefix function (infix is (infix return
112
Concept Programming in XL
Parse Tree (prefix Max (block( ) (infix ; (infix : A1 ordered) ...))) ordered) (block indent (infix (infix := result (prefix Max (block( ) ...))) (infix then (prefix if (infix < result A1)) (block indent (infix := result A1)))))))
Listing˚7-6 shows the command used to dump the sample code in HTML format, which can be rendered graphically using any web browser. THe only difference is that another style sheet is used, namely dbghtml.stylesheet. Figure 7-7 shows one such rendering. In this representation, the node types are color coded. Green areas correspond to prefix nodes, blue areas correspond to infix nodes, and pink areas correspond to block nodes. Special notations Indent and [CR] are used for indentation blocks and line-termination infix nodes respectively. LISTING 7-6: Dumping the sample code in graphical form $ xl -parse sample.xl -style dbghtml -show > sample.html
There are other style sheets available.For example, the html style generates a color-coded representation of the source code in HTML format, and is described in the file named html.stylesheet. Style sheets are an example of files using the XL syntax to represent data. They describes the sequence of characters used to represent each kind of node in a parse tree.
7.3.2. Leaf Nodes Leaf nodes are parse trees containing a single element, generated from a single token. There are four leaf node types: • Integer nodes, built from a single integer token • Real nodes, built from a single real token
Concept Programming in XL
113
Syntax
FIGURE 7-7: Color-coded representation of an XL parse tree
• Text nodes, built from a single text token • Name/Symbol nodes, built from a single name or symbol token. The parser always turns integer, real and text tokens into a corresponding leaf node type. For example, 1.26 is unambiguously represented by a real node, one of the leaf node types. For name tokens, however, the process is more complicated, because a name or symbol token may be a leaf node, but it may also indicate the name or symbol for an infix node. This is explained in Section˚7.3.4. Intuitively, in an expression like A and B, the tokens A and B are turned into leaf nodes, but the token and is the name of an infix node, and there is no leaf node associated with it.
7.3.3. Blocks, Parentheses and Indentation Block nodes are used for parameter lists as in F(3,5), array or structure indexes as in Dict[”Hello”], pragmas as in {serialize} and indentation. Blocks have a precedence, which is used to relate them to the surrounding operators. For example, A.B(3) parses as (A.B)(3), whereas A+B(3) parses as A+(B(3)). Table 7-8, Default operator precedence in XL, on page 116, lists the default precedence for operators. Pragmas have a precedence just above that of ; so that they apply to entire statements by default. Parenthese () and index [] block constructs have the same precedence as func-
114
Concept Programming in XL
Parse Tree
tions. Consequently, F(A) can be used as a function in an expression like F(A)(B). Block indentation has a precedence below that of a statement. Parentheses are not needed in expressions like sin X. However, if you are used to parentheses around function calls, it is possible to abide by your preferences, since sin(X) will be parsed as an infix node with sin on its left and (X) on its right. Parentheses are actually required for functions with more than one argument, like Combinations(4,2). This parses as a prefix with Combinations on the left and (4,2) on the right, where (4,2) is a block node containing 4,2, which is itself an infix node with , as the operator.
7.3.4. Prefix and Infix Nodes Names and symbols can be interpreted differently depending on context: • Leaf nodes, for instance pi in pi := 3.14 • Prefix nodes, for instance sin in sin X, or the symbol - in -3 • Infix nodes, for instance the name and and the symbol < in X --> 15 heading1 ”The XL0 file format” 16 text 17 ”This format is described in ” link ”http://xlr.sf.net”
7.4.2. Limitations of Priority-based Syntax XL0 is a priority-based syntax. We have seen, with the distinction between statements and expressions discussed in Section 7.3.7, Precedence Tricks , that human readers do not always parse using a simple priority-based scheme. This demonstrates that the natural way people group words or tokens in their head is not just priority based. It uses semantics, context, and a number of disambiguation tricks. There is no computer syntax that will allow a computer to make the right choice in a sentence like Time flies like an arrow , since there are at least two grammatically correct choices. The XL syntax is restricted to a single way of grouping words and symbols (except for the statement / expression tricks discussed in Section˚7.3.7). This sometimes means that it is not possible to accomodate the correct form for all concepts one might want to express. For example, consider the case of the boolean operators, and, or and xor. These have been given a low priority compared to relational operators, so that an expression like A
