Search the Catalog
Programming Perl, 3rd Edition

Programming Perl, 3rd Edition

Larry Wall, Tom Christiansen & Jon Orwant
3rd Edition July 2000
0-596-00027-8, Order Number: 0278
1104 pages, $49.95

Chapter 18
Compiling

Contents:
The Life Cycle of a Perl Program
Compiling Your Code
Executing Your Code
Compiler Backends
Code Generators
Code Development Tools
Avant-Garde Compiler, Retro Interpreter

If you came here looking for a Perl compiler, you may be surprised to discover that you already have one--your perl program (typically /usr/bin/perl) already contains a Perl compiler. That might not be what you were thinking, and if it wasn't, you may be pleased to know that we do also provide code generators (which some well-meaning folks call "compilers"), and we'll discuss those toward the end of this chapter. But first we want to talk about what we think of as The Compiler. Inevitably there's going to be a certain amount of low-level detail in this chapter that some people will be interested in, and some people will not. If you find that you're not, think of it as an opportunity to practice your speed-reading skills.

Imagine that you're a conductor who's ordered the score for a large orchestral work. When the box of music arrives, you find several dozen booklets, one for each member of the orchestra with just their part in it. But curiously, your master copy with all the parts is missing. Even more curiously, the parts you do have are written out using plain English instead of musical notation. Before you can put together a program for performance, or even give the music to your orchestra to play, you'll first have to translate the prose descriptions into the normal system of notes and bars. Then you'll need to compile the individual parts into one giant score so that you can get an idea of the overall program.

Similarly, when you hand the source code of your Perl script over to perl to execute, it is no more useful to the computer than the English description of the symphony was to the musicians. Before your program can run, Perl needs to compile[1] these English-looking directions into a special symbolic representation. Your program still isn't running, though, because the compiler only compiles. Like the conductor's score, even after your program has been converted to an instruction format suitable for interpretation, it still needs an active agent to interpret those instructions.

[1] Or translate, or transform, or transfigure, or transmute, or transmogrify.

The Life Cycle of a Perl Program

You can break up the life cycle of a Perl program into four distinct phases, each with separate stages of its own. The first and the last are the most interesting ones, and the middle two are optional. The stages are depicted in Figure 18.1.

Figure 18.1: The life cycle of a Perl program

Figure 18.1
  1. The Compilation Phase

    During phase 1, the compile phase, the Perl compiler converts your program into a data structure called a parse tree. Along with the standard parsing techniques, Perl employs a much more powerful one: it uses BEGIN blocks to guide further compilation. BEGIN blocks are handed off to the interpreter to be run as as soon as they are parsed, which effectively runs them in FIFO order (first in, first out). This includes any use and no declarations; these are really just BEGIN blocks in disguise. Any CHECK, INIT, and END blocks are scheduled by the compiler for delayed execution.

    Lexical declarations are noted, but assignments to them are not executed. All eval BLOCKs, s///e constructs, and noninterpolated regular expressions are compiled here, and constant expressions are pre-evaluated. The compiler is now done, unless it gets called back into service later. At the end of this phase, the interpreter is again called up to execute any scheduled CHECK blocks in LIFO order (last in, first out). The presence or absence of a CHECK block determines whether we next go to phase 2 or skip over to phase 4.

  2. The Code Generation Phase (optional)

    CHECK blocks are installed by code generators, so this optional phase occurs when you explicitly use one of the code generators (described later in "Code Generators"). These convert the compiled (but not yet run) program into either C source code or serialized Perl bytecodes--a sequence of values expressing internal Perl instructions. If you choose to generate C source code, it can eventually produce a file called an executable image in native machine language.[2]

    [2] Your original script is an executable file too, but it's not machine language, so we don't call it an image. An image file is called that because it's a verbatim copy of the machine codes your CPU knows how to execute directly.

    At this point, your program goes into suspended animation. If you made an executable image, you can go directly to phase 4; otherwise, you need to reconstitute the freeze-dried bytecodes in phase 3.

  3. The Parse Tree Reconstruction Phase (optional)

    To reanimate the program, its parse tree must be reconstructed. This phase exists only if code generation occurred and you chose to generate bytecode. Perl must first reconstitute its parse trees from that bytecode sequence before the program can run. Perl does not run directly from the bytecodes; that would be slow.

  4. The Execution Phase

    Finally, what you've all been waiting for: running your program. Hence, this is also called the run phase. The interpreter takes the parse tree (which it got either directly from the compiler or indirectly from code generation and subsequent parse tree reconstruction) and executes it. (Or, if you generated an executable image file, it can be run as a standalone program since it contains an embedded Perl interpreter.)

    At the start of this phase, before your main program gets to run, all scheduled INIT blocks are executed in FIFO order. Then your main program is run. The interpreter can call back into the compiler as needed upon encountering an eval STRING, a do FILE or require statement, an s///ee construct, or a pattern match with an interpolated variable that is found to contain a legal code assertion.

    When your main program finishes, any delayed END blocks are finally executed, this time in LIFO order. The very first one seen will execute last, and then you're done. (END blocks are skipped only if you exec or your process is blown away by an uncaught catastrophic error. Ordinary exceptions are not considered catastrophic.

Now we'll discuss these phases in greater detail, and in a different order.

Compiling Your Code

Perl is always in one of two modes of operation: either it is compiling your program, or it is executing it--never both at the same time. Throughout this book, we refer to certain events as happening at compile time, or we say that "the Perl compiler does this and that". At other points, we mention that something else occurs at run time, or that "the Perl interpreter does this and that". Although you can get by with thinking of both the compiler and interpreter as simply "Perl", understanding which of these two roles Perl is playing at any given point is essential to understanding why many things happen as they do. The perl executable implements both roles: first the compiler, then the interpreter. (Other roles are possible, too; perl is also an optimizer and a code generator. Occasionally, it's even a trickster--but all in good fun.)

It's also important to understand the distinction between compile phase and compile time, and between run phase and run time. A typical Perl program gets one compile phase, and then one run phase. A "phase" is a large-scale concept. But compile time and run time are small-scale concepts. A given compile phase does mostly compile-time stuff, but it also does some run-time stuff via BEGIN blocks. A given run phase does mostly run-time stuff, but it can do compile-time stuff through operators like eval STRING.

In the typical course of events, the Perl compiler reads through your entire program source before execution starts. This is when Perl parses the declarations, statements, and expressions to make sure they're syntactically legal. [3] If it finds a syntax error, the compiler attempts to recover from the error so it can report any other errors later in the source. Sometimes this works, and sometimes it doesn't; syntax errors have a noisy tendency to trigger a cascade of false alarms. Perl bails out in frustration after about 10 errors.

[3] No, there's no formal syntax diagram like a BNF, but you're welcome to peruse the perly.y file in the Perl source tree, which contains the yacc(1) grammar Perl uses. We recommend that you stay out of the lexer, which has been known to induce eating disorders in lab rats.

In addition to the interpreter that processes the BEGIN blocks, the compiler processes your program with the connivance of three notional agents. The lexer scans for each minimal unit of meaning in your program. These are sometimes called "lexemes", but you'll more often hear them referred to as tokens in texts about programming languages. The lexer is sometimes called a tokener or a scanner, and what it does is sometimes called lexing or tokenizing. The parser then tries to make sense out of groups of these tokens by assembling them into larger constructs, such as expressions and statements, based on the grammar of the Perl language. The optimizer rearranges and reduces these larger groupings into more efficient sequences. It picks its optimizations carefully, not wasting time on marginal optimizations, because the Perl compiler has to be blazing fast when used as a load-and-go compiler.

This doesn't happen in independent stages, but all at once with a lot of cross talk between the agents. The lexer occasionally needs hints from the parser to know which of several possible token types it's looking at. (Oddly, lexical scope is one of the things the lexical analyzer doesn't understand, because that's the other meaning of "lexical".) The optimizer also needs to keep track of what the parser is doing, because some optimizations can't happen until the parse has reached a certain point, like finishing an expression, statement, block, or subroutine.

You may think it odd that the Perl compiler does all these things at once instead of one after another, but it's really just the same messy process you go through to understand natural language on the fly, while you're listening to it or reading it. You don't wait till the end of a chapter to figure out what the first sentence meant. You could think of the following correspondences:

Computer LanguageNatural Language
CharacterLetter
TokenMorpheme
TermWord
ExpressionPhrase
StatementSentence
BlockParagraph
FileChapter
ProgramStory

Assuming the parse goes well, the compiler deems your input a valid story, er, program. If you use the -c switch when running your program, it prints out a "syntax OK" message and exits. Otherwise, the compiler passes the fruits of its efforts on to other agents. These "fruits" come in the form of a parse tree. Each fruit on the tree--or node, as it's called--represents one of Perl's internal opcodes, and the branches on the tree represent that tree's historical growth pattern. Eventually, the nodes will be strung together linearly, one after another, to indicate the execution order in which the run-time system will visit those nodes.

Each opcode is the smallest unit of executable instruction that Perl can think about. You might see an expression like $a = -($b + $c) as one statement, but Perl thinks of it as six separate opcodes. Laid out in a simplified format, the parse tree for that expression would look like Figure 18.2. The numbers represent the visitation order that the Perl run-time system will eventually follow.

Figure 18.2: Opcode visitation order of $a = -($b + $c)

Figure 18.2

Perl isn't a one-pass compiler as some might imagine. (One-pass compilers are great at making things easy for the computer and hard for the programmer.) It's really a multipass, optimizing compiler consisting of at least three different logical passes that are interleaved in practice. Passes 1 and 2 run alternately as the compiler repeatedly scurries up and down the parse tree during its construction; pass 3 happens whenever a subroutine or file is completely parsed. Here are those passes:

Pass 1: Bottom-Up Parsing
During this pass, the parse tree is built up by the yacc(1) parser using the tokens it's fed from the underlying lexer (which could be considered another logical pass in its own right). Bottom-up just means that the parser knows about the leaves of the tree before it knows about its branches and root. It really does figure things out from bottom to top in Figure 18.2, since we drew the root at the top, in the idiosyncratic fashion of computer scientists. (And linguists.)

As each opcode node is constructed, per-opcode sanity checks verify correct semantics, such as the correct number and types of arguments used to call built-in functions. As each subsection of the tree takes shape, the optimizer considers what transformations it can apply to the entire subtree now beneath it. For instance, once it knows that a list of values is being fed to a function that takes a specific number of arguments, it can throw away the opcode that records the number of arguments for functions that take a varying number of arguments. A more important optimization, known as constant folding, is described later in this section.

This pass also constructs the node visitation order used later for execution, which is a really neat trick because the first place to visit is almost never the top node. The compiler makes a temporary loop of opcodes, with the top node pointing to the first opcode to visit. When the top-level opcode is incorporated into something bigger, that loop of opcodes is broken, only to make a bigger loop with the new top node. Eventually the loop is broken for good when the start opcode gets poked into some other structure such as a subroutine descriptor. The subroutine caller can still find that first opcode despite its being way down at the bottom of the tree, as it is in Figure 18.2. There's no need for the interpreter to recurse back down the parse tree to figure out where to start.

Pass 2: Top-Down Optimizer

A person reading a snippet of Perl code (or of English code, for that matter) cannot determine the context without examining the surrounding lexical elements. Sometimes you can't decide what's really going on until you have more information. Don't feel bad, though, because you're not alone: neither can the compiler. In this pass, the compiler descends back down the subtree it's just built to apply local optimizations, the most notable of which is context propagation. The compiler marks subjacent nodes with the appropriate contexts (void, scalar, list, reference, or lvalue) imposed by the current node. Unwanted opcodes are nulled out but not deleted, because it's now too late to reconstruct the execution order. We'll rely on the third pass to remove them from the provisional execution order determined by the first pass.

Pass 3: Peephole Optimizer

Certain units of code have their own storage space in which they keep lexically scoped variables. (Such a space is called a scratchpad in Perl lingo.) These units include eval STRINGs, subroutines, and entire files. More importantly from the standpoint of the optimizer, they each have their own entry point, which means that while we know the execution order from here on, we can't know what happened before, because the construct could have been called from anywhere. So when one of these units is done being parsed, Perl runs a peephole optimizer on that code. Unlike the previous two passes, which walked the branch structure of the parse tree, this pass traverses the code in linear execution order, since this is basically the last opportunity to do so before we cut the opcode list off from the parser. Most optimizations were already performed in the first two passes, but some can't be.

Assorted late-term optimizations happen here, including stitching together the final execution order by skipping over nulled out opcodes, and recognizing when various opcode juxtapositions can be reduced to something simpler. The recognition of chained string concatenations is one important optimization, since you'd really like to avoid copying a string back and forth each time you add a little bit to the end. This pass doesn't just optimize; it also does a great deal of "real" work: trapping barewords, generating warnings on questionable constructs, checking for code unlikely to be reached, resolving pseudohash keys, and looking for subroutines called before their prototypes had been compiled.

Pass 4: Code Generation

This pass is optional; it isn't used in the normal scheme of things. But if any of the three code generators -- B::Bytecode, B::C, and B::CC -- are invoked, the parse tree is accessed one final time. The code generators emit either serialized Perl bytecodes used to reconstruct the parse tree later or literal C code representing the state of the compile-time parse tree.

Generation of C code comes in two different flavors. B::C simply reconstructs the parse tree and runs it using the usual runops() loop that Perl itself uses during execution. B::CC produces a linearized and optimized C equivalent of the run-time code path (which resembles a giant jump table) and executes that instead.

During compilation, Perl optimizes your code in many, many ways. It rearranges code to make it more efficient at execution time. It deletes code that can never be reached during execution, like an if (0) block, or the elsifs and the else in an if (1) block. If you use lexically typed variables declared with my ClassName $var or our ClassName $var, and the ClassName package was set up with the use fields pragma, accesses to constant fields from the underlying pseudohash are typo-checked at compile time and converted into array accesses instead. If you supply the sort operator with a simple enough comparison routine, such as {$a <=> $b} or {$b cmp $a}, this is replaced by a call to compiled C code.

Perl's most dramatic optimization is probably the way it resolves constant expressions as soon as possible. For example, consider the parse tree shown in Figure 18.2. If nodes 1 and 2 had both been literals or constant functions, nodes 1 through 4 would have been replaced by the result of that computation, something like Figure 18.3.

Figure 18.3: Constant folding

Figure 18.3

This is called constant folding. Constant folding isn't limited to simple cases such as turning 2**10 into 1024 at compile time. It also resolves function calls -- both built-ins and user-declared subroutines that meet the criteria from the section "Inlining Constant Functions" in Chapter 6, Subroutines. Reminiscent of FORTRAN compilers' notorious knowledge of their own intrinsic functions, Perl also knows which of its own built-ins to call during compilation. That's why if you try to take the log of 0.0 or the sqrt of a negative constant, you'll incur a compilation error, not a run-time error, and the interpreter is never run at all.[4]

[4] Actually, we're oversimplifying here. The interpreter does get run, because that's how the constant folder is implemented. But it is run immediately at compile time, similar to how BEGIN blocks are executed.

Even arbitrarily complicated expressions are resolved early, sometimes triggering the deletion of complete blocks such as the one here:

if (2 * sin(1)/cos(1) < 3 && somefn()) { whatever() }
No code is generated for what can never be evaluated. Because the first part is always false, neither somefn nor whatever can ever be called. (So don't expect to goto labels inside that block, because it won't even exist at run time.) If somefn were an inlinable constant function, then even switching the evaluation order like this:
if (somefn() && 2 * sin(1)/cos(1) < 3)) { whatever() }
wouldn't change the outcome, since the entire expression still resolves at compile time. If whatever were inlinable, it wouldn't be called at run time, nor even during compilation; its value would just be inserted as though it were a literal constant. You would then incur a warning about a "Useless use of a constant in void context". This might surprise you if you didn't realize it was a constant. However, if whatever were the last statement evaluated in a function called in a nonvoid context (as determined by the optimizer), you wouldn't see the warning.

You can see the final result of the constructed parse tree after all optimization stages with perl -Dx. (The -D switch requires a special, debugging-enabled build of Perl). Also see the section on B::Deparse described below.

All in all, the Perl compiler works hard (but not too hard) to optimize code so that, come run time, overall execution is sped up. It's about time to get your program running, so let's do that now.

Executing Your Code

To the first approximation, Sparc programs only run on Sparc machines, Intel programs only run on Intel machines, and Perl programs only run on Perl machines. A Perl machine possesses those attributes that a Perl program would find ideal in a computer: memory that is automatically allocated and deallocated, fundamental data types that are dynamic strings, arrays, and hashes, and have no size limits, and systems that all behave pretty much the same way. The job of the Perl interpreter is to make whatever computer it happens to be running on appear to be one of these idealistic Perl machines.

This fictitious machine presents the illusion of a computer specially designed to do nothing but run Perl programs. Each opcode produced by the compiler is a fundamental command in this emulated instruction set. Instead of a hardware program counter, the interpreter just keeps track of the current opcode to execute. Instead of a hardware stack pointer, the interpreter has its own virtual stack. This stack is very important because the Perl virtual machine (which we refuse to call a PVM) is a stack-based machine. Perl opcodes are internally called PP codes (short for "push-pop codes") because they manipulate the interpreter's virtual stack to find all operands, process temporary values, and store all results.

If you've ever programmed in Forth or PostScript, or used an HP scientific calculator with RPN ("Reverse Polish Notation") entry, you know how a stack machine works. Even if you haven't, the concept is simple: to add 3 and 4, you do things in the order 3 4 + instead of the more conventional 3 + 4. What this means in terms of the stack is that you push 3 and then 4 onto the stack, and + then pops both arguments off the stack, adds them, and pushes 7 back onto the stack, where it will sit until you do something else with it.

Compared with the Perl compiler, the Perl interpreter is a straightforward, almost boring, program. All it does is step through the compiled opcodes, one at a time, and dispatch them to the Perl run-time environment, that is, the Perl virtual machine. It's just a wad of C code, right?

Actually, it's not boring at all. A Perl virtual machine keeps track of a great deal of dynamic context on your behalf so that you don't have to. Perl maintains quite a few stacks, which you don't have to understand, but which we'll list here anyway just to impress you:

operand stack

That's the stack we already talked about.

save stack

Where localized values are saved pending restoration. Many internal routines also localize values without your knowing it.

scope stack

The lightweight dynamic context that controls when the save stack should be "popped".

context stack

The heavyweight dynamic context; who called whom to get where you are now. The caller function traverses this stack. Loop-control functions scan this stack to find out which loop to control. When you peel back the context stack, the scope stack gets peeled back appropriately, which restores all your local variables from the save stack, even if you left the earlier context by nefarious methods such as raising an exception and longjmp(3)ing out.

jumpenv stack

The stack of longjmp(3) contexts that allows us to raise exceptions or exit expeditiously.

return stack

Where we came from when we entered this subroutine.

mark stack

Where the current variadic argument list on the operand stack starts.

recursive lexical pad stacks

Where the lexical variables and other "scratch register" storage is kept when subroutines are called recursively.

And of course, there's the C stack on which all the C variables are stored. Perl actually tries to avoid relying on C's stack for the storage of saved values, since longjmp(3) bypasses the proper restoration of such values.

All this is to say that the usual view of an interpreter, a program that interprets another program, is really woefully inadequate to describe what's going on here. Yes, there's some C code implementing some opcodes, but when we say "interpreter", we mean something more than that, in the same way that when we say "musician", we mean something more than a set of DNA instructions for turning notes into sounds. Musicians are real, live organisms and have "state". So do interpreters.

Specifically, all this dynamic and lexical context, along with the global symbol tables, plus the parse trees, plus a thread of execution, is what we call an interpreter. As a context for execution, an interpreter really starts its existence even before the compiler starts, and can run in rudimentary form even as the compiler is building up the interpreter's context. In fact, that's precisely what's happening when the compiler calls into the interpreter to execute BEGIN blocks and such. And the interpreter can turn around and use the compiler to build itself up further. Every time you define another subroutine or load another module, the particular virtual Perl machine we call an interpreter is redefining itself. You can't really say that either the compiler or the interpreter is in control, because they're cooperating to control the bootstrap process we commonly call "running a Perl script". It's like bootstrapping a child's brain. Is it the DNA doing it or is it the neurons? A little of both, we think, with some input from external programmers.

It's possible to run multiple interpreters in the same process; they may or may not share parse trees, depending on whether they were started by cloning an existing interpreter or by building a new interpreter from scratch. It's also possible to run multiple threads in a single interpreter, in which case they share not only parse trees but also global symbols--see Chapter 17, Threads.

But most Perl programs use only a single Perl interpreter to execute their compiled code. And while you can run multiple, independent Perl interpreters within one process, the current API for this is only accessible from C.[5] Each individual Perl interpreter serves the role of a completely separate process, but doesn't cost as much to create as a whole new process does. That's how Apache's mod_perl extension gets such great performance: when you launch a CGI script under mod_perl, that script has already been compiled into Perl opcodes, eliminating the need for recompilation--but more importantly, eliminating the need to start a new process, which is the real bottleneck. Apache initializes a new Perl interpreter in an existing process and hands that interpreter the previously compiled code to execute. Of course, there's much more to it than that--there always is. For more about mod_perl, see Writing Apache Modules with Perl and C (O'Reilly, 1999).

[5] With one exception, so far: revision 5.6.0 of Perl can do cloned interpreters in support of fork emulation on Microsoft Windows. There may well be a Perl API to "ithreads", as they're called, by the time you read this.

Many other applications such as nvi, vim, and innd can embed Perl interpreters; we can't hope to list them all here. There are a number of commercial products that don't even advertise that they have embedded Perl engines. They just use it internally because it gets their job done in style.

Compiler Backends

So, if Apache can compile a Perl program now and execute it later, why can't you? Apache and other programs that contain embedded Perl interpreters have it easy--they never store the parse tree to an external file. If you're content with that approach, and don't mind using the C API to get at it, you can do the same thing. See the section "Embedding Perl" in Chapter 21, Internals and Externals, to learn how to access Perl from an enclosing C framework.

If you don't want to go that route, or have other needs, then there are a few options available. Instead of feeding the opcode output from the Perl compiler immediately into a Perl interpreter, you can invoke any of several alternative backends instead. These backends can serialize and store the compiled opcodes to an external file or even convert them into a couple different flavors of C code.

Please be aware that the code generators are all extremely experimental utilities that shouldn't be expected to work in a production environment. In fact, they shouldn't even be expected to work in a nonproduction environment except maybe once in a blue moon. Now that we've set your expectations low enough that any success at all will necessarily surpass them, it's safe to tell you how the backends work.

Some of the backend modules are code generators, like B::Bytecode, B::C, and B::CC. Others are really code-analysis and debugging tools, like B::Deparse, B::Lint, and B::Xref. Beyond those backends, the standard release includes several other low-level modules of potential interest to would-be authors of Perl code-development tools. Other backend modules can be found on CPAN, including (as of this writing) B::Fathom, B::Graph, B::JVM::Jasmin, and B::Size.

When you're using the Perl compiler for anything other than feeding the interpreter, the O module (that is, using the O.pm file) stands between the compiler and assorted backend modules. You don't call the backends directly; instead, you call the middle end, which in turn calls the designated backend. So, if you had a module called B::Backend, you would invoke it on a given script this way:
% perl -MO=Backend SCRIPTNAME
Some backends take options, specified as:
% perl -MO=Backend,OPTS SCRIPTNAME
Some backends already have their own frontends to invoke their middle ends for you so you don't have to remember their M.O. In particular, perlcc(1) invokes that code generator, which can be cumbersome to fire up.

Code Generators

The three current backends that convert Perl opcodes into some other format are all emphatically experimental. (Yes, we said this before, but we don't want you to forget.) Even when they happen to produce output that runs correctly, the resulting programs may take more disk space, more memory, and more CPU time than than they would ordinarily. This is an area of ongoing research and development. Things will get better.

The Bytecode Generator

The B::Bytecode module writes the parse tree's opcodes out in a platform-independent encoding. You can take a Perl script compiled down to bytecodes and copy that to any other machine with Perl installed on it.

The standard but currently experimental perlcc(1) command knows how to convert Perl source code into a byte-compiled Perl program. All you have to do is:
% perlcc -b -o pbyscript srcscript
And now you should be able to directly "execute" the resulting pbyscript. The start of that file looks somewhat like this:
#!/usr/bin/perl
use ByteLoader 0.03;
^C^@^E^A^C^@^@^@^A^F^@^C^@^@^@^B^F^@^C^@^@^@^C^F^@^C^@^@^@
B^@^@^@^H9^A8M-^?M-^?M-^?M-^?7M-^?M-^?M-^?M-^?6^@^@^@^A6^@
^G^D^D^@^@^@^KR^@^@^@^HS^@^@^@^HV^@M-2W<^FU^@^@^@^@X^Y@Z^@
...
There you find a small script header followed by purely binary data. This may seem like deep magic, but its dweomer, er, dwimmer is at most a minor one. The ByteLoader module uses a technique called a source filter to alter the source code before Perl gets a chance to see it. A source filter is a kind of preprocessor that applies to everything below it in the current file. Instead of being limited to simplistic transformations the way macro processors like cpp(1) and m4(1) are, here there are no constraints. Source filters have been used to augment Perl's syntax, to compress or encrypt source code, even to write Perl programs in Latin. E perlibus unicode; cogito, ergo substr; carp dbm, et al. Er, caveat scriptor.

The ByteLoader module is a source filter that knows how to disassemble the serialized opcodes produced by B::Bytecode to reconstruct the original parse tree. The reconstituted Perl code is spliced into the current parse tree without using the compiler. When the interpreter hits those opcodes, it just executes them as though they'd been there waiting for it all along.

The C Code Generators

The remaining code generators, B::C and B::CC, both produce C code instead of serialized Perl opcodes. The code they generate is far from readable, and if you try to read it you'll just go blind. It's not something you can use to plug little translated Perl-to-C bits into a larger C program. For that, see Chapter 21.

The B::C module just writes out the C data structures needed to recreate the entire Perl run-time environment. You get a dedicated interpreter with all the compiler-built data structures pre-initialized. In some senses, the code generated is like what B::Bytecode produces. Both are a straight translation of the opcode trees that the compiler built, but where B::Bytecode lays them out in symbolic form to be recreated later and plugged into a running Perl interpreter, B::C lays those opcodes down in C. When you compile this C code with your C compiler and link in the Perl library, the resulting program won't need a Perl interpreter installed on the target system. (It might need some shared libraries, though, if you didn't link everything statically.) However, this program isn't really any different than the regular Perl interpreter that runs your script. It's just precompiled into a standalone executable image.

The B::CC module, however, tries to do more than that. The beginning of the C source file it generates looks pretty much like what B::C produced,[6] but eventually, any similarity ends. In the B::C code, you have a big opcode table in C that's manipulated just as the interpreter would do on its own, whereas in the C code generated by B::CC is laid out in the order corresponding to the run-time flow of your program. It even has a C function corresponding to each function in your program. Some amount of optimization based on variable types is done; a few benchmarks can run twice as fast as in the standard interpreter. This is the most ambitious of the current code generators, the one that holds the greatest promise for the future. By no coincidence, it is also the least stable of the three.

[6] But then, so does everything once you've gone blind. Didn't we warn you not to peek?

Computer science students looking for graduate thesis projects need look no further. There are plenty of diamonds in the rough waiting to be polished off here.

Code Development Tools

The O module has many interesting Modi Operandi beyond feeding the exasperatingly experimental code generators. By providing relatively painless access to the Perl compiler's output, this module makes it easy to build other tools that need to know everything about a Perl program.

The B::Lint module is named after lint(1), the C program verifier. It inspects programs for questionable constructs that often trip up beginners but don't normally trigger warnings. Call the module directly:
% perl -MO=Lint,all myprog
Only a few checks are currently defined, such as using an array in an implicit scalar context, relying on default variables, and accessing another package's (nominally private) identifiers that start with _. See B::Lint(3) for details.

The B::Xref module generates cross-reference listings of the declaration and use of all variables (both global and lexically scoped), subroutines, and formats in a program, broken down by file and subroutine. Call the module this way:

% perl -MO=Xref myprog > myprof.pxref
For instance, here's a partial report:
Subroutine parse_argv
  Package (lexical)
    $on               i113, 114
    $opt              i113, 114
    %getopt_cfg       i107, 113
    @cfg_args         i112, 114, 116, 116
  Package Getopt::Long
    $ignorecase       101
    &GetOptions       &124
  Package main
    $Options          123, 124, 141, 150, 165, 169
    %$Options         141, 150, 165, 169
    &check_read       &167
    @ARGV             121, 157, 157, 162, 166, 166
This shows that the parse_argv subroutine had four lexical variables of its own; it also accessed global identifiers from both the main package and from Getopt::Long. The numbers are the lines where that item was used: a leading i indicates that the item was first introduced at the following line number, and a leading & means a subroutine was called there. Dereferences are listed separately, which is why both $Options and %$Options are shown.

The B::Deparse is a pretty printer that can demystify Perl code and help you understand what transformations the optimizer has taken with your code. For example, this shows what defaults Perl uses for various constructs:

% perl -MO=Deparse -ne 'for (1 .. 10) { print if -t }'
LINE: while (defined($_ = <ARGV>)) { foreach $_ (1 .. 10) { print $_ if -t STDIN; } }
The -p switch adds parentheses so you can see Perl's idea of precedence:
% perl -MO=Deparse,-p -e 'print $a ** 3 + sqrt(2) / 10 ** -2 ** $c'
print((($a ** 3) + (1.4142135623731 / (10 ** (-(2 ** $c))))));
You can use -q to see what primitives interpolated strings are compiled into:
% perl -MO=Deparse,-q -e '"A $name and some @ARGV\n"'
'A ' . $name . ' and some ' . join($", @ARGV) . "\n";
And this shows how Perl really compiles a three-part for loop into a while loop:
% perl -MO=Deparse -e 'for ($i=0;$i<10;$i++) { $x++ }'
$i = 0; while ($i < 10) { ++$x; } continue { ++$i }
You could even call B::Deparse on a Perl bytecode file produced by perlcc -b, and have it decompile that binary file for you. Serialized Perl opcodes may be a tad tough to read, but strong encryption they are not.

Avant-Garde Compiler, Retro Interpreter

There's a right time to think about everything; sometimes that time is beforehand, and sometimes it's after. Sometimes it's somewhere in the middle. Perl doesn't presume to know when it's the right time to think, so it gives the programmer a number of options for telling it when to think. Other times it knows that some sort of thinking is necessary but doesn't have any idea what it ought to think, so it needs ways of asking your program. Your program answers these kinds of questions by defining subroutines with names appropriate to what Perl is trying to find out.

Not only can the compiler call into the interpreter when it wants to be forward thinking, but the interpreter can also call back to the compiler when it wants to revise history. Your program can use several operators to call back into the compiler. Like the compiler, the interpreter can also call into named subroutines when it wants to find things out. Because of all this give and take between the compiler, the interpreter, and your program, you need to be aware of what things happen when. First we'll talk about when these named subroutines are triggered.

In Chapter 10, Packages, we saw how a package's AUTOLOAD subroutine is triggered when an undefined function in that package is called. In Chapter 12, Objects, we met the DESTROY method which is invoked when an object's memory is about to be automatically reclaimed by Perl. And in Chapter 14, Tied Variables, we encountered the many functions implicitly called when a tied variable is accessed.

These subroutines all follow the convention that, if a subroutine is triggered automatically by either the compiler or the interpreter, we write its name in uppercase. Associated with the different stages of your program's lifetime are four other such subroutines, named BEGIN, CHECK, INIT, and END. The sub keyword is optional before their declarations. Perhaps they are better called "blocks", because they're in some ways more like named blocks than real subroutines.

For instance, unlike regular subroutines, there's no harm in declaring these blocks multiple times, since Perl keeps track of when to call them, so you never have to call them by name. (They are also unlike regular subroutines in that shift and pop act as though you were in the main program, and so they act on @ARGV by default, not @_.)

These four block types run in this order:

BEGIN

Runs ASAP (as soon as parsed) whenever encountered during compilation, before compiling the rest of the file.

CHECK

Runs when compilation is complete, but before the program starts. (CHECK can mean "checkpoint" or "double-check" or even just "stop".)

INIT

Runs at the beginning of execution right before the main flow of your program starts.

END

Runs at the end of execution right after the program finishes.

If you declare more than one of these by the same name, even in separate modules, the BEGINs all run before any CHECKs, which all run before any INITs, which all run before any ENDs--which all run dead last, after your main program has finished. Multiple BEGINs and INITs run in declaration order (FIFO), and the CHECKs and ENDs run in inverse declaration order (LIFO).

This is probably easiest to see in a demo:

#!/usr/bin/perl -l
print       "start main running here";
die         "main now dying here\n";
die         "XXX: not reached\n";
END         { print "1st END: done running"   }
CHECK       { print "1st CHECK: done compiling" }
INIT        { print "1st INIT: started running"  }
END         { print "2nd END: done running"   }
BEGIN       { print "1st BEGIN: still compiling" }
INIT        { print "2nd INIT: started running"  }
BEGIN       { print "2nd BEGIN: still compiling" }
CHECK       { print "2nd CHECK: done compiling" }
END         { print "3rd END: done running"   }
When run, that demo program produces this output:
1st BEGIN: still compiling
2nd BEGIN: still compiling
2nd CHECK: done compiling
1st CHECK: done compiling
1st INIT: started running
2nd INIT: started running
start main running here
main now dying here
3rd END: done running
2nd END: done running
1st END: done running
Because a BEGIN block executes immediately, it can pull in subroutine declarations, definitions, and importations before the rest of the file is even compiled. These can alter how the compiler parses the rest of the current file, particularly if you import subroutine definitions. At the very least, declaring a subroutine lets it be used as a list operator, making parentheses optional. If the imported subroutine is declared with a prototype, calls to it can be parsed like built-ins and can even override built-ins of the same name in order to give them different semantics. The use declaration is just a BEGIN block with an attitude.

END blocks, by contrast, are executed as late as possible: when your program exits the Perl interpreter, even if as a result of an untrapped die or other fatal exception. There are two situations in which an END block (or a DESTROY method) is skipped. It isn't run if, instead of exiting, the current process just morphs itself from one program to another via exec. A process blown out of the water by an uncaught signal also skips its END routines. (See the use sigtrap pragma described in Chapter 31, Pragmatic Modules, for an easy way to convert catchable signals into exceptions. For general information on signal handling, see "Signals" in Chapter 16, Interprocess Communication.) To avoid all END processing, you can call POSIX::_exit, say kill -9, $$, or just exec any innocuous program, such as /bin/true on Unix systems.

Inside an END block, $? contains the status the program is going to exit with. You can modify $? from within the END block to change the exit value of the program. Beware of changing $? accidentally by running another program with system or backticks.

If you have several END blocks within a file, they execute in reverse order of their definition. That is, the last END block defined is the first one executed when your program finishes. This reversal enables related BEGIN and END blocks to nest the way you'd expect, if you pair them up. For example, if the main program and a module it loads both have their own paired BEGIN and END subroutines, like so:

BEGIN { print "main begun" }
END { print "main ended" }
use Module;
and in that module, these declarations:
BEGIN { print "module begun" }
END { print "module ended" }
then the main program knows that its BEGIN will always happen first, and its END will always happen last. (Yes, BEGIN is really a compile-time block, but similar arguments apply to paired INIT and END blocks at run time.) This principle is recursively true for any file that includes another when both have declarations like these. This nesting property makes these blocks work well as package constructors and destructors. Each module can have its own set-up and tear-down functions that Perl will call automatically. This way the programmer doesn't have to remember that if a particular library is used, what special initialization or clean-up code ought to be invoked, and when. The module's declarations assure these events.

If you think of an eval STRING as a call back from the interpreter to the compiler, then you might think of a BEGIN as a call forward from the compiler into the interpreter. Both temporarily put the current activity on hold and switch modes of operation. When we say that a BEGIN block is executed as early as possible, we mean it's executed just as soon as it is completely defined, even before the rest of the containing file is parsed. BEGIN blocks are therefore executed during compile time, never during run time. Once a BEGIN block has run, it is immediately undefined and any code it used is returned to Perl's memory pool. You couldn't call a BEGIN block as a subroutine even if you tried, because by the time it's there, it's already gone.

Similar to BEGIN blocks, INIT blocks are run just before the Perl run time begins execution, in "first in, first out" (FIFO) order. For example, the code generators documented in perlcc make use of INIT blocks to initialize and resolve pointers to XSUBs. INIT blocks are really just like BEGIN blocks, except they let the programmer distinguish construction that must happen at compile phase from construction that must happen at run phase. When you're running a script directly, that's not terribly important because the compiler gets invoked every time anyway; but when compilation is separate from execution, the distinction can be crucial. The compiler may only be invoked once, and the resulting executable may be invoked many times.

Similar to END blocks, CHECK blocks are run just after the Perl compile phase ends but before run phase begins, in LIFO order. CHECK blocks are useful for "winding down" the compiler just as END blocks are useful for winding down your program. In particular, the backends all use CHECK blocks as the hook from which to invoke their respective code generators. All they need to do is put a CHECK block into their own module, and it will run at the right time, so you don't have to install a CHECK into your program. For this reason, you'll rarely write a CHECK block yourself, unless you're writing such a module.

Putting it all together, Table 18.1 lists various constructs with details on when they compile and when they run the code represented by "…".

Table 18.1: What Happens When

Block

Compiles

Traps

Runs

Traps

Call

or

During

Compile

During

Run

Trigger

Expression

Phase

Errors

Phase

Errors

Policy

use …CNoCNoNow
no …CNoCNoNow
BEGIN {…}CNoCNoNow
CHECK {…}CNoCNoLate
INIT {…}CNoRNoEarly
END {…}CNoRNoLate
eval {…}CNoRYesInline
eval "…"RYesRYesInline
foo(…)CNoRNoInline
sub foo {…}CNoRNoCall anytime
eval "sub {…}"RYesRNoCall later
s/pat/…/eCNoRNoInline
s/pat/"…"/eeRYesRYesInline

Now that you know the score, we hope you'll be able to compose and perform your Perl pieces with greater confidence.

Back to: Programming Perl, 3rd Edition


O'Reilly Home | O'Reilly Bookstores | How to Order | O'Reilly Contacts
International | About O'Reilly | Affiliated Companies

© 2001, O'Reilly & Associates, Inc.
webmaster@oreilly.com