TOUR revision 17987
11556Srgrimes# @(#)TOUR 8.1 (Berkeley) 5/31/93 217987Speter# $Id: TOUR,v 1.2 1994/09/24 02:57:19 davidg Exp $ 31556Srgrimes 41556SrgrimesNOTE -- This is the original TOUR paper distributed with ash and 51556Srgrimesdoes not represent the current state of the shell. It is provided anyway 61556Srgrimessince it provides helpful information for how the shell is structured, 71556Srgrimesbut be warned that things have changed -- the current shell is 81556Srgrimesstill under development. 91556Srgrimes 101556Srgrimes================================================================ 111556Srgrimes 121556Srgrimes A Tour through Ash 131556Srgrimes 141556Srgrimes Copyright 1989 by Kenneth Almquist. 151556Srgrimes 161556Srgrimes 171556SrgrimesDIRECTORIES: The subdirectory bltin contains commands which can 181556Srgrimesbe compiled stand-alone. The rest of the source is in the main 191556Srgrimesash directory. 201556Srgrimes 211556SrgrimesSOURCE CODE GENERATORS: Files whose names begin with "mk" are 221556Srgrimesprograms that generate source code. A complete list of these 231556Srgrimesprograms is: 241556Srgrimes 251556Srgrimes program intput files generates 261556Srgrimes ------- ------------ --------- 271556Srgrimes mkbuiltins builtins builtins.h builtins.c 281556Srgrimes mkinit *.c init.c 291556Srgrimes mknodes nodetypes nodes.h nodes.c 301556Srgrimes mksignames - signames.h signames.c 311556Srgrimes mksyntax - syntax.h syntax.c 3217987Speter mktokens - token.h 331556Srgrimes bltin/mkexpr unary_op binary_op operators.h operators.c 341556Srgrimes 351556SrgrimesThere are undoubtedly too many of these. Mkinit searches all the 361556SrgrimesC source files for entries looking like: 371556Srgrimes 381556Srgrimes INIT { 391556Srgrimes x = 1; /* executed during initialization */ 401556Srgrimes } 411556Srgrimes 421556Srgrimes RESET { 431556Srgrimes x = 2; /* executed when the shell does a longjmp 441556Srgrimes back to the main command loop */ 451556Srgrimes } 461556Srgrimes 471556Srgrimes SHELLPROC { 481556Srgrimes x = 3; /* executed when the shell runs a shell procedure */ 491556Srgrimes } 501556Srgrimes 511556SrgrimesIt pulls this code out into routines which are when particular 521556Srgrimesevents occur. The intent is to improve modularity by isolating 531556Srgrimesthe information about which modules need to be explicitly 541556Srgrimesinitialized/reset within the modules themselves. 551556Srgrimes 561556SrgrimesMkinit recognizes several constructs for placing declarations in 571556Srgrimesthe init.c file. 581556Srgrimes INCLUDE "file.h" 591556Srgrimesincludes a file. The storage class MKINIT makes a declaration 601556Srgrimesavailable in the init.c file, for example: 611556Srgrimes MKINIT int funcnest; /* depth of function calls */ 621556SrgrimesMKINIT alone on a line introduces a structure or union declara- 631556Srgrimestion: 641556Srgrimes MKINIT 651556Srgrimes struct redirtab { 661556Srgrimes short renamed[10]; 671556Srgrimes }; 681556SrgrimesPreprocessor #define statements are copied to init.c without any 691556Srgrimesspecial action to request this. 701556Srgrimes 711556SrgrimesINDENTATION: The ash source is indented in multiples of six 721556Srgrimesspaces. The only study that I have heard of on the subject con- 731556Srgrimescluded that the optimal amount to indent is in the range of four 741556Srgrimesto six spaces. I use six spaces since it is not too big a jump 751556Srgrimesfrom the widely used eight spaces. If you really hate six space 761556Srgrimesindentation, use the adjind (source included) program to change 771556Srgrimesit to something else. 781556Srgrimes 791556SrgrimesEXCEPTIONS: Code for dealing with exceptions appears in 801556Srgrimesexceptions.c. The C language doesn't include exception handling, 811556Srgrimesso I implement it using setjmp and longjmp. The global variable 821556Srgrimesexception contains the type of exception. EXERROR is raised by 831556Srgrimescalling error. EXINT is an interrupt. EXSHELLPROC is an excep- 841556Srgrimestion which is raised when a shell procedure is invoked. The pur- 851556Srgrimespose of EXSHELLPROC is to perform the cleanup actions associated 861556Srgrimeswith other exceptions. After these cleanup actions, the shell 871556Srgrimescan interpret a shell procedure itself without exec'ing a new 881556Srgrimescopy of the shell. 891556Srgrimes 901556SrgrimesINTERRUPTS: In an interactive shell, an interrupt will cause an 911556SrgrimesEXINT exception to return to the main command loop. (Exception: 921556SrgrimesEXINT is not raised if the user traps interrupts using the trap 931556Srgrimescommand.) The INTOFF and INTON macros (defined in exception.h) 941556Srgrimesprovide uninterruptable critical sections. Between the execution 951556Srgrimesof INTOFF and the execution of INTON, interrupt signals will be 961556Srgrimesheld for later delivery. INTOFF and INTON can be nested. 971556Srgrimes 981556SrgrimesMEMALLOC.C: Memalloc.c defines versions of malloc and realloc 991556Srgrimeswhich call error when there is no memory left. It also defines a 1001556Srgrimesstack oriented memory allocation scheme. Allocating off a stack 1011556Srgrimesis probably more efficient than allocation using malloc, but the 1021556Srgrimesbig advantage is that when an exception occurs all we have to do 1031556Srgrimesto free up the memory in use at the time of the exception is to 1041556Srgrimesrestore the stack pointer. The stack is implemented using a 1051556Srgrimeslinked list of blocks. 1061556Srgrimes 1071556SrgrimesSTPUTC: If the stack were contiguous, it would be easy to store 1081556Srgrimesstrings on the stack without knowing in advance how long the 1091556Srgrimesstring was going to be: 1101556Srgrimes p = stackptr; 1111556Srgrimes *p++ = c; /* repeated as many times as needed */ 1121556Srgrimes stackptr = p; 1131556SrgrimesThe folloing three macros (defined in memalloc.h) perform these 1141556Srgrimesoperations, but grow the stack if you run off the end: 1151556Srgrimes STARTSTACKSTR(p); 1161556Srgrimes STPUTC(c, p); /* repeated as many times as needed */ 1171556Srgrimes grabstackstr(p); 1181556Srgrimes 1191556SrgrimesWe now start a top-down look at the code: 1201556Srgrimes 1211556SrgrimesMAIN.C: The main routine performs some initialization, executes 1221556Srgrimesthe user's profile if necessary, and calls cmdloop. Cmdloop is 1231556Srgrimesrepeatedly parses and executes commands. 1241556Srgrimes 1251556SrgrimesOPTIONS.C: This file contains the option processing code. It is 1261556Srgrimescalled from main to parse the shell arguments when the shell is 1271556Srgrimesinvoked, and it also contains the set builtin. The -i and -j op- 1281556Srgrimestions (the latter turns on job control) require changes in signal 1291556Srgrimeshandling. The routines setjobctl (in jobs.c) and setinteractive 1301556Srgrimes(in trap.c) are called to handle changes to these options. 1311556Srgrimes 1321556SrgrimesPARSING: The parser code is all in parser.c. A recursive des- 1331556Srgrimescent parser is used. Syntax tables (generated by mksyntax) are 1341556Srgrimesused to classify characters during lexical analysis. There are 1351556Srgrimesthree tables: one for normal use, one for use when inside single 1361556Srgrimesquotes, and one for use when inside double quotes. The tables 1371556Srgrimesare machine dependent because they are indexed by character vari- 1381556Srgrimesables and the range of a char varies from machine to machine. 1391556Srgrimes 1401556SrgrimesPARSE OUTPUT: The output of the parser consists of a tree of 1411556Srgrimesnodes. The various types of nodes are defined in the file node- 1421556Srgrimestypes. 1431556Srgrimes 1441556SrgrimesNodes of type NARG are used to represent both words and the con- 1451556Srgrimestents of here documents. An early version of ash kept the con- 1461556Srgrimestents of here documents in temporary files, but keeping here do- 1471556Srgrimescuments in memory typically results in significantly better per- 1481556Srgrimesformance. It would have been nice to make it an option to use 1491556Srgrimestemporary files for here documents, for the benefit of small 1501556Srgrimesmachines, but the code to keep track of when to delete the tem- 1511556Srgrimesporary files was complex and I never fixed all the bugs in it. 1521556Srgrimes(AT&T has been maintaining the Bourne shell for more than ten 1531556Srgrimesyears, and to the best of my knowledge they still haven't gotten 1541556Srgrimesit to handle temporary files correctly in obscure cases.) 1551556Srgrimes 1561556SrgrimesThe text field of a NARG structure points to the text of the 1571556Srgrimesword. The text consists of ordinary characters and a number of 1581556Srgrimesspecial codes defined in parser.h. The special codes are: 1591556Srgrimes 1601556Srgrimes CTLVAR Variable substitution 1611556Srgrimes CTLENDVAR End of variable substitution 1621556Srgrimes CTLBACKQ Command substitution 1631556Srgrimes CTLBACKQ|CTLQUOTE Command substitution inside double quotes 1641556Srgrimes CTLESC Escape next character 1651556Srgrimes 1661556SrgrimesA variable substitution contains the following elements: 1671556Srgrimes 1681556Srgrimes CTLVAR type name '=' [ alternative-text CTLENDVAR ] 1691556Srgrimes 1701556SrgrimesThe type field is a single character specifying the type of sub- 1711556Srgrimesstitution. The possible types are: 1721556Srgrimes 1731556Srgrimes VSNORMAL $var 1741556Srgrimes VSMINUS ${var-text} 1751556Srgrimes VSMINUS|VSNUL ${var:-text} 1761556Srgrimes VSPLUS ${var+text} 1771556Srgrimes VSPLUS|VSNUL ${var:+text} 1781556Srgrimes VSQUESTION ${var?text} 1791556Srgrimes VSQUESTION|VSNUL ${var:?text} 1801556Srgrimes VSASSIGN ${var=text} 1811556Srgrimes VSASSIGN|VSNUL ${var=text} 1821556Srgrimes 1831556SrgrimesIn addition, the type field will have the VSQUOTE flag set if the 1841556Srgrimesvariable is enclosed in double quotes. The name of the variable 1851556Srgrimescomes next, terminated by an equals sign. If the type is not 1861556SrgrimesVSNORMAL, then the text field in the substitution follows, ter- 1871556Srgrimesminated by a CTLENDVAR byte. 1881556Srgrimes 1891556SrgrimesCommands in back quotes are parsed and stored in a linked list. 1901556SrgrimesThe locations of these commands in the string are indicated by 1911556SrgrimesCTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether 1921556Srgrimesthe back quotes were enclosed in double quotes. 1931556Srgrimes 1941556SrgrimesThe character CTLESC escapes the next character, so that in case 1951556Srgrimesany of the CTL characters mentioned above appear in the input, 1961556Srgrimesthey can be passed through transparently. CTLESC is also used to 1971556Srgrimesescape '*', '?', '[', and '!' characters which were quoted by the 1981556Srgrimesuser and thus should not be used for file name generation. 1991556Srgrimes 2001556SrgrimesCTLESC characters have proved to be particularly tricky to get 2011556Srgrimesright. In the case of here documents which are not subject to 2021556Srgrimesvariable and command substitution, the parser doesn't insert any 2031556SrgrimesCTLESC characters to begin with (so the contents of the text 2041556Srgrimesfield can be written without any processing). Other here docu- 2051556Srgrimesments, and words which are not subject to splitting and file name 2061556Srgrimesgeneration, have the CTLESC characters removed during the vari- 2071556Srgrimesable and command substitution phase. Words which are subject 2081556Srgrimessplitting and file name generation have the CTLESC characters re- 2091556Srgrimesmoved as part of the file name phase. 2101556Srgrimes 2111556SrgrimesEXECUTION: Command execution is handled by the following files: 2121556Srgrimes eval.c The top level routines. 2131556Srgrimes redir.c Code to handle redirection of input and output. 2141556Srgrimes jobs.c Code to handle forking, waiting, and job control. 2151556Srgrimes exec.c Code to to path searches and the actual exec sys call. 2161556Srgrimes expand.c Code to evaluate arguments. 2171556Srgrimes var.c Maintains the variable symbol table. Called from expand.c. 2181556Srgrimes 2191556SrgrimesEVAL.C: Evaltree recursively executes a parse tree. The exit 2201556Srgrimesstatus is returned in the global variable exitstatus. The alter- 2211556Srgrimesnative entry evalbackcmd is called to evaluate commands in back 2221556Srgrimesquotes. It saves the result in memory if the command is a buil- 2231556Srgrimestin; otherwise it forks off a child to execute the command and 2241556Srgrimesconnects the standard output of the child to a pipe. 2251556Srgrimes 2261556SrgrimesJOBS.C: To create a process, you call makejob to return a job 2271556Srgrimesstructure, and then call forkshell (passing the job structure as 2281556Srgrimesan argument) to create the process. Waitforjob waits for a job 2291556Srgrimesto complete. These routines take care of process groups if job 2301556Srgrimescontrol is defined. 2311556Srgrimes 2321556SrgrimesREDIR.C: Ash allows file descriptors to be redirected and then 2331556Srgrimesrestored without forking off a child process. This is accom- 2341556Srgrimesplished by duplicating the original file descriptors. The redir- 2351556Srgrimestab structure records where the file descriptors have be dupli- 2361556Srgrimescated to. 2371556Srgrimes 2381556SrgrimesEXEC.C: The routine find_command locates a command, and enters 2391556Srgrimesthe command in the hash table if it is not already there. The 2401556Srgrimesthird argument specifies whether it is to print an error message 2411556Srgrimesif the command is not found. (When a pipeline is set up, 2421556Srgrimesfind_command is called for all the commands in the pipeline be- 2431556Srgrimesfore any forking is done, so to get the commands into the hash 2441556Srgrimestable of the parent process. But to make command hashing as 2451556Srgrimestransparent as possible, we silently ignore errors at that point 2461556Srgrimesand only print error messages if the command cannot be found 2471556Srgrimeslater.) 2481556Srgrimes 2491556SrgrimesThe routine shellexec is the interface to the exec system call. 2501556Srgrimes 2511556SrgrimesEXPAND.C: Arguments are processed in three passes. The first 2521556Srgrimes(performed by the routine argstr) performs variable and command 2531556Srgrimessubstitution. The second (ifsbreakup) performs word splitting 2541556Srgrimesand the third (expandmeta) performs file name generation. If the 2551556Srgrimes"/u" directory is simulated, then when "/u/username" is replaced 2561556Srgrimesby the user's home directory, the flag "didudir" is set. This 2571556Srgrimestells the cd command that it should print out the directory name, 2581556Srgrimesjust as it would if the "/u" directory were implemented using 2591556Srgrimessymbolic links. 2601556Srgrimes 2611556SrgrimesVAR.C: Variables are stored in a hash table. Probably we should 2621556Srgrimesswitch to extensible hashing. The variable name is stored in the 2631556Srgrimessame string as the value (using the format "name=value") so that 2641556Srgrimesno string copying is needed to create the environment of a com- 2651556Srgrimesmand. Variables which the shell references internally are preal- 2661556Srgrimeslocated so that the shell can reference the values of these vari- 2671556Srgrimesables without doing a lookup. 2681556Srgrimes 2691556SrgrimesWhen a program is run, the code in eval.c sticks any environment 2701556Srgrimesvariables which precede the command (as in "PATH=xxx command") in 2711556Srgrimesthe variable table as the simplest way to strip duplicates, and 2721556Srgrimesthen calls "environment" to get the value of the environment. 2731556SrgrimesThere are two consequences of this. First, if an assignment to 2741556SrgrimesPATH precedes the command, the value of PATH before the assign- 2751556Srgrimesment must be remembered and passed to shellexec. Second, if the 2761556Srgrimesprogram turns out to be a shell procedure, the strings from the 2771556Srgrimesenvironment variables which preceded the command must be pulled 2781556Srgrimesout of the table and replaced with strings obtained from malloc, 2791556Srgrimessince the former will automatically be freed when the stack (see 2801556Srgrimesthe entry on memalloc.c) is emptied. 2811556Srgrimes 2821556SrgrimesBUILTIN COMMANDS: The procedures for handling these are scat- 2831556Srgrimestered throughout the code, depending on which location appears 2841556Srgrimesmost appropriate. They can be recognized because their names al- 2851556Srgrimesways end in "cmd". The mapping from names to procedures is 2861556Srgrimesspecified in the file builtins, which is processed by the mkbuil- 2871556Srgrimestins command. 2881556Srgrimes 2891556SrgrimesA builtin command is invoked with argc and argv set up like a 2901556Srgrimesnormal program. A builtin command is allowed to overwrite its 2911556Srgrimesarguments. Builtin routines can call nextopt to do option pars- 2921556Srgrimesing. This is kind of like getopt, but you don't pass argc and 2931556Srgrimesargv to it. Builtin routines can also call error. This routine 2941556Srgrimesnormally terminates the shell (or returns to the main command 2951556Srgrimesloop if the shell is interactive), but when called from a builtin 2961556Srgrimescommand it causes the builtin command to terminate with an exit 2971556Srgrimesstatus of 2. 2981556Srgrimes 2991556SrgrimesThe directory bltins contains commands which can be compiled in- 3001556Srgrimesdependently but can also be built into the shell for efficiency 3011556Srgrimesreasons. The makefile in this directory compiles these programs 3021556Srgrimesin the normal fashion (so that they can be run regardless of 3031556Srgrimeswhether the invoker is ash), but also creates a library named 3041556Srgrimesbltinlib.a which can be linked with ash. The header file bltin.h 3051556Srgrimestakes care of most of the differences between the ash and the 3061556Srgrimesstand-alone environment. The user should call the main routine 3071556Srgrimes"main", and #define main to be the name of the routine to use 3081556Srgrimeswhen the program is linked into ash. This #define should appear 3091556Srgrimesbefore bltin.h is included; bltin.h will #undef main if the pro- 3101556Srgrimesgram is to be compiled stand-alone. 3111556Srgrimes 3121556SrgrimesCD.C: This file defines the cd and pwd builtins. The pwd com- 3131556Srgrimesmand runs /bin/pwd the first time it is invoked (unless the user 3141556Srgrimeshas already done a cd to an absolute pathname), but then 3151556Srgrimesremembers the current directory and updates it when the cd com- 3161556Srgrimesmand is run, so subsequent pwd commands run very fast. The main 3171556Srgrimescomplication in the cd command is in the docd command, which 3181556Srgrimesresolves symbolic links into actual names and informs the user 3191556Srgrimeswhere the user ended up if he crossed a symbolic link. 3201556Srgrimes 3211556SrgrimesSIGNALS: Trap.c implements the trap command. The routine set- 3221556Srgrimessignal figures out what action should be taken when a signal is 3231556Srgrimesreceived and invokes the signal system call to set the signal ac- 3241556Srgrimestion appropriately. When a signal that a user has set a trap for 3251556Srgrimesis caught, the routine "onsig" sets a flag. The routine dotrap 3261556Srgrimesis called at appropriate points to actually handle the signal. 3271556SrgrimesWhen an interrupt is caught and no trap has been set for that 3281556Srgrimessignal, the routine "onint" in error.c is called. 3291556Srgrimes 3301556SrgrimesOUTPUT: Ash uses it's own output routines. There are three out- 3311556Srgrimesput structures allocated. "Output" represents the standard out- 3321556Srgrimesput, "errout" the standard error, and "memout" contains output 3331556Srgrimeswhich is to be stored in memory. This last is used when a buil- 3341556Srgrimestin command appears in backquotes, to allow its output to be col- 3351556Srgrimeslected without doing any I/O through the UNIX operating system. 3361556SrgrimesThe variables out1 and out2 normally point to output and errout, 3371556Srgrimesrespectively, but they are set to point to memout when appropri- 3381556Srgrimesate inside backquotes. 3391556Srgrimes 3401556SrgrimesINPUT: The basic input routine is pgetc, which reads from the 3411556Srgrimescurrent input file. There is a stack of input files; the current 3421556Srgrimesinput file is the top file on this stack. The code allows the 3431556Srgrimesinput to come from a string rather than a file. (This is for the 3441556Srgrimes-c option and the "." and eval builtin commands.) The global 3451556Srgrimesvariable plinno is saved and restored when files are pushed and 3461556Srgrimespopped from the stack. The parser routines store the number of 3471556Srgrimesthe current line in this variable. 3481556Srgrimes 3491556SrgrimesDEBUGGING: If DEBUG is defined in shell.h, then the shell will 3501556Srgrimeswrite debugging information to the file $HOME/trace. Most of 3511556Srgrimesthis is done using the TRACE macro, which takes a set of printf 3521556Srgrimesarguments inside two sets of parenthesis. Example: 3531556Srgrimes"TRACE(("n=%d0, n))". The double parenthesis are necessary be- 3541556Srgrimescause the preprocessor can't handle functions with a variable 3551556Srgrimesnumber of arguments. Defining DEBUG also causes the shell to 3561556Srgrimesgenerate a core dump if it is sent a quit signal. The tracing 3571556Srgrimescode is in show.c. 358