TOUR revision 1556
11556Srgrimes# @(#)TOUR 8.1 (Berkeley) 5/31/93 21556Srgrimes 31556SrgrimesNOTE -- This is the original TOUR paper distributed with ash and 41556Srgrimesdoes not represent the current state of the shell. It is provided anyway 51556Srgrimessince it provides helpful information for how the shell is structured, 61556Srgrimesbut be warned that things have changed -- the current shell is 71556Srgrimesstill under development. 81556Srgrimes 91556Srgrimes================================================================ 101556Srgrimes 111556Srgrimes A Tour through Ash 121556Srgrimes 131556Srgrimes Copyright 1989 by Kenneth Almquist. 141556Srgrimes 151556Srgrimes 161556SrgrimesDIRECTORIES: The subdirectory bltin contains commands which can 171556Srgrimesbe compiled stand-alone. The rest of the source is in the main 181556Srgrimesash directory. 191556Srgrimes 201556SrgrimesSOURCE CODE GENERATORS: Files whose names begin with "mk" are 211556Srgrimesprograms that generate source code. A complete list of these 221556Srgrimesprograms is: 231556Srgrimes 241556Srgrimes program intput files generates 251556Srgrimes ------- ------------ --------- 261556Srgrimes mkbuiltins builtins builtins.h builtins.c 271556Srgrimes mkinit *.c init.c 281556Srgrimes mknodes nodetypes nodes.h nodes.c 291556Srgrimes mksignames - signames.h signames.c 301556Srgrimes mksyntax - syntax.h syntax.c 311556Srgrimes mktokens - token.def 321556Srgrimes bltin/mkexpr unary_op binary_op operators.h operators.c 331556Srgrimes 341556SrgrimesThere are undoubtedly too many of these. Mkinit searches all the 351556SrgrimesC source files for entries looking like: 361556Srgrimes 371556Srgrimes INIT { 381556Srgrimes x = 1; /* executed during initialization */ 391556Srgrimes } 401556Srgrimes 411556Srgrimes RESET { 421556Srgrimes x = 2; /* executed when the shell does a longjmp 431556Srgrimes back to the main command loop */ 441556Srgrimes } 451556Srgrimes 461556Srgrimes SHELLPROC { 471556Srgrimes x = 3; /* executed when the shell runs a shell procedure */ 481556Srgrimes } 491556Srgrimes 501556SrgrimesIt pulls this code out into routines which are when particular 511556Srgrimesevents occur. The intent is to improve modularity by isolating 521556Srgrimesthe information about which modules need to be explicitly 531556Srgrimesinitialized/reset within the modules themselves. 541556Srgrimes 551556SrgrimesMkinit recognizes several constructs for placing declarations in 561556Srgrimesthe init.c file. 571556Srgrimes INCLUDE "file.h" 581556Srgrimesincludes a file. The storage class MKINIT makes a declaration 591556Srgrimesavailable in the init.c file, for example: 601556Srgrimes MKINIT int funcnest; /* depth of function calls */ 611556SrgrimesMKINIT alone on a line introduces a structure or union declara- 621556Srgrimestion: 631556Srgrimes MKINIT 641556Srgrimes struct redirtab { 651556Srgrimes short renamed[10]; 661556Srgrimes }; 671556SrgrimesPreprocessor #define statements are copied to init.c without any 681556Srgrimesspecial action to request this. 691556Srgrimes 701556SrgrimesINDENTATION: The ash source is indented in multiples of six 711556Srgrimesspaces. The only study that I have heard of on the subject con- 721556Srgrimescluded that the optimal amount to indent is in the range of four 731556Srgrimesto six spaces. I use six spaces since it is not too big a jump 741556Srgrimesfrom the widely used eight spaces. If you really hate six space 751556Srgrimesindentation, use the adjind (source included) program to change 761556Srgrimesit to something else. 771556Srgrimes 781556SrgrimesEXCEPTIONS: Code for dealing with exceptions appears in 791556Srgrimesexceptions.c. The C language doesn't include exception handling, 801556Srgrimesso I implement it using setjmp and longjmp. The global variable 811556Srgrimesexception contains the type of exception. EXERROR is raised by 821556Srgrimescalling error. EXINT is an interrupt. EXSHELLPROC is an excep- 831556Srgrimestion which is raised when a shell procedure is invoked. The pur- 841556Srgrimespose of EXSHELLPROC is to perform the cleanup actions associated 851556Srgrimeswith other exceptions. After these cleanup actions, the shell 861556Srgrimescan interpret a shell procedure itself without exec'ing a new 871556Srgrimescopy of the shell. 881556Srgrimes 891556SrgrimesINTERRUPTS: In an interactive shell, an interrupt will cause an 901556SrgrimesEXINT exception to return to the main command loop. (Exception: 911556SrgrimesEXINT is not raised if the user traps interrupts using the trap 921556Srgrimescommand.) The INTOFF and INTON macros (defined in exception.h) 931556Srgrimesprovide uninterruptable critical sections. Between the execution 941556Srgrimesof INTOFF and the execution of INTON, interrupt signals will be 951556Srgrimesheld for later delivery. INTOFF and INTON can be nested. 961556Srgrimes 971556SrgrimesMEMALLOC.C: Memalloc.c defines versions of malloc and realloc 981556Srgrimeswhich call error when there is no memory left. It also defines a 991556Srgrimesstack oriented memory allocation scheme. Allocating off a stack 1001556Srgrimesis probably more efficient than allocation using malloc, but the 1011556Srgrimesbig advantage is that when an exception occurs all we have to do 1021556Srgrimesto free up the memory in use at the time of the exception is to 1031556Srgrimesrestore the stack pointer. The stack is implemented using a 1041556Srgrimeslinked list of blocks. 1051556Srgrimes 1061556SrgrimesSTPUTC: If the stack were contiguous, it would be easy to store 1071556Srgrimesstrings on the stack without knowing in advance how long the 1081556Srgrimesstring was going to be: 1091556Srgrimes p = stackptr; 1101556Srgrimes *p++ = c; /* repeated as many times as needed */ 1111556Srgrimes stackptr = p; 1121556SrgrimesThe folloing three macros (defined in memalloc.h) perform these 1131556Srgrimesoperations, but grow the stack if you run off the end: 1141556Srgrimes STARTSTACKSTR(p); 1151556Srgrimes STPUTC(c, p); /* repeated as many times as needed */ 1161556Srgrimes grabstackstr(p); 1171556Srgrimes 1181556SrgrimesWe now start a top-down look at the code: 1191556Srgrimes 1201556SrgrimesMAIN.C: The main routine performs some initialization, executes 1211556Srgrimesthe user's profile if necessary, and calls cmdloop. Cmdloop is 1221556Srgrimesrepeatedly parses and executes commands. 1231556Srgrimes 1241556SrgrimesOPTIONS.C: This file contains the option processing code. It is 1251556Srgrimescalled from main to parse the shell arguments when the shell is 1261556Srgrimesinvoked, and it also contains the set builtin. The -i and -j op- 1271556Srgrimestions (the latter turns on job control) require changes in signal 1281556Srgrimeshandling. The routines setjobctl (in jobs.c) and setinteractive 1291556Srgrimes(in trap.c) are called to handle changes to these options. 1301556Srgrimes 1311556SrgrimesPARSING: The parser code is all in parser.c. A recursive des- 1321556Srgrimescent parser is used. Syntax tables (generated by mksyntax) are 1331556Srgrimesused to classify characters during lexical analysis. There are 1341556Srgrimesthree tables: one for normal use, one for use when inside single 1351556Srgrimesquotes, and one for use when inside double quotes. The tables 1361556Srgrimesare machine dependent because they are indexed by character vari- 1371556Srgrimesables and the range of a char varies from machine to machine. 1381556Srgrimes 1391556SrgrimesPARSE OUTPUT: The output of the parser consists of a tree of 1401556Srgrimesnodes. The various types of nodes are defined in the file node- 1411556Srgrimestypes. 1421556Srgrimes 1431556SrgrimesNodes of type NARG are used to represent both words and the con- 1441556Srgrimestents of here documents. An early version of ash kept the con- 1451556Srgrimestents of here documents in temporary files, but keeping here do- 1461556Srgrimescuments in memory typically results in significantly better per- 1471556Srgrimesformance. It would have been nice to make it an option to use 1481556Srgrimestemporary files for here documents, for the benefit of small 1491556Srgrimesmachines, but the code to keep track of when to delete the tem- 1501556Srgrimesporary files was complex and I never fixed all the bugs in it. 1511556Srgrimes(AT&T has been maintaining the Bourne shell for more than ten 1521556Srgrimesyears, and to the best of my knowledge they still haven't gotten 1531556Srgrimesit to handle temporary files correctly in obscure cases.) 1541556Srgrimes 1551556SrgrimesThe text field of a NARG structure points to the text of the 1561556Srgrimesword. The text consists of ordinary characters and a number of 1571556Srgrimesspecial codes defined in parser.h. The special codes are: 1581556Srgrimes 1591556Srgrimes CTLVAR Variable substitution 1601556Srgrimes CTLENDVAR End of variable substitution 1611556Srgrimes CTLBACKQ Command substitution 1621556Srgrimes CTLBACKQ|CTLQUOTE Command substitution inside double quotes 1631556Srgrimes CTLESC Escape next character 1641556Srgrimes 1651556SrgrimesA variable substitution contains the following elements: 1661556Srgrimes 1671556Srgrimes CTLVAR type name '=' [ alternative-text CTLENDVAR ] 1681556Srgrimes 1691556SrgrimesThe type field is a single character specifying the type of sub- 1701556Srgrimesstitution. The possible types are: 1711556Srgrimes 1721556Srgrimes VSNORMAL $var 1731556Srgrimes VSMINUS ${var-text} 1741556Srgrimes VSMINUS|VSNUL ${var:-text} 1751556Srgrimes VSPLUS ${var+text} 1761556Srgrimes VSPLUS|VSNUL ${var:+text} 1771556Srgrimes VSQUESTION ${var?text} 1781556Srgrimes VSQUESTION|VSNUL ${var:?text} 1791556Srgrimes VSASSIGN ${var=text} 1801556Srgrimes VSASSIGN|VSNUL ${var=text} 1811556Srgrimes 1821556SrgrimesIn addition, the type field will have the VSQUOTE flag set if the 1831556Srgrimesvariable is enclosed in double quotes. The name of the variable 1841556Srgrimescomes next, terminated by an equals sign. If the type is not 1851556SrgrimesVSNORMAL, then the text field in the substitution follows, ter- 1861556Srgrimesminated by a CTLENDVAR byte. 1871556Srgrimes 1881556SrgrimesCommands in back quotes are parsed and stored in a linked list. 1891556SrgrimesThe locations of these commands in the string are indicated by 1901556SrgrimesCTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether 1911556Srgrimesthe back quotes were enclosed in double quotes. 1921556Srgrimes 1931556SrgrimesThe character CTLESC escapes the next character, so that in case 1941556Srgrimesany of the CTL characters mentioned above appear in the input, 1951556Srgrimesthey can be passed through transparently. CTLESC is also used to 1961556Srgrimesescape '*', '?', '[', and '!' characters which were quoted by the 1971556Srgrimesuser and thus should not be used for file name generation. 1981556Srgrimes 1991556SrgrimesCTLESC characters have proved to be particularly tricky to get 2001556Srgrimesright. In the case of here documents which are not subject to 2011556Srgrimesvariable and command substitution, the parser doesn't insert any 2021556SrgrimesCTLESC characters to begin with (so the contents of the text 2031556Srgrimesfield can be written without any processing). Other here docu- 2041556Srgrimesments, and words which are not subject to splitting and file name 2051556Srgrimesgeneration, have the CTLESC characters removed during the vari- 2061556Srgrimesable and command substitution phase. Words which are subject 2071556Srgrimessplitting and file name generation have the CTLESC characters re- 2081556Srgrimesmoved as part of the file name phase. 2091556Srgrimes 2101556SrgrimesEXECUTION: Command execution is handled by the following files: 2111556Srgrimes eval.c The top level routines. 2121556Srgrimes redir.c Code to handle redirection of input and output. 2131556Srgrimes jobs.c Code to handle forking, waiting, and job control. 2141556Srgrimes exec.c Code to to path searches and the actual exec sys call. 2151556Srgrimes expand.c Code to evaluate arguments. 2161556Srgrimes var.c Maintains the variable symbol table. Called from expand.c. 2171556Srgrimes 2181556SrgrimesEVAL.C: Evaltree recursively executes a parse tree. The exit 2191556Srgrimesstatus is returned in the global variable exitstatus. The alter- 2201556Srgrimesnative entry evalbackcmd is called to evaluate commands in back 2211556Srgrimesquotes. It saves the result in memory if the command is a buil- 2221556Srgrimestin; otherwise it forks off a child to execute the command and 2231556Srgrimesconnects the standard output of the child to a pipe. 2241556Srgrimes 2251556SrgrimesJOBS.C: To create a process, you call makejob to return a job 2261556Srgrimesstructure, and then call forkshell (passing the job structure as 2271556Srgrimesan argument) to create the process. Waitforjob waits for a job 2281556Srgrimesto complete. These routines take care of process groups if job 2291556Srgrimescontrol is defined. 2301556Srgrimes 2311556SrgrimesREDIR.C: Ash allows file descriptors to be redirected and then 2321556Srgrimesrestored without forking off a child process. This is accom- 2331556Srgrimesplished by duplicating the original file descriptors. The redir- 2341556Srgrimestab structure records where the file descriptors have be dupli- 2351556Srgrimescated to. 2361556Srgrimes 2371556SrgrimesEXEC.C: The routine find_command locates a command, and enters 2381556Srgrimesthe command in the hash table if it is not already there. The 2391556Srgrimesthird argument specifies whether it is to print an error message 2401556Srgrimesif the command is not found. (When a pipeline is set up, 2411556Srgrimesfind_command is called for all the commands in the pipeline be- 2421556Srgrimesfore any forking is done, so to get the commands into the hash 2431556Srgrimestable of the parent process. But to make command hashing as 2441556Srgrimestransparent as possible, we silently ignore errors at that point 2451556Srgrimesand only print error messages if the command cannot be found 2461556Srgrimeslater.) 2471556Srgrimes 2481556SrgrimesThe routine shellexec is the interface to the exec system call. 2491556Srgrimes 2501556SrgrimesEXPAND.C: Arguments are processed in three passes. The first 2511556Srgrimes(performed by the routine argstr) performs variable and command 2521556Srgrimessubstitution. The second (ifsbreakup) performs word splitting 2531556Srgrimesand the third (expandmeta) performs file name generation. If the 2541556Srgrimes"/u" directory is simulated, then when "/u/username" is replaced 2551556Srgrimesby the user's home directory, the flag "didudir" is set. This 2561556Srgrimestells the cd command that it should print out the directory name, 2571556Srgrimesjust as it would if the "/u" directory were implemented using 2581556Srgrimessymbolic links. 2591556Srgrimes 2601556SrgrimesVAR.C: Variables are stored in a hash table. Probably we should 2611556Srgrimesswitch to extensible hashing. The variable name is stored in the 2621556Srgrimessame string as the value (using the format "name=value") so that 2631556Srgrimesno string copying is needed to create the environment of a com- 2641556Srgrimesmand. Variables which the shell references internally are preal- 2651556Srgrimeslocated so that the shell can reference the values of these vari- 2661556Srgrimesables without doing a lookup. 2671556Srgrimes 2681556SrgrimesWhen a program is run, the code in eval.c sticks any environment 2691556Srgrimesvariables which precede the command (as in "PATH=xxx command") in 2701556Srgrimesthe variable table as the simplest way to strip duplicates, and 2711556Srgrimesthen calls "environment" to get the value of the environment. 2721556SrgrimesThere are two consequences of this. First, if an assignment to 2731556SrgrimesPATH precedes the command, the value of PATH before the assign- 2741556Srgrimesment must be remembered and passed to shellexec. Second, if the 2751556Srgrimesprogram turns out to be a shell procedure, the strings from the 2761556Srgrimesenvironment variables which preceded the command must be pulled 2771556Srgrimesout of the table and replaced with strings obtained from malloc, 2781556Srgrimessince the former will automatically be freed when the stack (see 2791556Srgrimesthe entry on memalloc.c) is emptied. 2801556Srgrimes 2811556SrgrimesBUILTIN COMMANDS: The procedures for handling these are scat- 2821556Srgrimestered throughout the code, depending on which location appears 2831556Srgrimesmost appropriate. They can be recognized because their names al- 2841556Srgrimesways end in "cmd". The mapping from names to procedures is 2851556Srgrimesspecified in the file builtins, which is processed by the mkbuil- 2861556Srgrimestins command. 2871556Srgrimes 2881556SrgrimesA builtin command is invoked with argc and argv set up like a 2891556Srgrimesnormal program. A builtin command is allowed to overwrite its 2901556Srgrimesarguments. Builtin routines can call nextopt to do option pars- 2911556Srgrimesing. This is kind of like getopt, but you don't pass argc and 2921556Srgrimesargv to it. Builtin routines can also call error. This routine 2931556Srgrimesnormally terminates the shell (or returns to the main command 2941556Srgrimesloop if the shell is interactive), but when called from a builtin 2951556Srgrimescommand it causes the builtin command to terminate with an exit 2961556Srgrimesstatus of 2. 2971556Srgrimes 2981556SrgrimesThe directory bltins contains commands which can be compiled in- 2991556Srgrimesdependently but can also be built into the shell for efficiency 3001556Srgrimesreasons. The makefile in this directory compiles these programs 3011556Srgrimesin the normal fashion (so that they can be run regardless of 3021556Srgrimeswhether the invoker is ash), but also creates a library named 3031556Srgrimesbltinlib.a which can be linked with ash. The header file bltin.h 3041556Srgrimestakes care of most of the differences between the ash and the 3051556Srgrimesstand-alone environment. The user should call the main routine 3061556Srgrimes"main", and #define main to be the name of the routine to use 3071556Srgrimeswhen the program is linked into ash. This #define should appear 3081556Srgrimesbefore bltin.h is included; bltin.h will #undef main if the pro- 3091556Srgrimesgram is to be compiled stand-alone. 3101556Srgrimes 3111556SrgrimesCD.C: This file defines the cd and pwd builtins. The pwd com- 3121556Srgrimesmand runs /bin/pwd the first time it is invoked (unless the user 3131556Srgrimeshas already done a cd to an absolute pathname), but then 3141556Srgrimesremembers the current directory and updates it when the cd com- 3151556Srgrimesmand is run, so subsequent pwd commands run very fast. The main 3161556Srgrimescomplication in the cd command is in the docd command, which 3171556Srgrimesresolves symbolic links into actual names and informs the user 3181556Srgrimeswhere the user ended up if he crossed a symbolic link. 3191556Srgrimes 3201556SrgrimesSIGNALS: Trap.c implements the trap command. The routine set- 3211556Srgrimessignal figures out what action should be taken when a signal is 3221556Srgrimesreceived and invokes the signal system call to set the signal ac- 3231556Srgrimestion appropriately. When a signal that a user has set a trap for 3241556Srgrimesis caught, the routine "onsig" sets a flag. The routine dotrap 3251556Srgrimesis called at appropriate points to actually handle the signal. 3261556SrgrimesWhen an interrupt is caught and no trap has been set for that 3271556Srgrimessignal, the routine "onint" in error.c is called. 3281556Srgrimes 3291556SrgrimesOUTPUT: Ash uses it's own output routines. There are three out- 3301556Srgrimesput structures allocated. "Output" represents the standard out- 3311556Srgrimesput, "errout" the standard error, and "memout" contains output 3321556Srgrimeswhich is to be stored in memory. This last is used when a buil- 3331556Srgrimestin command appears in backquotes, to allow its output to be col- 3341556Srgrimeslected without doing any I/O through the UNIX operating system. 3351556SrgrimesThe variables out1 and out2 normally point to output and errout, 3361556Srgrimesrespectively, but they are set to point to memout when appropri- 3371556Srgrimesate inside backquotes. 3381556Srgrimes 3391556SrgrimesINPUT: The basic input routine is pgetc, which reads from the 3401556Srgrimescurrent input file. There is a stack of input files; the current 3411556Srgrimesinput file is the top file on this stack. The code allows the 3421556Srgrimesinput to come from a string rather than a file. (This is for the 3431556Srgrimes-c option and the "." and eval builtin commands.) The global 3441556Srgrimesvariable plinno is saved and restored when files are pushed and 3451556Srgrimespopped from the stack. The parser routines store the number of 3461556Srgrimesthe current line in this variable. 3471556Srgrimes 3481556SrgrimesDEBUGGING: If DEBUG is defined in shell.h, then the shell will 3491556Srgrimeswrite debugging information to the file $HOME/trace. Most of 3501556Srgrimesthis is done using the TRACE macro, which takes a set of printf 3511556Srgrimesarguments inside two sets of parenthesis. Example: 3521556Srgrimes"TRACE(("n=%d0, n))". The double parenthesis are necessary be- 3531556Srgrimescause the preprocessor can't handle functions with a variable 3541556Srgrimesnumber of arguments. Defining DEBUG also causes the shell to 3551556Srgrimesgenerate a core dump if it is sent a quit signal. The tracing 3561556Srgrimescode is in show.c. 357