11556Srgrimes# @(#)TOUR 8.1 (Berkeley) 5/31/93 250471Speter# $FreeBSD$ 31556Srgrimes 41556SrgrimesNOTE -- This is the original TOUR paper distributed with ash and 51556Srgrimesdoes not represent the current state of the shell. It is provided anyway 61556Srgrimessince it provides helpful information for how the shell is structured, 71556Srgrimesbut be warned that things have changed -- the current shell is 81556Srgrimesstill under development. 91556Srgrimes 101556Srgrimes================================================================ 111556Srgrimes 121556Srgrimes A Tour through Ash 131556Srgrimes 141556Srgrimes Copyright 1989 by Kenneth Almquist. 151556Srgrimes 161556Srgrimes 171556SrgrimesDIRECTORIES: The subdirectory bltin contains commands which can 181556Srgrimesbe compiled stand-alone. The rest of the source is in the main 191556Srgrimesash directory. 201556Srgrimes 211556SrgrimesSOURCE CODE GENERATORS: Files whose names begin with "mk" are 221556Srgrimesprograms that generate source code. A complete list of these 231556Srgrimesprograms is: 241556Srgrimes 25157789Sschweikh program input files generates 26157789Sschweikh ------- ----------- --------- 271556Srgrimes mkbuiltins builtins builtins.h builtins.c 281556Srgrimes mkinit *.c init.c 291556Srgrimes mknodes nodetypes nodes.h nodes.c 301556Srgrimes mksyntax - syntax.h syntax.c 3117987Speter mktokens - token.h 321556Srgrimes 331556SrgrimesThere are undoubtedly too many of these. Mkinit searches all the 341556SrgrimesC source files for entries looking like: 351556Srgrimes 361556Srgrimes INIT { 371556Srgrimes x = 1; /* executed during initialization */ 381556Srgrimes } 391556Srgrimes 401556Srgrimes RESET { 411556Srgrimes x = 2; /* executed when the shell does a longjmp 421556Srgrimes back to the main command loop */ 431556Srgrimes } 441556Srgrimes 451556SrgrimesIt pulls this code out into routines which are when particular 461556Srgrimesevents occur. The intent is to improve modularity by isolating 471556Srgrimesthe information about which modules need to be explicitly 481556Srgrimesinitialized/reset within the modules themselves. 491556Srgrimes 501556SrgrimesMkinit recognizes several constructs for placing declarations in 511556Srgrimesthe init.c file. 521556Srgrimes INCLUDE "file.h" 531556Srgrimesincludes a file. The storage class MKINIT makes a declaration 541556Srgrimesavailable in the init.c file, for example: 551556Srgrimes MKINIT int funcnest; /* depth of function calls */ 561556SrgrimesMKINIT alone on a line introduces a structure or union declara- 571556Srgrimestion: 581556Srgrimes MKINIT 591556Srgrimes struct redirtab { 601556Srgrimes short renamed[10]; 611556Srgrimes }; 621556SrgrimesPreprocessor #define statements are copied to init.c without any 631556Srgrimesspecial action to request this. 641556Srgrimes 651556SrgrimesEXCEPTIONS: Code for dealing with exceptions appears in 661556Srgrimesexceptions.c. The C language doesn't include exception handling, 671556Srgrimesso I implement it using setjmp and longjmp. The global variable 681556Srgrimesexception contains the type of exception. EXERROR is raised by 69218306Sjillescalling error. EXINT is an interrupt. 701556Srgrimes 711556SrgrimesINTERRUPTS: In an interactive shell, an interrupt will cause an 721556SrgrimesEXINT exception to return to the main command loop. (Exception: 731556SrgrimesEXINT is not raised if the user traps interrupts using the trap 741556Srgrimescommand.) The INTOFF and INTON macros (defined in exception.h) 75157789Sschweikhprovide uninterruptible critical sections. Between the execution 761556Srgrimesof INTOFF and the execution of INTON, interrupt signals will be 771556Srgrimesheld for later delivery. INTOFF and INTON can be nested. 781556Srgrimes 791556SrgrimesMEMALLOC.C: Memalloc.c defines versions of malloc and realloc 801556Srgrimeswhich call error when there is no memory left. It also defines a 811556Srgrimesstack oriented memory allocation scheme. Allocating off a stack 821556Srgrimesis probably more efficient than allocation using malloc, but the 831556Srgrimesbig advantage is that when an exception occurs all we have to do 841556Srgrimesto free up the memory in use at the time of the exception is to 851556Srgrimesrestore the stack pointer. The stack is implemented using a 861556Srgrimeslinked list of blocks. 871556Srgrimes 881556SrgrimesSTPUTC: If the stack were contiguous, it would be easy to store 891556Srgrimesstrings on the stack without knowing in advance how long the 901556Srgrimesstring was going to be: 911556Srgrimes p = stackptr; 921556Srgrimes *p++ = c; /* repeated as many times as needed */ 931556Srgrimes stackptr = p; 94157789SschweikhThe following three macros (defined in memalloc.h) perform these 951556Srgrimesoperations, but grow the stack if you run off the end: 961556Srgrimes STARTSTACKSTR(p); 971556Srgrimes STPUTC(c, p); /* repeated as many times as needed */ 981556Srgrimes grabstackstr(p); 991556Srgrimes 1001556SrgrimesWe now start a top-down look at the code: 1011556Srgrimes 1021556SrgrimesMAIN.C: The main routine performs some initialization, executes 103157789Sschweikhthe user's profile if necessary, and calls cmdloop. Cmdloop 1041556Srgrimesrepeatedly parses and executes commands. 1051556Srgrimes 1061556SrgrimesOPTIONS.C: This file contains the option processing code. It is 1071556Srgrimescalled from main to parse the shell arguments when the shell is 108222362Sjillesinvoked, and it also contains the set builtin. The -i and -m op- 1091556Srgrimestions (the latter turns on job control) require changes in signal 1101556Srgrimeshandling. The routines setjobctl (in jobs.c) and setinteractive 1111556Srgrimes(in trap.c) are called to handle changes to these options. 1121556Srgrimes 1131556SrgrimesPARSING: The parser code is all in parser.c. A recursive des- 1141556Srgrimescent parser is used. Syntax tables (generated by mksyntax) are 1151556Srgrimesused to classify characters during lexical analysis. There are 116222362Sjillesfour tables: one for normal use, one for use when inside single 117222362Sjillesquotes and dollar single quotes, one for use when inside double 118222362Sjillesquotes and one for use in arithmetic. The tables are machine 119222362Sjillesdependent because they are indexed by character variables and 120222362Sjillesthe range of a char varies from machine to machine. 1211556Srgrimes 1221556SrgrimesPARSE OUTPUT: The output of the parser consists of a tree of 1231556Srgrimesnodes. The various types of nodes are defined in the file node- 1241556Srgrimestypes. 1251556Srgrimes 1261556SrgrimesNodes of type NARG are used to represent both words and the con- 1271556Srgrimestents of here documents. An early version of ash kept the con- 1281556Srgrimestents of here documents in temporary files, but keeping here do- 1291556Srgrimescuments in memory typically results in significantly better per- 1301556Srgrimesformance. It would have been nice to make it an option to use 1311556Srgrimestemporary files for here documents, for the benefit of small 1321556Srgrimesmachines, but the code to keep track of when to delete the tem- 1331556Srgrimesporary files was complex and I never fixed all the bugs in it. 1341556Srgrimes(AT&T has been maintaining the Bourne shell for more than ten 1351556Srgrimesyears, and to the best of my knowledge they still haven't gotten 1361556Srgrimesit to handle temporary files correctly in obscure cases.) 1371556Srgrimes 1381556SrgrimesThe text field of a NARG structure points to the text of the 1391556Srgrimesword. The text consists of ordinary characters and a number of 1401556Srgrimesspecial codes defined in parser.h. The special codes are: 1411556Srgrimes 1421556Srgrimes CTLVAR Variable substitution 1431556Srgrimes CTLENDVAR End of variable substitution 1441556Srgrimes CTLBACKQ Command substitution 1451556Srgrimes CTLBACKQ|CTLQUOTE Command substitution inside double quotes 1461556Srgrimes CTLESC Escape next character 1471556Srgrimes 1481556SrgrimesA variable substitution contains the following elements: 1491556Srgrimes 1501556Srgrimes CTLVAR type name '=' [ alternative-text CTLENDVAR ] 1511556Srgrimes 1521556SrgrimesThe type field is a single character specifying the type of sub- 1531556Srgrimesstitution. The possible types are: 1541556Srgrimes 1551556Srgrimes VSNORMAL $var 1561556Srgrimes VSMINUS ${var-text} 1571556Srgrimes VSMINUS|VSNUL ${var:-text} 1581556Srgrimes VSPLUS ${var+text} 1591556Srgrimes VSPLUS|VSNUL ${var:+text} 1601556Srgrimes VSQUESTION ${var?text} 1611556Srgrimes VSQUESTION|VSNUL ${var:?text} 1621556Srgrimes VSASSIGN ${var=text} 163157789Sschweikh VSASSIGN|VSNUL ${var:=text} 1641556Srgrimes 1651556SrgrimesIn addition, the type field will have the VSQUOTE flag set if the 1661556Srgrimesvariable is enclosed in double quotes. The name of the variable 1671556Srgrimescomes next, terminated by an equals sign. If the type is not 1681556SrgrimesVSNORMAL, then the text field in the substitution follows, ter- 1691556Srgrimesminated by a CTLENDVAR byte. 1701556Srgrimes 1711556SrgrimesCommands in back quotes are parsed and stored in a linked list. 1721556SrgrimesThe locations of these commands in the string are indicated by 1731556SrgrimesCTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether 1741556Srgrimesthe back quotes were enclosed in double quotes. 1751556Srgrimes 1761556SrgrimesThe character CTLESC escapes the next character, so that in case 1771556Srgrimesany of the CTL characters mentioned above appear in the input, 1781556Srgrimesthey can be passed through transparently. CTLESC is also used to 1791556Srgrimesescape '*', '?', '[', and '!' characters which were quoted by the 1801556Srgrimesuser and thus should not be used for file name generation. 1811556Srgrimes 1821556SrgrimesCTLESC characters have proved to be particularly tricky to get 1831556Srgrimesright. In the case of here documents which are not subject to 1841556Srgrimesvariable and command substitution, the parser doesn't insert any 1851556SrgrimesCTLESC characters to begin with (so the contents of the text 1861556Srgrimesfield can be written without any processing). Other here docu- 1871556Srgrimesments, and words which are not subject to splitting and file name 1881556Srgrimesgeneration, have the CTLESC characters removed during the vari- 189157789Sschweikhable and command substitution phase. Words which are subject to 1901556Srgrimessplitting and file name generation have the CTLESC characters re- 1911556Srgrimesmoved as part of the file name phase. 1921556Srgrimes 1931556SrgrimesEXECUTION: Command execution is handled by the following files: 1941556Srgrimes eval.c The top level routines. 1951556Srgrimes redir.c Code to handle redirection of input and output. 1961556Srgrimes jobs.c Code to handle forking, waiting, and job control. 197157789Sschweikh exec.c Code to do path searches and the actual exec sys call. 1981556Srgrimes expand.c Code to evaluate arguments. 1991556Srgrimes var.c Maintains the variable symbol table. Called from expand.c. 2001556Srgrimes 2011556SrgrimesEVAL.C: Evaltree recursively executes a parse tree. The exit 2021556Srgrimesstatus is returned in the global variable exitstatus. The alter- 2031556Srgrimesnative entry evalbackcmd is called to evaluate commands in back 2041556Srgrimesquotes. It saves the result in memory if the command is a buil- 2051556Srgrimestin; otherwise it forks off a child to execute the command and 2061556Srgrimesconnects the standard output of the child to a pipe. 2071556Srgrimes 2081556SrgrimesJOBS.C: To create a process, you call makejob to return a job 2091556Srgrimesstructure, and then call forkshell (passing the job structure as 2101556Srgrimesan argument) to create the process. Waitforjob waits for a job 2111556Srgrimesto complete. These routines take care of process groups if job 2121556Srgrimescontrol is defined. 2131556Srgrimes 2141556SrgrimesREDIR.C: Ash allows file descriptors to be redirected and then 2151556Srgrimesrestored without forking off a child process. This is accom- 2161556Srgrimesplished by duplicating the original file descriptors. The redir- 217157789Sschweikhtab structure records where the file descriptors have been dupli- 2181556Srgrimescated to. 2191556Srgrimes 2201556SrgrimesEXEC.C: The routine find_command locates a command, and enters 2211556Srgrimesthe command in the hash table if it is not already there. The 2221556Srgrimesthird argument specifies whether it is to print an error message 2231556Srgrimesif the command is not found. (When a pipeline is set up, 2241556Srgrimesfind_command is called for all the commands in the pipeline be- 2251556Srgrimesfore any forking is done, so to get the commands into the hash 2261556Srgrimestable of the parent process. But to make command hashing as 2271556Srgrimestransparent as possible, we silently ignore errors at that point 2281556Srgrimesand only print error messages if the command cannot be found 2291556Srgrimeslater.) 2301556Srgrimes 2311556SrgrimesThe routine shellexec is the interface to the exec system call. 2321556Srgrimes 2331556SrgrimesEXPAND.C: Arguments are processed in three passes. The first 2341556Srgrimes(performed by the routine argstr) performs variable and command 2351556Srgrimessubstitution. The second (ifsbreakup) performs word splitting 236222362Sjillesand the third (expandmeta) performs file name generation. 2371556Srgrimes 2381556SrgrimesVAR.C: Variables are stored in a hash table. Probably we should 2391556Srgrimesswitch to extensible hashing. The variable name is stored in the 2401556Srgrimessame string as the value (using the format "name=value") so that 2411556Srgrimesno string copying is needed to create the environment of a com- 2421556Srgrimesmand. Variables which the shell references internally are preal- 2431556Srgrimeslocated so that the shell can reference the values of these vari- 2441556Srgrimesables without doing a lookup. 2451556Srgrimes 2461556SrgrimesWhen a program is run, the code in eval.c sticks any environment 2471556Srgrimesvariables which precede the command (as in "PATH=xxx command") in 2481556Srgrimesthe variable table as the simplest way to strip duplicates, and 2491556Srgrimesthen calls "environment" to get the value of the environment. 2501556Srgrimes 2511556SrgrimesBUILTIN COMMANDS: The procedures for handling these are scat- 2521556Srgrimestered throughout the code, depending on which location appears 2531556Srgrimesmost appropriate. They can be recognized because their names al- 2541556Srgrimesways end in "cmd". The mapping from names to procedures is 255157789Sschweikhspecified in the file builtins, which is processed by the mkbuilt- 256157789Sschweikhins command. 2571556Srgrimes 2581556SrgrimesA builtin command is invoked with argc and argv set up like a 2591556Srgrimesnormal program. A builtin command is allowed to overwrite its 2601556Srgrimesarguments. Builtin routines can call nextopt to do option pars- 2611556Srgrimesing. This is kind of like getopt, but you don't pass argc and 2621556Srgrimesargv to it. Builtin routines can also call error. This routine 2631556Srgrimesnormally terminates the shell (or returns to the main command 2641556Srgrimesloop if the shell is interactive), but when called from a builtin 2651556Srgrimescommand it causes the builtin command to terminate with an exit 2661556Srgrimesstatus of 2. 2671556Srgrimes 2681556SrgrimesThe directory bltins contains commands which can be compiled in- 2691556Srgrimesdependently but can also be built into the shell for efficiency 2701556Srgrimesreasons. The makefile in this directory compiles these programs 2711556Srgrimesin the normal fashion (so that they can be run regardless of 2721556Srgrimeswhether the invoker is ash), but also creates a library named 2731556Srgrimesbltinlib.a which can be linked with ash. The header file bltin.h 2741556Srgrimestakes care of most of the differences between the ash and the 2751556Srgrimesstand-alone environment. The user should call the main routine 2761556Srgrimes"main", and #define main to be the name of the routine to use 2771556Srgrimeswhen the program is linked into ash. This #define should appear 2781556Srgrimesbefore bltin.h is included; bltin.h will #undef main if the pro- 2791556Srgrimesgram is to be compiled stand-alone. 2801556Srgrimes 281222362SjillesCD.C: This file defines the cd and pwd builtins. 2821556Srgrimes 2831556SrgrimesSIGNALS: Trap.c implements the trap command. The routine set- 2841556Srgrimessignal figures out what action should be taken when a signal is 2851556Srgrimesreceived and invokes the signal system call to set the signal ac- 2861556Srgrimestion appropriately. When a signal that a user has set a trap for 2871556Srgrimesis caught, the routine "onsig" sets a flag. The routine dotrap 2881556Srgrimesis called at appropriate points to actually handle the signal. 2891556SrgrimesWhen an interrupt is caught and no trap has been set for that 2901556Srgrimessignal, the routine "onint" in error.c is called. 2911556Srgrimes 2921556SrgrimesOUTPUT: Ash uses it's own output routines. There are three out- 2931556Srgrimesput structures allocated. "Output" represents the standard out- 2941556Srgrimesput, "errout" the standard error, and "memout" contains output 2951556Srgrimeswhich is to be stored in memory. This last is used when a buil- 2961556Srgrimestin command appears in backquotes, to allow its output to be col- 2971556Srgrimeslected without doing any I/O through the UNIX operating system. 2981556SrgrimesThe variables out1 and out2 normally point to output and errout, 2991556Srgrimesrespectively, but they are set to point to memout when appropri- 3001556Srgrimesate inside backquotes. 3011556Srgrimes 3021556SrgrimesINPUT: The basic input routine is pgetc, which reads from the 3031556Srgrimescurrent input file. There is a stack of input files; the current 3041556Srgrimesinput file is the top file on this stack. The code allows the 3051556Srgrimesinput to come from a string rather than a file. (This is for the 3061556Srgrimes-c option and the "." and eval builtin commands.) The global 3071556Srgrimesvariable plinno is saved and restored when files are pushed and 3081556Srgrimespopped from the stack. The parser routines store the number of 3091556Srgrimesthe current line in this variable. 3101556Srgrimes 3111556SrgrimesDEBUGGING: If DEBUG is defined in shell.h, then the shell will 3121556Srgrimeswrite debugging information to the file $HOME/trace. Most of 3131556Srgrimesthis is done using the TRACE macro, which takes a set of printf 3141556Srgrimesarguments inside two sets of parenthesis. Example: 3151556Srgrimes"TRACE(("n=%d0, n))". The double parenthesis are necessary be- 3161556Srgrimescause the preprocessor can't handle functions with a variable 3171556Srgrimesnumber of arguments. Defining DEBUG also causes the shell to 3181556Srgrimesgenerate a core dump if it is sent a quit signal. The tracing 3191556Srgrimescode is in show.c. 320