Advanced Bash-Scripting HOWTO: A guide to shell scripting, using Bash | ||
---|---|---|
Prev | Chapter 3. Tutorial / Reference | Next |
This is a descriptive listing of standard UNIX commands useful in shell scripts. The power of scripts comes from coupling system commands and shell directives with simple programming constructs.
prints (to stdout) an expression or variable (see Example 3-5).
1 echo Hello 2 echo $a |
Normally, each echo command prints a terminal newline, but the -n option suppresses this.
The basic file "list" command. It is all too easy to underestimate the power of this humble command. For example, using the -R, recursive option, ls provides a tree-like listing of a directory structure.
Example 3-46. Using ls to create a table of contents for burning a CDR disk
1 #!/bin/bash 2 3 # Script to automate burning a CDR. 4 5 # Uses Joerg Schilling's "cdrecord" package 6 # (http://www.fokus.gmd.de/nthp/employees/schilling/cdrecord.html) 7 8 # If this script invoked as an ordinary user, need to suid cdrecord 9 # (chmod u+s /usr/bin/cdrecord, as root). 10 11 if [ -z $1 ] 12 then 13 IMAGE_DIRECTORY=/opt 14 # Default directory, if not specified on command line. 15 else 16 IMAGE_DIRECTORY=$1 17 fi 18 19 ls -lRF $IMAGE_DIRECTORY > $IMAGE_DIRECTORY/contents 20 # The "l" option gives a "long" file listing. 21 # The "R" option makes the listing recursive. 22 # The "F" option marks the file types (directories suffixed by a /). 23 echo "Creating table of contents." 24 25 mkisofs -r -o cdimage.iso $IMAGE_DIRECTORY 26 echo "Creating ISO9660 file system image (cdimage.iso)." 27 28 cdrecord -v -isosize speed=2 dev=0,0 cdimage.iso 29 # Change speed parameter to speed of your burner. 30 echo "Burning the disk." 31 echo "Please be patient, this will take a while." 32 33 exit 0 |
cat, an acronym for concatenate, lists a file to stdout. When combined with redirection (> or >>), it is commonly used to concatenate files.
1 cat filename 2 cat file.1 file.2 file.3 > file.123 |
tac, is the inverse of cat, listing a file backwards from its end.
reverses each line of a file, and outputs to stdout. This is not the same effect as tac, as it preserves the order of the lines, but flips each one around.
bash$ cat file1.txt This is line 1. This is line 2. |
bash$ tac file1.txt This is line 2. This is line 1. |
bash$ rev file1.txt .1 enil si sihT .2 enil si sihT |
The familiar cd change directory command finds use in scripts where execution of a command requires being in a specified directory.
1 (cd /source/directory && tar cf - . ) | (cd /dest/directory && tar xvfp -) |
This is the file copy command. cp file1 file2 copies file1 to file2, overwriting file2 if it already exists (see Example 3-49).
This is the file move command. It is equivalent to a combination of cp and rm. It may be used to move multiple files to a directory. For some examples of using mv in a script, see Example 3-7 and Example A-2.
Delete (remove) a file or files. The -f forces removal of even readonly files.
![]() | When used with the recursive flag -r, this command removes files all the way down the directory tree. |
Remove directory. The directory must be empty of all files, including dotfiles, for this command to succeed.
Make directory, creates a new directory. mkdir -p project/programs/December creates the named directory. The -p option automatically creates any necessary parent directories.
Changes the attributes of an existing file (see Example 3-51).
1 chmod +x filename 2 # Makes "filename" executable for all users. |
1 chmod 644 filename 2 # Makes "filename" readable/writable to owner, readable to 3 # others 4 # (octal mode). |
1 chmod 1777 directory-name 2 # Gives everyone read, write, and execute permission in directory, 3 # however also sets the "sticky bit", which means that 4 # only the directory owner can change files in the directory. |
Change file attributes. This has the same effect as chmod above, but with a different invocation syntax.
Creates links to pre-existings files. Most often used with the -s, symbolic or "soft" link flag. This permits referencing the linked file by more than one name and is a superior alternative to aliasing (see Example 3-21).
ln -s oldfile newfile links the previously existing oldfile to the newly created link, newfile.
exec COMMAND \;
Carries out COMMAND on each file that find scores a hit on. COMMAND terminates with \; (the ; is escaped to make certain the shell passes it to find literally, which concludes the command sequence). If COMMAND contains {}, then find substitutes the full path name of the selected file.
Example 3-47. Badname, eliminate file names in current directory containing bad characters and white space.
1 #!/bin/bash 2 3 # Delete filenames in current directory containing bad characters. 4 5 for filename in * 6 do 7 badname=`echo "$filename" | sed -n /[\+\{\;\"\\\=\?~\(\)\<\>\&\*\|\$]/p` 8 # Files containing those nasties: + { ; " \ = ? ~ ( ) < > & * | $ 9 rm $badname 2>/dev/null 10 # So error messages deep-sixed. 11 done 12 13 # Now, take care of files containing all manner of whitespace. 14 find . -name "* *" -exec rm -f {} \; 15 # The path name of the file that "find" finds replaces the "{}". 16 # The '\' ensures that the ';' is interpreted literally, as end of command. 17 18 exit 0 |
See the man page for find for more detail.
A filter for feeding arguments to a command, and also a tool for assembling the commands themselves. It breaks a data stream into small enough chunks for filters and commands to process. Consider it as a powerful replacement for backquotes. In situations where backquotes fail with a too many arguments error, substituting xargs often works. Normally, xargs reads from 'stdin' or from a pipe, but it can also be given the output of a file.
ls | xargs -p -l gzip gzips every file in current directory, one at a time, prompting before each operation.
One of the more interesting xargs options is -n XX, which limits the number of arguments passed to XX.
ls | xargs -n 8 echo lists the files in the current directory in 8 columns.
![]() | The default command for xargs is echo. |
Example 3-48. Log file using xargs to monitor system log
1 #!/bin/bash 2 3 # Generates a log file in current directory 4 # from the tail end of /var/log messages. 5 6 # Note: /var/log/messages must be readable by ordinary users 7 # if invoked by same (#root chmod 755 /var/log/messages). 8 9 ( date; uname -a ) >>logfile 10 # Time and machine name 11 echo --------------------------------------------------------------------- >>logfile 12 tail -5 /var/log/messages | xargs | fmt -s >>logfile 13 echo >>logfile 14 echo >>logfile 15 16 exit 0 |
Example 3-49. copydir, copying files in current directory to another, using xargs
1 #!/bin/bash 2 3 # Copy (verbose) all files in current directory 4 # to directory specified on command line. 5 6 if [ -z $1 ] 7 # Exit if no argument given. 8 then 9 echo "Usage: `basename $0` directory-to-copy-to" 10 exit 1 11 fi 12 13 ls . | xargs -i -t cp ./{} $1 14 # This is the exact equivalent of 15 # cp * $1 16 17 exit 0 |
Translates into commands the arguments in a list (useful for code generation within a script).
Example 3-50. Showing the effect of eval
1 #!/bin/bash 2 3 y=`eval ls -l` 4 echo $y 5 6 y=`eval df` 7 echo $y 8 # Note that LF's not preserved 9 10 exit 0 |
Example 3-51. Forcing a log-off
1 #!/bin/bash 2 3 y=`eval ps ax | sed -n '/ppp/p' | awk '{ print $1 }'` 4 # Finding the process number of 'ppp' 5 6 kill -9 $y 7 # Killing it 8 9 10 # Restore to previous state... 11 12 chmod 666 /dev/ttyS3 13 # Doing a SIGKILL on ppp changes the permissions 14 # on the serial port. Must be restored. 15 16 rm /var/lock/LCK..ttyS3 17 # Remove the serial port lock file. 18 19 exit 0 |
All-purpose expression evaluator: Concatenates and evaluates the arguments according to the operation given (arguments must be separated by spaces). Operations may be arithmetic, comparison, string, or logical.
returns 8
returns 2
incrementing variable, same as let y=y+1 and y=$(($y+1)), as discussed elsewhere
Note that external programs, such as sed and Perl have far superior string parsing facilities, and it might well be advisable to use them instead of the built-in bash ones.
Example 3-52. Using expr
1 #!/bin/bash 2 3 # Demonstrating some of the uses of 'expr' 4 # +++++++++++++++++++++++++++++++++++++++ 5 6 echo 7 8 # Arithmetic Operators 9 10 echo Arithmetic Operators 11 echo 12 a=`expr 5 + 3` 13 echo 5 + 3 = $a 14 15 a=`expr $a + 1` 16 echo 17 echo a + 1 = $a 18 echo \(incrementing a variable\) 19 20 a=`expr 5 % 3` 21 # modulo 22 echo 23 echo 5 mod 3 = $a 24 25 echo 26 echo 27 28 # Logical Operators 29 30 echo Logical Operators 31 echo 32 33 a=3 34 echo a = $a 35 b=`expr $a \> 10` 36 echo 'b=`expr $a \> 10`, therefore...' 37 echo "If a > 10, b = 0 (false)" 38 echo b = $b 39 40 b=`expr $a \< 10` 41 echo "If a < 10, b = 1 (true)" 42 echo b = $b 43 44 45 echo 46 echo 47 48 # Comparison Operators 49 50 echo Comparison Operators 51 echo 52 a=zipper 53 echo a is $a 54 if [ `expr $a = snap` ] 55 # Force re-evaluation of variable 'a' 56 then 57 echo "a is not zipper" 58 fi 59 60 echo 61 echo 62 63 64 65 # String Operators 66 67 echo String Operators 68 echo 69 70 a=1234zipper43231 71 echo The string being operated upon is $a. 72 73 # index: position of substring 74 b=`expr index $a 23` 75 echo Numerical position of first 23 in $a is $b. 76 77 # substr: print substring, starting position & length specified 78 b=`expr substr $a 2 6` 79 echo Substring of $a, starting at position 2 and 6 chars long is $b. 80 81 # length: length of string 82 b=`expr length $a` 83 echo Length of $a is $b. 84 85 # 'match' operations similarly to 'grep' 86 b=`expr match $a [0-9]*` 87 echo Number of digits at the beginning of $a is $b. 88 b=`expr match $a '\([0-9]*\)'` 89 echo The digits at the beginning of $a are $b. 90 91 echo 92 93 exit 0 |
Note that : can substitute for match. b=`expr $a : [0-9]*` is an exact equivalent of b=`expr match $a [0-9]*` in the above example.
The let command carries out arithmetic operations on variables. In many cases, it functions as a less complex version of expr.
Example 3-53. Letting let do some arithmetic.
1 #!/bin/bash 2 3 echo 4 5 let a=11 6 # Same as 'a=11' 7 let a=a+5 8 # Equivalent to let "a = a + 5" 9 # (double quotes makes it more readable) 10 echo "a = $a" 11 let "a <<= 3" 12 # Equivalent of let "a = a << 3" 13 echo "a left-shifted 3 places = $a" 14 15 let "a /= 4" 16 # Equivalent to let "a = a / 4" 17 echo $a 18 let "a -= 5" 19 # Equivalent to let "a = a - 5" 20 echo $a 21 let "a = a * 10" 22 echo $a 23 let "a %= 8" 24 echo $a 25 26 exit 0 |
Simply invoked, date prints the date and time to stdout. Where this command gets interesting is in its formatting and parsing options.
Example 3-54. Using date
1 #!/bin/bash 2 3 #Using the 'date' command 4 5 # Needs a leading '+' to invoke formatting. 6 7 echo "The number of days since the year's beginning is `date +%j`." 8 # %j gives day of year. 9 10 11 echo "The number of seconds elapsed since 01/01/1970 is `date +%s`." 12 # %s yields number of seconds since "UNIX epoch" began, 13 # but how is this useful? 14 15 prefix=temp 16 suffix=`eval date +%s` 17 filename=$prefix.$suffix 18 echo $filename 19 # It's great for creating "unique" temp filenames, 20 # even better than using $$. 21 22 # Read the 'date' man page for more formatting options. 23 24 exit 0 |
Outputs very verbose timing statistics for executing a command.
time ls -l / gives something like this:
0.00user 0.01system 0:00.05elapsed 16%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (149major+27minor)pagefaults 0swaps |
See also the very similar times command in the previous section.
Utility for updating access/modification times of a file to current system time or other specified time, but also useful for creating a new file. The command touch zzz will create a new file of zero length, named zzz, assuming that zzz did not previously exist. Time-stamping empty files in this way is useful for storing date information, for example in keeping track of modification times on a project. See Example 3-11.
The at job control command executes a given set of commands at a specified time. This is a user version of cron.
at 2pm January 15 prompts for a set of commands to execute at that time. These commands may include executable shell scripts.
Using either the -f option or input redirection (<), at reads a command list from a file. This file can include shell scripts, though they should, of course, be noninteractive.
bash$ at 2:30 am Friday < at-jobs.list job 2 at 2000-10-27 02:30 |
The batch job control command is similar to at, but it runs a command list when the system load drops below .8. Like at, it can read commands from a file with the -f option.
Prints a neatly formatted monthly calendar to stdout. Will do current year or a large range of past and future years.
This is the shell equivalent of a wait loop. It pauses for a specified number of seconds, doing nothing. This can be useful for timing or in processes running in the background, checking for a specific event every so often (see Example 3-101).
1 sleep 3 2 # Pauses 3 seconds. |
File sorter, often used as a filter in a pipe. See the man page for options.
Simple file comparison utility. This compares the target files line-by-line sequentially. In some applications, such as comparing word dictionaries, it may be helpful to filter the files through sort and uniq before piping them to diff. diff file-1 file-2 outputs the lines in the files that differ, with carets showing which file each particular line belongs to. A common use for diff is generating difference files to be used with patch (see below). The -e option outputs files suitable for ed or ex scripts.
1 patch -p1 <patch-file 2 # Takes all the changes listed in 'patch-file' and applies them 3 # to the files referenced therein. 4 5 cd /usr/src 6 gzip -cd patchXX.gz | patch -p0 7 # Upgrading kernel source using 'patch'. 8 # From the Linux kernel docs "README", 9 # by anonymous author (Alan Cox?). |
There are available various fancy frontends for diff, such as spiff, wdiff, xdiff, and mgdiff.
Versatile file comparison utility. The files must be sorted for this to be useful.
comm -options first-file second-file
comm file-1 file-2 outputs three columns:
column 1 = lines unique to file-1
column 2 = lines unique to file-2
column 3 = lines common to both.
The options allow suppressing output of one or more columns.
-1 suppresses column 1
-2 suppresses column 2
-3 suppresses column 3
-12 suppresses both columns 1 and 2, etc.
This filter removes duplicate lines from a sorted file. It is often seen in a pipe coupled with sort.
1 cat list-1 list-2 list-3 | sort | uniq > final.list 2 # Concatenates the list files, 3 # sorts them, 4 # removes duplicate lines, 5 # and finally writes the result to an output file. |
A filter than converts tabs to spaces, often seen in a pipe.
A tool for extracting fields from files. It is similar to the print $N command set in awk, but more limited. It may be simpler to use cut in a script than awk. Particularly important are the -d (delimiter) and -f (field specifier) options.
Using cut to obtain a listing of the mounted filesystems:
1 cat /etc/mtab | cut -d ' ' -f1,2 |
Using cut to list the OS and kernel version:
1 uname -a | cut -d" " -f1,3,11,12 |
cut -d ' ' -f2,3 filename is equivalent to awk '{ print $2, $3 }' filename
Column removal filter. This removes columns (characters) from a file and writes them, lacking the range of specified columns, back to stdout. colrm 2 4 <filename removes the second through fourth characters from each line of the text file filename.
![]() | If the file contains tabs or nonprintable characters, this may cause unpredictable behavior. |
Tool for merging together different files into a single, multi-column file. In combination with cut, useful for creating system log files.
Consider this a more flexible version of paste. It works on exactly two files, but permits specifying which fields to paste together, and in which order.
lists the first 10 lines of a file to stdout (see Example 3-66).
lists the end of a file to stdout (the default is 10 lines, but this can be changed). Commonly used to keep track of changes to a system logfile, using the -f option, which outputs lines appended to the file.
Example 3-48, Example 3-66, and Example 3-101 show tail in action.
A multi-purpose file search tool that uses regular expressions. Originally a command/filter in the ancient ed line editor, g/re/p, or global - regular expression - print.
grep pattern [file...]
search the files file, etc. for occurrences of pattern.ls -l | grep '.txt' has the same effect as ls -l *.txt.
The -i option to grep causes a case-insensitive search.
Example 3-101 demonstrates how to use grep to search for a keyword in a system log file.
Example 3-55. Emulating "grep" in a script
1 #!/bin/bash 2 3 # Very crude reimplementation of 'grep'. 4 5 if [ -z $1 ] # Check for argument to script. 6 then 7 echo "Usage: `basename $0` pattern" 8 exit 1 9 fi 10 11 echo 12 13 for file in * # Traverse all files in $PWD. 14 do 15 output=$(sed -n /"$1"/p $file) # Command substitution. 16 17 if [ ! -z "$output" ] # Variable $file quoted, otherwise error on multi-line output. 18 then 19 echo -n "$file: " 20 echo $output 21 fi 22 23 echo 24 done 25 26 echo 27 28 exit 0 29 30 # Exercises for reader: 31 # ------------------- 32 # 1) Add newlines to output, if more than one match in any given file. 33 # 2) Add features. |
![]() | egrep is the same as grep -E. This uses a somewhat different, extended set of regular expressions, which may make the search somewhat more flexible. |
![]() | fgrep is the same as grep -F. It does a literal string search (no regular expressions), which generally speeds things up quite a bit. |
![]() | To search compressed files, use zgrep. It also works on non-compressed files, though slower than plain grep. This is handy for searching through a mixed set of files, some of them compressed, some not. |
The command look works like grep, but does a lookup on a "dictionary", a sorted word list. By default, look searches for a match in /usr/dict/words, but a different dictionary file may be specified.
Example 3-56. Checking words in a list for validity
1 #!/bin/bash 2 # lookup: 3 # Does a dictionary lookup on each word in a data file. 4 5 file=words.data # Data file to read words to test from. 6 7 echo 8 9 while [ "$word" != end ] # Last word in data file. 10 do 11 read word # From data file, because of redirection at end of loop. 12 look $word > /dev/null # Don't want to display lines in dictionary file. 13 lookup=$? # Exit value of 'look'. 14 15 if [ "$lookup" -eq 0 ] 16 then 17 echo "\"$word\" is valid." 18 else 19 echo "\"$word\" is invalid." 20 fi 21 22 done <$file # Redirects stdin to $file, so "reads" come from there. 23 24 echo 25 26 exit 0 |
Scripting languages especially suited for parsing text files and command output. May be embedded singly or in combination in pipes and shell scripts.
Non-interactive "stream editor", permits using many ex commands in batch mode. It finds many uses in shell scripts. See Appendix B.
Programmable file extractor and formatter, good for manipulating and/or extracting fields (columns) in structured text files. Its syntax is similar to C. See Section B.2.
Text markup languages. Used for preparing copy for printing or formatted video display.
Man pages use groff (see Example A-1). Ghostscript (gs) is the GPL version of Postscript. TeX is Donald Knuth's elaborate typsetting system. It is often convenient to write a shell script encapsulating all the options and arguments passed to one of these markup languages.
wc gives a "word count" on a file or I/O stream:
$ wc /usr/doc/sed-3.02/README 20 127 838 /usr/doc/sed-3.02/README [20 lines 127 words 838 characters] |
wc -w gives only the word count.
wc -l gives only the line count.
wc -c gives only the character count.
wc -L gives only the length of the longest line.
Using wc to count how many .txt files are in current working directory:
1 $ ls *.txt | wc -l |
See Example 3-66 and Example 3-75.
character translation filter.
![]() | must use quoting and/or brackets, as appropriate. |
tr "A-Z" "*" <filename changes all the uppercase letters in filename to asterisks (writes to stdout).
tr -d [0-9] <filename deletes all digits from the file filename.
Example 3-57. toupper: Transforms a file to all uppercase.
1 #!/bin/bash 2 3 # Changes a file to all uppercase. 4 5 if [ -z $1 ] 6 # Standard check whether command line arg is present. 7 then 8 echo "Usage: `basename $0` filename" 9 exit 1 10 fi 11 12 tr [a-z] [A-Z] <$1 13 14 exit 0 |
Example 3-58. lowercase: Changes all filenames in working directory to lowercase.
1 #! /bin/bash 2 # 3 # Changes every filename in working directory to all lowercase. 4 # 5 # Inspired by a script of john dubois, 6 # which was translated into into bash by Chet Ramey, 7 # and considerably simplified by Mendel Cooper, 8 # author of this HOWTO. 9 10 11 for filename in * #Traverse all files in directory. 12 do 13 fname=`basename $filename` 14 n=`echo $fname | tr A-Z a-z` #Change name to lowercase. 15 if [ $fname != $n ] # Rename only files not already lowercase. 16 then 17 mv $fname $n 18 fi 19 done 20 21 exit 0 |
Example 3-59. rot13: rot13, ultra-weak encryption.
1 #!/bin/bash 2 # Classic rot13 algorithm, encryption that might fool a 3-year old. 3 # Usage: ./rot13.sh filename 4 # or ./rot13.sh <filename 5 # or ./rot13.sh and supply keyboard input (stdin) 6 7 cat "$@" | tr 'a-zA-Z' 'n-za-mN-ZA-M' # "a" goes to "n", "b" to "o", etc. 8 # The 'cat "$@"' construction permits getting input either from stdin or from a file. 9 10 exit 0 |
A filter that wraps inputted lines to a specified width (see Example 3-62).
Simple-minded file formatter, used as a filter in a pipe to "wrap" long lines of text output (see Example 3-48 and Example 3-62).
The ptx [targetfile] command outputs a permuted index (cross-reference list) of the targetfile. This may be further filtered and formatted in a pipe, if necessary.
Column formatter. This filter transforms list-type text output into a "pretty-printed" table by inserting tabs at appropriate places.
Example 3-60. Using column to format a directory listing
1 #!/bin/bash 2 # This is a slight modification of the example file in the "column" man page. 3 4 5 (printf "PERMISSIONS LINKS OWNER GROUP SIZE MONTH DAY HH:MM PROG-NAME\n" \ 6 ; ls -l | sed 1d) | column -t 7 8 # The "sed 1d" in the pipe deletes the first line of output, 9 # which would be "total N", 10 # where "N" is the total number of files found by "ls -l". 11 12 # The -t option to "column" pretty-prints a table. 13 14 exit 0 |
Line numbering filter. nl filename lists filename to stdout, but inserts consecutive numbers at the beginning of each non-blank line. If filename omitted, operates on stdin.
Example 3-61. nl: A self-numbering script.
1 #!/bin/bash 2 3 # This file echoes itself twice to stdout with its lines numbered. 4 5 # 'nl' sees this as line 3 since it does not number blank lines. 6 # 'cat -n' sees the above line as number 5. 7 8 nl `basename $0` 9 10 echo; echo # Now, let's try it with 'cat -n' 11 12 cat -n `basename $0` 13 # The difference is that 'cat -n' numbers the blank lines. 14 15 exit 0 |
Print formatting filter. This will paginate a file (or stdout) into sections suitable for hard copy printing. A particularly useful option is -d, forcing double-spacing.
The printf, formatted print, command is an enhanced echo. It is a limited variant of the C language printf, and the syntax is somewhat different.
printf format-string... parameter...
See the printf man page for in-depth coverage.
![]() | Older versions of bash may not support printf. |
Example 3-63. printf in action
1 #!/bin/bash 2 3 # printf demo 4 5 PI=3.14159265358979 6 DecimalConstant=31373 7 Message1="Greetings," 8 Message2="Earthling." 9 10 echo 11 12 printf "Pi to 2 decimal places = %1.2f" $PI 13 echo 14 printf "Pi to 9 decimal places = %1.9f" $PI 15 # Note correct round off. 16 17 printf "\n" 18 # Prints a line feed, equivalent to 'echo'. 19 20 printf "Constant = \t%d\n" $DecimalConstant 21 # Insert tab (\t) 22 23 printf "%s %s \n" $Message1 $Message2 24 25 echo 26 27 exit 0 |
The standard UNIX archiving utility. Originally a Tape ARchiving program, from whence it derived its name, it has developed into a general purpose package that can handle all manner of archiving with all types of destination devices, ranging from tape drives to regular files to even stdout (see Example 3-4). GNU tar has long since been patched to accept gzip compression options, such as tar czvf archive-name.tar.gz *, which recursively archives and compresses all files (except "dotfiles") in a directory tree.
This specialized archiving copy command is rarely used any more, having been supplanted by tar/gzip. It still has its uses, such as moving a directory tree.
The standard GNU/UNIX compression utility, replacing the inferior and proprietary compress. The corresponding decompression command is gunzip, which is the equivalent of gzip -d.
The filter zcat decompresses a gzipped file to stdout, as possible input to a pipe or redirection. This is, in effect, a cat command that works on compressed files (including files processed with the older compress utility). See Example 3-14.
An alternate compression utility, usually more efficient than gzip, especially on large files. The corresponding decompression command is bunzip2.
Yet another compression utility, a filter that works only on sorted ASCII word lists. It uses the standard invocation syntax for a filter, sq < input-file > output-file. Fast, but not nearly as efficient as gzip. The corresponding uncompression filter is unsq, invoked like sq.
![]() | The output of sq may be piped to gzip for further compression. |
Shell archiving utility. The files in a shell archive are concatenated without compression, and the resultant archive is essentially a shell script, complete with #!/bin/sh header, and containing all the necessary unarchiving commands. Shar archives still show up in Internet newsgroups, but otherwise shar has been pretty well replaced by tar/gzip. The unshar command unpacks shar archives.
Utility for splitting a file into smaller chunks. Usually used for splitting up large files in order to back them up on floppies or preparatory to e-mailing or uploading them.
A utility for identifying file types. The command file file-name will return a file specification for file-name, such as ascii text or data. It references the magic numbers found in /usr/share/magic, /etc/magic, or /usr/lib/magic, depending on the Linux/UNIX distribution.
Example 3-65. stripping comments from C program files
1 #!/bin/bash 2 3 # Strips out the comments (/* comment */) in a C program. 4 5 NOARGS=1 6 WRONG_FILE_TYPE=2 7 8 if [ $# = 0 ] 9 then 10 echo "Usage: `basename $0` C-program-file" >&2 # Error message to stderr. 11 exit $NOARGS 12 fi 13 14 # Test for correct file type. 15 type=`eval file $1 | awk '{ print $2, $3, $4, $5 }'` 16 # "file $1" echoes file type... 17 # then awk removes the first field of this, the filename... 18 # then the result is fed into the variable "type". 19 correct_type="ASCII C program text" 20 21 if [ "$type" != "$correct_type" ] 22 then 23 echo 24 echo "This script works on C program files only." 25 echo 26 exit $WRONG_FILE_TYPE 27 fi 28 29 30 # Rather cryptic sed script: 31 #-------- 32 sed ' 33 /^\/\*/d 34 /.*\/\*/d 35 ' $1 36 #-------- 37 # Easy to understand if you take several hours to learn sed fundamentals. 38 39 40 # Need to add one more line to the sed script to deal with 41 # case where line of code has a comment following it on same line. 42 # This is left as a non-trivial exercise for the reader. 43 44 exit 0 |
This utility encodes binary files into ASCII characters, making them suitable for transmission in the body of an e-mail message or in a newsgroup posting.
This reverses the encoding, decoding uuencoded files back into the original binaries.
Example 3-66. uudecoding encoded files
1 #!/bin/bash 2 3 lines=35 4 # Allow 35 lines for the header (very generous). 5 6 for File in * 7 # Test all the files in the current working directory... 8 do 9 search1=`head -$lines $File | grep begin | wc -w` 10 search2=`tail -$lines $File | grep end | wc -w` 11 # Uuencoded files have a "begin" near the beginning, and an "end" near the end. 12 if [ $search1 -gt 0 ] 13 then 14 if [ $search2 -gt 0 ] 15 then 16 echo "uudecoding - $File -" 17 uudecode $File 18 fi 19 fi 20 done 21 22 exit 0 |
These are utilities for generating checksums. A checksum is a number mathematically calculated from the contents of a file, for the purpose of checking its integrity. A script might refer to a list of checksums for security purposes, such as ensuring that the contents of key system files have not been altered or corrupted.
Use the strings command to find printable strings in a binary or data file. It will list sequences of printable characters found in the target file. This might be handy for a quick 'n dirty examination of a core dump or for looking at an unknown graphic image file (strings image-file | more might show something like JFIF, which would identify the file as a jpeg graphic). In a script, you would probably parse the output of strings with grep or sed.
Pagers that display a text file or stream to stdout, one screenful at a time. These may be used to filter the output of a script.
Searches for information about an Internet host by name or IP address, using DNS.
Verify an Internet e-mail address.
Do an Internet "name server lookup" on a host by IP address. This may be run either interactively or noninteractively, i.e., from within a script.
Similar to nslookup, do an Internet "name server lookup" on a host. May be run either interactively or noninteractively, i.e., from within a script.
Trace the route taken by packets sent to a remote host. This command works within a LAN, WAN, or over the Internet. The remote host may be specified by an IP address. The output of this command may be filtered by grep or sed in a pipe.
"Remote copy", copies files between two different networked machines. Using rcp and similar utilities with security implications in a shell script may not be advisable. Consider instead, using an expect script.
The sx and rx command set serves to transfer files to and from a remote host using the xmodem protocol. These are generally part of a communications package, such as minicom.
The sz and rz command set serves to transfer files to and from a remote host using the zmodem protocol. Zmodem has certain advantages over xmodem, such as greater transmission rate and resumption of interrupted file transfers. Like sx and rx, these are generally part of a communications package.
UNIX to UNIX copy. This is a communications package for transferring files between UNIX servers. A shell script is an effective way to handle a uucp command sequence.
Since the advent of the Internet and e-mail, uucp seems to have faded into obscurity, but it still exists and remains perfectly workable in situations where an Internet connection is not available or appropriate.
These utilities emit a sequence of integers, with a user selected increment. This can be used to advantage in a for loop.
which <command-xxx> gives the full path to "command-xxx". This is useful for finding out whether a particular command or utility is installed on the system.
$bash which pgp
/usr/bin/pgp |
This utility records (saves to a file) all the user keystrokes at the command line in a console or an xterm window. This, in effect, create a record of a session.
The lp and lpr commands send file(s) to the print queue, to be printed as hard copy. [1] These commands trace the origin of their names to the line printers of another era.
bash$ cat file1.txt | lp
It is often useful to pipe the formatted output from pr to lp.
bash$ pr -options file1.txt | lp
Formatting packages, such as groff and Ghostscript may send their output directly to lp.
bash$ groff -Tascii file.tr | lp
bash$ gs -options | lp file.ps
Related commands are lpq, for viewing the print queue, and lprm, for removing jobs from the print queue.
[UNIX borrows an idea here from the plumbing trade.]
This is a redirection operator, but with a difference. Like the plumber's tee, it permits "siponing off" the output of a command or commands within a pipe, but without affecting the result. This is useful for printing an ongoing process to a file or paper, perhaps to keep track of it for debugging purposes.
tee |------> to file | ===============|=============== command--->----|-operator-->---> result of command(s) =============================== |
1 cat listfile* | sort | tee check.file | uniq > result.file |
The clear command simply clears the text screen at the console or in an xterm. The prompt and cursor reappear at the upper lefthand corner of the screen or xterm window. This command may be used either at the command line or in a script. See Example 3-36.
In its default behavior the yes command feeds a continuous string of the character y followed by a line feed to stdout. A control-c terminates the run. A different output string may be specified, as in yes different string, which would continually output different string to stdout. One might well ask the purpose of this. From the command line or in a script, the output of yes can be redirected or piped into a program expecting user input. In effect, this becomes a sort of poor man's version of expect.
This obscure command creates a named pipe, a temporary First-In-First-Out buffer for transferring data between processes. Typically, one process writes to the FIFO, and the other reads from it. See Example A-7.
This command checks the validity of a filename. If the filename exceeds the maximum allowable length (255 characters) or one or more of the directories in its path is not searchable, then an error message results. Unfortunately, pathchk does not return a recognizable error code, and it is therefore pretty much useless in a script.
This is the somewhat obscure and much feared "data duplicator" command. It simply copies a file (or stdin/stdout), but with conversions. Possible conversions are ASCII/EBCDIC, upper/lower case, swapping of byte pairs between input and output, and skipping and/or truncating the head or tail of the input file. A dd --help lists the conversion and other options that this powerful utility takes.
The dd command can copy raw data and disk images to and from devices, such as floppies and tape drives. It can even be used to create boot floppies.
1 dd if=kernel-image of=/dev/fd0H1440 |
[1] | The print queue is the group of jobs "waiting in line" to be printed. |