The curly-bracket syntax allows for the shell’s string operators. String operators allow you to manipulate values of variables in various useful ways without having to write full-blown programs or resort to external UNIX utilities. You can do a lot with string-handling operators even if you haven’t yet mastered the programming features we’ll see in later chapters.
In particular, string operators let you do the following:
Ensure that variables exist (i.e., are defined and have non-null values)
Set default values for variables
Catch errors that result from variables not being set
Remove portions of variables’ values that match patterns
The basic idea behind the syntax of string operators is that special characters that denote operations are inserted between the variable’s name and the right curly bracket. Any argument that the operator may need is inserted to the operator’s right.
The first group of string-handling operators tests for the existence of variables and allows substitutions of default values under certain conditions. These are listed in Table 4.1. [51]
Table 4-1. Substitution Operators
| Operator | Substitution |
|---|---|
| $ { varname :- word } |
If varname exists and isn’t null, return its value; otherwise return word. |
| Purpose: |
Returning a default value if the variable is undefined. |
| Example: |
${count:-0} evaluates to 0 if count is undefined. |
| $ { varname := word} |
If varname exists and isn’t null, return its value; otherwise set it to word and then return its value. Positional and special parameters cannot be assigned this way. |
| Purpose: |
Setting a variable to a default value if it is undefined. |
| Example: |
$ {count := 0} sets count to 0 if it is undefined. |
| $ { varname :? message } |
If varname exists and isn’t null, return its value; otherwise print varname : followed by message, and abort the current command or script (non-interactive shells only). Omitting message produces the default message parameter null or not set. |
| Purpose: |
Catching errors that result from variables being undefined. |
| Example: |
{count :?” undefined! " } prints “count: undefined!” and exits if count is undefined. |
| $ { varname :+word } |
If varname exists and isn’t null, return word; otherwise return null. |
| Purpose: |
Testing for the existence of a variable. |
| Example: |
$ {count :+ 1} returns 1 (which could mean “true”) if count is defined. |
| $ { varname : offset } | |
| $ { varname : offset:length } |
Performs substring expansion. a It returns the substring of $varname starting at offset and up to length characters. The first character in $varname is position 0. If length is omitted, the substring starts at offset and continues to the end of $varname. If offset is less than 0 then the position is taken from the end of $varname. If varname is @, the length is the number of positional parameters starting at parameter offset. |
| Purpose: |
Returning parts of a string (substrings or slices). |
| Example: |
If count is set to frogfootman, $ {count :4} returns footman. $ {count :4:4} returns foot. |
[52]
The first of these operators is ideal for setting defaults for command-line arguments in case the user omits them. We’ll use this technique in our first programming task.
By far the best approach to this type of script is to use built-in UNIX utilities, combining them with I/O redirectors and pipes. This is the classic “building-block” philosophy of UNIX that is another reason for its great popularity with programmers. The building-block technique lets us write a first version of the script that is only one line long:
sort -nr $1 | head -${2:-10}Here is how this works: the sort program sorts the data in the file whose name is given as the first argument ($1). The -n option tells sort to interpret the first word on each line as a number (instead of as a character string); the -r tells it to reverse the comparisons, so as to sort in descending order.
The output of sort is piped into the head utility, which, when given the argument - N, prints the first N lines of its input on the standard output. The expression -${2:-10} evaluates to a dash (-) followed by the second argument if it is given, or to -10 if it’s not; notice that the variable in this expression is 2, which is the second positional parameter.
Assume the script we want to write is called highest. Then if the user types highest myfile, the line that actually runs is:
sort -nr myfile | head -10
Or if the user types highest myfile 22, the line that runs is:
sort -nr myfile | head -22
Make sure you understand how the :- string operator provides a default value.
This is a perfectly good, runnable script—but it has a few problems. First, its one line is a bit cryptic. While this isn’t much of a problem for such a tiny script, it’s not wise to write long, elaborate scripts in this manner. A few minor changes will make the code more readable.
First, we can add comments to the code; anything between # and the end of a line is a comment. At a minimum, the script should start with a few comment lines that indicate what the script does and what arguments it accepts. Second, we can improve the variable names by assigning the values of the positional parameters to regular variables with mnemonic names. Finally, we can add blank lines to space things out; blank lines, like comments, are ignored. Here is a more readable version:
#
# highest filename [howmany]
#
# Print howmany highest-numbered lines in file filename.
# The input file is assumed to have lines that start with
# numbers. Default for howmany is 10.
#
filename=$1
howmany=${2:-10}
sort -nr $filename | head -$howmanyThe square brackets around howmany in the comments adhere to the convention in UNIX documentation that square brackets denote optional arguments.
The changes we just made improve the code’s readability but not how it runs. What if the user were to invoke the script without any arguments? Remember that positional parameters default to null if they aren’t defined. If there are no arguments, then $1 and $2 are both null. The variable howmany ($2) is set up to default to 10, but there is no default for filename ($1). The result would be that this command runs:
sort -nr | head -10
As it happens, if sort is called without a filename argument, it expects input to come from standard input, e.g., a pipe (|) or a user’s terminal. Since it doesn’t have the pipe, it will expect the terminal. This means that the script will appear to hang! Although you could always hit CTRL-D or CTRL-C to get out of the script, a naive user might not know this.
Therefore we need to make sure that the user supplies at least one argument. There are a few ways of doing this; one of them involves another string operator. We’ll replace the line:
filename=$1
with:
filename=${1:?"filename missing."}This will cause two things to happen if a user invokes the script without any arguments: first the shell will print the somewhat unfortunate message:
highest: 1: filename missing.
to the standard error output. Second, the script will exit without running the remaining code. With a somewhat “kludgy” modification, we can get a slightly better error message.
Consider this code:
filename=$1
filename=${filename:?"missing."}This results in the message:
highest: filename: missing.
(Make sure you understand why.) Of course, there are ways of printing whatever message is desired; we’ll find out how in Chapter 5.
Before we move on, we’ll look more closely at the three remaining operators in Table 4.1 and see how we can incorporate them into our task solution. The := operator does roughly the same thing as :-, except that it has the “side effect” of setting the value of the variable to the given word if the variable doesn’t exist.
Therefore we would like to use := in our script in place of :-, but we can’t; we’d be trying to set the value of a positional parameter, which is not allowed. But if we replaced:
howmany=${2:-10}with just:
howmany=$2
and moved the substitution down to the actual command line (as we did at the start), then we could use the := operator:
sort -nr $filename | head -${howmany:=10}Using := has the added benefit of setting the value of howmany to 10 in case we need it afterwards in later versions of the script.
The operator :+ substitutes a value if the given variable exists and isn’t null. Here is how we can use it in our example: Let’s say we want to give the user the option of adding a header line to the script’s output. If he or she types the option -h, then the output will be preceded by the line:
ALBUMS ARTIST
Assume further that this option ends up in the variable header, i.e., $header is -h if the option is set or null if not. (Later we will see how to do this without disturbing the other positional parameters.)
The following expression yields null if the variable header is null, or ALBUMSARTIST\n if it is non-null:
${header:+"ALBUMSARTIST\n"}This means that we can put the line:
echo -e -n ${header:+"ALBUMSARTIST\n"}right before the command line that does the actual work. The -n option to echo causes it not to print a LINEFEED after printing its arguments. Therefore this echo statement will print nothing—not even a blank line—if header is null; otherwise it will print the header line and a LINEFEED (\n). The -e option makes echo interpret the \n as a LINEFEED rather than literally.
The final operator, substring expansion, returns sections of a string. We can use it to “pick out” parts of a string that are of interest. Assume that our script is able to assign lines of the sorted list, one at a time, to the variable album_line. If we want to print out just the album name and ignore the number of albums, we can use substring expansion:
echo ${album_line:8}This prints everything from character position 8, which is the start of each album name, onwards.
If we just want to print the numbers and not the album names, we can do so by supplying the length of the substring:
echo ${album_line:0:7}Although this example may seem rather useless, it should give you a feel for how to use substrings. When combined with some of the programming features discussed later in the book, substrings can be extremely useful.
We’ll continue refining our solution to Task 4-1 later in this chapter. The next type of string operator is used to match portions of a variable’s string value against patterns. Patterns, as we saw in Chapter 1, are strings that can contain wildcard characters (*, ?, and [] for character sets and ranges).
Table 4.2 lists bash’s pattern-matching operators.
Table 4-2. Pattern-Matching Operators
| Operator | Meaning |
|---|---|
| ${variable # pattern} |
If the pattern matches the beginning of the variable’s value, delete the shortest part that matches and return the rest. |
| ${variable ## pattern} |
If the pattern matches the beginning of the variable’s value, delete the longest part that matches and return the rest. |
| ${variable % pattern} |
If the pattern matches the end of the variable’s value, delete the shortest part that matches and return the rest. |
| ${variable %% pattern} |
If the pattern matches the end of the variable’s value, delete the longest part that matches and return the rest. |
| ${variable / pattern/string} | |
| ${variable // pattern/string} |
The longest match to pattern in variable is replaced by string. In the first form, only the first match is replaced. In the second form, all matches are replaced. If the pattern is begins with a #, it must match at the start of the variable. If it begins with a %, it must match with the end of the variable. If string is null, the matches are deleted. If variable is @ or *, the operation is applied to each positional parameter in turn and the expansion is the resultant list. a |
[53]
These can be hard to remember, so here’s a handy mnemonic device: # matches the front because number signs precede numbers; % matches the rear because percent signs follow numbers.
The classic use for pattern-matching operators is in stripping off components of pathnames, such as directory prefixes and filename suffixes. With that in mind, here is an example that shows how all of the operators work. Assume that the variable path has the value /home/cam/book/long.file.name; then:
Expression Result
${path##/*/} long.file.name
${path#/*/} cam/book/long.file.name
$path /home/cam/book/long.file.name
${path%.*} /home/cam/book/long.file
${path%%.*} /home/cam/book/longThe two patterns used here are /*/, which matches anything between two slashes, and . *, which matches a dot followed by anything.
The longest and shortest pattern-matching operators produce the same output unless they are used with the * wildcard operator. As an example, if filename had the value alicece, then both ${filename%ce} and ${filename%%ce} would produce the result alice. This is because ce is an exact match; for a match to occur, the string ce must appear on the end $filename. Both the short and long matches will then match the last grouping of ce and delete it. If, however, we had used the * wildcard, then ${filename%ce*} would produce alice because it matches the shortest occurrence of ce followed by anything else. ${filename%%ce*} would return ali because it matches the longest occurrence of ce followed by anything else; in this case the first and second ce.
The next task will incorporate one of these pattern-matching operators.
Graphics file conversion utilities are quite common because of the plethora of different graphics formats and file types. They allow you to specify an input file, usually from a range of different formats, and convert it to an output file of a different format. In this case, we want to take a PCX file, which can’t be displayed with a Web browser, and convert it to a GIF which can be displayed by nearly all browsers. Part of this process is taking the filename of the PCX file, which ends in .pcx, and changing it to one ending in .gif for the output file. In essence, you want to take the original filename and strip off the .pcx, then append .gif. A single shell statement will do this:
outfile=${filename%.pcx}.gifThe shell takes the filename and looks for .pcx on the end of the string. If it is found, .pcx is stripped off and the rest of the string is returned. For example, if filename had the value alice.pcx, the expression ${filename%.pcx} would return alice. The .gif is appended to form the desired alice.gif, which is then stored in the variable outfile.
If filename had an inappropriate value (without the .pcx) such as alice.jpg, the above expression would evaluate to alice.jpg.gif: since there was no match, nothing is deleted from the value of filename, and .gif is appended anyway. Note, however, that if filename contained more than one dot (e.g., if it were alice.1.pcx—the expression would still produce the desired value alice.1.gif).
The next task uses the longest pattern-matching operator.
Clearly, the objective is to remove the directory prefix from the pathname. The following line will do it:
bannername=${pathname##*/}This solution is similar to the first line in the examples shown before. If pathname were just a filename, the pattern * / (anything followed by a slash) would not match and the value of the expression would be pathname untouched. If pathname were something like book/wonderland, the prefix book/ would match the pattern and be deleted, leaving just wonderland as the expression’s value. The same thing would happen if pathname were something like /home/cam/ book/wonderland: since the ## deletes the longest match, it deletes the entire /home/cam/book/.
If we used # */ instead of ## */, the expression would have the incorrect value home/cam/book/wonderland, because the shortest instance of “anything followed by a slash” at the beginning of the string is just a slash (/).
The construct ${ variable ## * /} is actually equivalent to the UNIX utility basename. basename takes a pathname as argument and returns the filename only; it is meant to be used with the shell’s command substitution mechanism (see the following explanation). basename is less efficient than ${ variable ##* / } because it runs in its own separate process rather than within the shell. Another utility, dirname, does essentially the opposite of basename: it returns the directory prefix only. It is equivalent to the bash expression ${ variable %/ * } and is less efficient for the same reason.
The last operator in the table matches patterns and performs substitutions. Task 4-4 is a simple task where it comes in useful.
As directory names are separated by colons, the easiest way would be to replace each colon with a LINEFEED:
$ echo -e ${PATH//:/'\n'}
/home/cam/bin
/usr/local/bin
/bin
/usr/bin
/usr/X11R6/binEach occurrence of the colon is replaced by \n. As we saw earlier, the -e option allows echo to interpret \n as a LINEFEED. In this case we used the second of the two substitution forms. If we’d used the first form, only the first colon would have been replaced with a \n.
There is one remaining operator on variables. It is ${# varname }, which returns the length of the value of the variable as a character string. (In Chapter 6, we will see how to treat this and similar values as actual numbers so they can be used in arithmetic expressions.) For example, if filename has the value alice.c, then ${#filename} would have the value 7.
[51] The colon (:) in all but the last of these operators is actually optional. If the colon is omitted, then change “exists and isn’t null” to “exists” in each definition, i.e., the operator tests for existence only.
[52] The substring expansion operator is not available in versions of bash prior to 2.0.
[53] The pattern-matching and replacement operator is not available in versions of bash prior to 2.0.
[54] PCX is a popular graphics file format under Microsoft Windows. GIF (Graphics Interchange Format) is a common graphics format on the Internet and is used to a great extent on Web pages.
Get Learning the bash Shell, Second Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
