Prepare distribution patches with gawk

I maintain GNU Awk. As part of making releases, I have to create a patch script to convert the file tree of the previous release into the current one. This means writing rm commands to remove any files that have been removed. This is fairly straightforward using tools like find, sort, and comm.

However, for the 4.1.2 release, I also changed the permissions (mode) on some files. I want to create chmod commands to update these files’ permission settings as well. This is a little harder, so I decided to write an awk script that will do this for me.

Let’s take a look at some of the sophistication and control you can achieve using awk, such as recursion, the use of arrays of arrays, and extension functions for using operating system facilities.

This script, comptrees.awk, uses the fts() extension function to do the heavy lifting. This function walks file trees, building up a representation of those trees using gawk‘s arrays of arrays.

The script then uses an awk function to compare the two trees’ arrays. We start with a #! header and some descriptive comments:

1 ! /usr/local/bin/gawk -f
2
3 comptrees.awk --- compare two file trees and print commands to synchronize them
4 #
5 # Arnold Robbins
6 # arnold@skeeve.com
7 # April, 2015
8

The next statement loads the filefuncs extension, which includes the fts() function:

1 @load "filefuncs"
2

The program is run from a BEGIN rule. The first thing to do is check the number of arguments and print an error message if that count is incorrect:

1 BEGIN {
2     # argument count checking
3     if (ARGC != 3) {
4         print "usage: comptrees dir1 dir2" > "/dev/stderr"
5         exit 1
6     }
7

The next step is to remove the program name from ARGV, leaving just the two file names in the array. This lets us pass ARGV directly to fts().

1 #remove program name
2 delete ARGV[0]
3

The fts() function walks the trees. The first argument is an array whose element values are the paths to walk. The second is one or more flag values ORed together; in this case symbolic links are not followed. The final argument holds the results as an array of arrays.

1 # walk the trees
2 fts(ARGV, FTS_PHYSICAL, results)
3

The fts() function walks the trees. The first argument is an array whose element values are the paths to walk. The second is one or more flag values ORed together; in this case symbolic links are not followed. The final argument holds the results as an array of arrays.

1 # walk the trees
2 fts(ARGV, FTS_PHYSICAL, results)
3

The top level indices in the results array are the final component of the full path. Thus, a simple basename() function strips out the leading path components to get at each subarray. We pass the full names and subarrays into the comparison function, which does the work, and then we’re done:

1     # compare them
2     compare(ARGV[1], results[basename(ARGV[1])],
3         ARGV[2], results[basename(ARGV[2])])
4 }
5

The basename() function returns the final component of the input pathname, using gawk‘s gensub() function to do the work:

1 # basename --- strip out all but the last part of a filename
2
3 function basename(path)
4 {
5     return gensub(".*/", "", "g", path)
6 }
7   

The arrays created by fts() are a little bit complicated. See the filefuncs.3am man page in the gawk distribution and the documentation for the details. Basically, directories are represented by arrays where each file is a subarray. Files are arrays with a few special elements, including one named “stat” which is an array with file information such as owner and permissions. The compare() function has to carefully walk the two arrays representing the trees. The header lists the parameters and the single local variable:

1 # compare --- compare two trees
2
3 function compare(oldname, oldtree, newname, newtree,    i)
4 {
5

The function loops over all the elements in oldtree, skipping any of the special ones:

1     # loop over all elements in the array
2     for (i in oldtree) {
3         # skip special elements filled in by fts()
4         if (i == "." || i == "stat" || i == "type" ||
5             i == "path" || i == "error")
6             continue
7

If an element is itself a directory, compare the directories recursively:

1         if ("." in oldtree[i]) { # directory
2             # recurse
3             compare(oldname "/" i, oldtree[i],
4                 newname "/" i, newtree[i])
5         }

Next thing to check. If the element is not in the new tree, it was removed, so print an rm command:

1         else if (! (i in newtree)) {
2             # removed file
3             printf("rm -fr %s/%s\n", oldname, i)
4         }
5

Finally, if an element is a file and the permissions are different between the old and new trees, print a chmod command. The permission value is ANDed with 0777 to get just the file permissions, since the mode value also contains bits indicating the file type:

1         else if (oldtree[i]["stat"]["type"] == "file") {
2             if (oldtree[i]["stat"]["mode"] != newtree[i]["stat"]["mode"]) {
3                 # file permissions change
4                 printf("chmod %o %s/%s\n",
5                     and(newtree[i]["stat"]["mode"], 0777),
6                     newname, i)
7             }
8         }
9     }
10 }
11

That’s it! 63 lines of awk that will save me a lot of time as I prepare future gawk releases. I think this script nicely demonstrates the power of the fts() extension function and gawk‘s arrays of arrays.