bash Cookbook, 2nd Edition

bash Cookbook, 2nd Edition

by Carl Albing, JP Vossen
Released October 2017
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781491975336

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

For system administrators, programmers, and end users, shell command or carefully crafted shell script can save you time and effort, or facilitate consistency and repeatability for a variety of common tasks. This cookbook provides more than 300 practical recipes for using bash, the popular Unix shell that enables you to harness and customize the power of any Unix or Linux system.

Ideal for new and experienced users alike—including proficient Windows users and sysadmins—this updated second edition helps you solve a wide range of problems. You’ll learn ways to handle input/output, file manipulation, program execution, administrative tasks, and many other challenges. Each recipe includes one or more scripting examples and a discussion of why the solution works.

You’ll find recipes for problems including:

  • Standard output and input, and executing commands
  • Shell variables, shell logic, and arithmetic
  • Intermediate shell tools and advanced scripting
  • Searching for files with find, locate, and slocate
  • Working with dates and times
  • Creating shell scripts for various end-user tasks
  • Working with tasks that require parsing
  • Writing secure shell scripts
  • Configuring and customizing bash

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. Who Should Read This Book
    2. About This Book
    3. Conventions Used in This Book
    4. Using Code Examples
    5. O’Reilly Safari
    6. We’d Like to Hear from You
    7. Acknowledgments
  2. Beginning bash
    1. 1.1. Why bash?
    2. 1.2. The bash Shell
    3. 1.3. Decoding the Prompt
    4. 1.4. Showing Where You Are
    5. 1.5. Finding and Running Commands
    6. 1.6. Getting Information About Files
    7. 1.7. Showing All Hidden (Dot) Files in the Current Directory
    8. 1.8. Using Shell Quoting
    9. 1.9. Using or Replacing Builtins and External Commands
    10. 1.10. Determining if You Are Running Interactively
    11. 1.11. Setting bash as Your Default Shell
    12. 1.12. Keeping bash Updated
    13. 1.13. Getting bash for Linux
    14. 1.14. Getting bash for xBSD
    15. 1.15. Getting bash for macOS
    16. 1.16. Getting bash for Unix
    17. 1.17. Getting bash for Windows
    18. 1.18. Getting bash Without Getting bash
    19. 1.19. Learning More About bash Documentation
  3. Standard Output
    1. 2.1. Writing Output to the Terminal/Window
    2. 2.2. Writing Output but Preserving Spacing
    3. 2.3. Writing Output with More Formatting Control
    4. 2.4. Writing Output Without the Newline
    5. 2.5. Saving Output from a Command
    6. 2.6. Saving Output to Other Files
    7. 2.7. Saving Output from the ls Command
    8. 2.8. Sending Output and Error Messages to Different Files
    9. 2.9. Sending Output and Error Messages to the Same File
    10. 2.10. Appending Rather than Clobbering Output
    11. 2.11. Using Just the Beginning or End of a File
    12. 2.12. Skipping a Header in a File
    13. 2.13. Throwing Output Away
    14. 2.14. Saving or Grouping Output from Several Commands
    15. 2.15. Connecting Two Programs by Using Output as Input
    16. 2.16. Saving a Copy of Output Even While Using It as Input
    17. 2.17. Connecting Two Programs by Using Output as Arguments
    18. 2.18. Using Multiple Redirects on One Line
    19. 2.19. Saving Output When Redirect Doesn’t Seem to Work
    20. 2.20. Swapping STDERR and STDOUT
    21. 2.21. Keeping Files Safe from Accidental Overwriting
    22. 2.22. Clobbering a File on Purpose
  4. Standard Input
    1. 3.1. Getting Input from a File
    2. 3.2. Keeping Your Data with Your Script
    3. 3.3. Preventing Weird Behavior in a Here-Document
    4. 3.4. Indenting Here-Documents
    5. 3.5. Getting User Input
    6. 3.6. Getting Yes or No Input
    7. 3.7. Selecting from a List of Options
    8. 3.8. Prompting for a Password
  5. Executing Commands
    1. 4.1. Running Any Executable
    2. 4.2. Running Several Commands in Sequence
    3. 4.3. Running Several Commands All at Once
    4. 4.4. Telling Whether a Command Succeeded or Not
    5. 4.5. Running a Command Only if Another Command Succeeded
    6. 4.6. Using Fewer if Statements
    7. 4.7. Running Long Jobs Unattended
    8. 4.8. Displaying Error Messages When Failures Occur
    9. 4.9. Running Commands from a Variable
    10. 4.10. Running All Scripts in a Directory
  6. Basic Scripting: Shell Variables
    1. 5.1. Documenting Your Script
    2. 5.2. Embedding Documentation in Shell Scripts
    3. 5.3. Promoting Script Readability
    4. 5.4. Separating Variable Names from Surrounding Text
    5. 5.5. Exporting Variables
    6. 5.6. Seeing All Variable Values
    7. 5.7. Using Parameters in a Shell Script
    8. 5.8. Looping Over Arguments Passed to a Script
    9. 5.9. Handling Parameters with Spaces
    10. 5.10. Handling Lists of Parameters with Spaces
    11. 5.11. Counting Arguments
    12. 5.12. Consuming Arguments
    13. 5.13. Getting Default Values
    14. 5.14. Setting Default Values
    15. 5.15. Using null as a Valid Default Value
    16. 5.16. Using More than Just a Constant String for Default
    17. 5.17. Giving an Error Message for Unset Parameters
    18. 5.18. Changing Pieces of a String
    19. 5.19. Getting the Absolute Value of a Number
    20. 5.20. Using bash for basename
    21. 5.21. Using bash for dirname
    22. 5.22. Using Alternate Values for Comma Separated Values
    23. 5.23. Using Array Variables
    24. 5.24. Converting Between Upper- and Lowercase
    25. 5.25. Converting to Camel Case
  7. Shell Logic and Arithmetic
    1. 6.1. Doing Arithmetic in Your Shell Script
    2. 6.2. Branching on Conditions
    3. 6.3. Testing for File Characteristics
    4. 6.4. Testing for More than One Thing
    5. 6.5. Testing for String Characteristics
    6. 6.6. Testing for Equality
    7. 6.7. Testing with Pattern Matches
    8. 6.8. Testing with Regular Expressions
    9. 6.9. Changing Behavior with Redirections
    10. 6.10. Looping for a While
    11. 6.11. Looping with a read
    12. 6.12. Looping with a Count
    13. 6.13. Looping with Floating-Point Values
    14. 6.14. Branching Many Ways
    15. 6.15. Parsing Command-Line Arguments
    16. 6.16. Creating Simple Menus
    17. 6.17. Changing the Prompt on Simple Menus
    18. 6.18. Creating a Simple RPN Calculator
    19. 6.19. Creating a Command-Line Calculator
  8. Intermediate Shell Tools I
    1. 7.1. Sifting Through Files for a String
    2. 7.2. Getting Just the Filename from a Search
    3. 7.3. Getting a Simple True/False from a Search
    4. 7.4. Searching for Text While Ignoring Case
    5. 7.5. Doing a Search in a Pipeline
    6. 7.6. Paring Down What the Search Finds
    7. 7.7. Searching with More Complex Patterns
    8. 7.8. Searching for an SSN
    9. 7.9. Grepping Compressed Files
    10. 7.10. Keeping Some Output, Discarding the Rest
    11. 7.11. Keeping Only a Portion of a Line of Output
    12. 7.12. Reversing the Words on Each Line
    13. 7.13. Summing a List of Numbers
    14. 7.14. Counting String Values with awk
    15. 7.15. Counting String Values with bash
    16. 7.16. Showing Data as a Quick and Easy Histogram
    17. 7.17. An Easy Histogram with bash
    18. 7.18. Showing a Paragraph of Text After a Found Phrase
  9. Intermediate Shell Tools II
    1. 8.1. Sorting Your Output
    2. 8.2. Sorting Numbers
    3. 8.3. Sorting IP Addresses
    4. 8.4. Cutting Out Parts of Your Output
    5. 8.5. Removing Duplicate Lines
    6. 8.6. Compressing Files
    7. 8.7. Uncompressing Files
    8. 8.8. Checking a tar Archive for Unique Directories
    9. 8.9. Translating Characters
    10. 8.10. Converting Uppercase to Lowercase
    11. 8.11. Converting DOS Files to Linux Format
    12. 8.12. Removing Smart Quotes
    13. 8.13. Counting Lines, Words, or Characters in a File
    14. 8.14. Rewrapping Paragraphs
    15. 8.15. Doing More with less
  10. Finding Files: find, locate, slocate
    1. 9.1. Finding All Your MP3 Files
    2. 9.2. Handling Filenames Containing Odd Characters
    3. 9.3. Speeding Up Operations on Found Files
    4. 9.4. Finding Files Across Symbolic Links
    5. 9.5. Finding Files Irrespective of Case
    6. 9.6. Finding Files by Date
    7. 9.7. Finding Files by Type
    8. 9.8. Finding Files by Size
    9. 9.9. Finding Files by Content
    10. 9.10. Finding Existing Files and Content Fast
    11. 9.11. Finding a File Using a List of Possible Locations
  11. Additional Features for Scripting
    1. 10.1. “Daemon-izing” Your Script
    2. 10.2. Reusing Code with Includes and Sourcing
    3. 10.3. Using Configuration Files in a Script
    4. 10.4. Defining Functions
    5. 10.5. Using Functions: Parameters and Return Values
    6. 10.6. Trapping Interrupts
    7. 10.7. Redefining Commands with alias
    8. 10.8. Avoiding Aliases and Functions
    9. 10.9. Counting Elapsed Time
    10. 10.10. Writing Wrappers
  12. Working with Dates and Times
    1. 11.1. Formatting Dates for Display
    2. 11.2. Supplying a Default Date
    3. 11.3. Automating Date Ranges
    4. 11.4. Converting Dates and Times to Epoch Seconds
    5. 11.5. Converting Epoch Seconds to Dates and Times
    6. 11.6. Getting Yesterday or Tomorrow with Perl
    7. 11.7. Figuring Out Date and Time Arithmetic
    8. 11.8. Handling Time Zones, Daylight Saving Time, and Leap Years
    9. 11.9. Using date and cron to Run a Script on the Nth Day
    10. 11.10. Logging with Dates
  13. End-User Tasks as Shell Scripts
    1. 12.1. Starting Simple by Printing Dashes
    2. 12.2. Viewing Photos in an Album
    3. 12.3. Loading Your MP3 Player
    4. 12.4. Burning a CD
    5. 12.5. Comparing Two Documents
  14. Parsing and Similar Tasks
    1. 13.1. Parsing Arguments for Your Shell Script
    2. 13.2. Parsing Arguments with Your Own Error Messages
    3. 13.3. Parsing Some HTML
    4. 13.4. Parsing Output into an Array
    5. 13.5. Parsing Output with a Function Call
    6. 13.6. Parsing Text with a read Statement
    7. 13.7. Parsing with read into an Array
    8. 13.8. Reading an Entire File
    9. 13.9. Getting Your Plurals Right
    10. 13.10. Taking It One Character at a Time
    11. 13.11. Cleaning Up an SVN Source Tree
    12. 13.12. Setting Up a Database with MySQL
    13. 13.13. Isolating Specific Fields in Data
    14. 13.14. Updating Specific Fields in Datafiles
    15. 13.15. Trimming Whitespace
    16. 13.16. Compressing Whitespace
    17. 13.17. Processing Fixed-Length Records
    18. 13.18. Processing Files with No Line Breaks
    19. 13.19. Converting a Datafile to CSV
    20. 13.20. Parsing a CSV Datafile
  15. Writing Secure Shell Scripts
    1. 14.1. Avoiding Common Security Problems
    2. 14.2. Avoiding Interpreter Spoofing
    3. 14.3. Setting a Secure $PATH
    4. 14.4. Clearing All Aliases
    5. 14.5. Clearing the Command Hash
    6. 14.6. Preventing Core Dumps
    7. 14.7. Setting a Secure $IFS
    8. 14.8. Setting a Secure umask
    9. 14.9. Finding World-Writable Directories in Your $PATH
    10. 14.10. Adding the Current Directory to the $PATH
    11. 14.11. Using Secure Temporary Files
    12. 14.12. Validating Input
    13. 14.13. Setting Permissions
    14. 14.14. Leaking Passwords into the Process List
    15. 14.15. Writing setuid or setgid Scripts
    16. 14.16. Restricting Guest Users
    17. 14.17. Using chroot Jails
    18. 14.18. Running as a Non-root User
    19. 14.19. Using sudo More Securely
    20. 14.20. Using Passwords in Scripts
    21. 14.21. Using SSH Without a Password
    22. 14.22. Restricting SSH Commands
    23. 14.23. Disconnecting Inactive Sessions
  16. Advanced Scripting
    1. 15.1. Finding bash Portably for #!
    2. 15.2. Setting a POSIX $PATH
    3. 15.3. Developing Portable Shell Scripts
    4. 15.4. Testing Scripts Using Virtual Machines
    5. 15.5. Using for Loops Portably
    6. 15.6. Using echo Portably
    7. 15.7. Splitting Output Only When Necessary
    8. 15.8. Viewing Output in Hex
    9. 15.9. Using bash Net-Redirection
    10. 15.10. Finding My IP Address
    11. 15.11. Getting Input from Another Machine
    12. 15.12. Redirecting Output for the Life of a Script
    13. 15.13. Working Around “Argument list too long” Errors
    14. 15.14. Logging to syslog from Your Script
    15. 15.15. Using logger Correctly
    16. 15.16. Sending Email from Your Script
    17. 15.17. Automating a Process Using Phases
    18. 15.18. Doing Two Things at Once
    19. 15.19. Running an SSH command on multiple hosts
  17. Configuring and Customizing bash
    1. 16.1. bash Startup Options
    2. 16.2. Customizing Your Prompt
    3. 16.3. A Prompt Before Your Program Runs
    4. 16.4. Changing Your $PATH Permanently
    5. 16.5. Changing Your $PATH Temporarily
    6. 16.6. Setting Your $CDPATH
    7. 16.7. When Programs Are Not Found
    8. 16.8. Shortening or Changing Command Names
    9. 16.9. Adjusting Shell Behavior and Environment
    10. 16.10. Adjusting readline Behavior Using .inputrc
    11. 16.11. Keeping a Private Stash of Utilities by Adding ~/bin
    12. 16.12. Using Secondary Prompts: $PS2, $PS3, $PS4
    13. 16.13. Synchronizing Shell History Between Sessions
    14. 16.14. Setting Shell History Options
    15. 16.15. Creating a Better cd Command
    16. 16.16. Creating and Changing Into a New Directory in One Step
    17. 16.17. Getting to the Bottom of Things
    18. 16.18. Adding New Features to bash Using Loadable Builtins
    19. 16.19. Improving Programmable Completion
    20. 16.20. Using Initialization Files Correctly
    21. 16.21. Creating Self-Contained, Portable rc Files
    22. 16.22. Getting Started with a Custom Configuration
  18. Housekeeping and Administrative Tasks
    1. 17.1. Renaming Many Files
    2. 17.2. Using GNU Texinfo and info on Linux
    3. 17.3. Unzipping Many ZIP Files
    4. 17.4. Recovering Disconnected Sessions Using screen
    5. 17.5. Sharing a Single bash Session
    6. 17.6. Logging an Entire Session or Batch Job
    7. 17.7. Clearing the Screen When You Log Out
    8. 17.8. Capturing File Metadata for Recovery
    9. 17.9. Creating an Index of Many Files
    10. 17.10. Using diff and patch
    11. 17.11. Counting Differences in Files
    12. 17.12. Removing or Renaming Files Named with Special Characters
    13. 17.13. Prepending Data to a File
    14. 17.14. Editing a File in Place
    15. 17.15. Using sudo on a Group of Commands
    16. 17.16. Finding Lines That Appear in One File but Not in Another
    17. 17.17. Keeping the Most Recent N Objects
    18. 17.18. Writing to a Circular Log
    19. 17.19. Circular Backups
    20. 17.20. Grepping ps Output Without Also Getting the grep Process Itself
    21. 17.21. Finding Out Whether a Process Is Running
    22. 17.22. Adding a Prefix or Suffix to Output
    23. 17.23. Numbering Lines
    24. 17.24. Writing Sequences
    25. 17.25. Emulating the DOS Pause Command
    26. 17.26. Commifying Numbers
  19. Working Faster by Typing Less
    1. 18.1. Moving Quickly Among Arbitrary Directories
    2. 18.2. Repeating the Last Command
    3. 18.3. Running Almost the Same Command
    4. 18.4. Quick Substitution
    5. 18.5. Reusing Arguments
    6. 18.6. Finishing Names for You
    7. 18.7. Playing It Safe
    8. 18.8. Big Changes, More Lines
  20. Tips and Traps: Common Goofs for Novices
    1. 19.1. Forgetting to Set Execute Permissions
    2. 19.2. Fixing “No such file or directory” Errors
    3. 19.3. Forgetting That the Current Directory Is Not in the $PATH
    4. 19.4. Naming Your Script “test”
    5. 19.5. Expecting to Change Exported Variables
    6. 19.6. Forgetting Quotes Leads to “command not found” on Assignments
    7. 19.7. Forgetting that Pattern Matching Alphabetizes
    8. 19.8. Forgetting that Pipelines Make Subshells
    9. 19.9. Making Your Terminal Sane Again
    10. 19.10. Deleting Files Using an Empty Variable
    11. 19.11. Seeing Odd Behavior from printf
    12. 19.12. Testing bash Script Syntax
    13. 19.13. Debugging Scripts
    14. 19.14. Avoiding “command not found” When Using Functions
    15. 19.15. Confusing Shell Wildcards and Regular Expressions
  21. Reference Lists
    1. bash Invocation
    2. Prompt String Customizations
    3. ANSI Color Escape Sequences
    4. Builtin Commands
    5. bash Reserved Words
    6. Builtin Shell Variables
    7. set Options
    8. shopt Options
    9. Test Operators
    10. I/O Redirection
    11. echo Options and Escape Sequences
    12. printf
    13. Date and Time String Formatting with strftime
    14. Pattern-Matching Characters
    15. extglob Extended Pattern-Matching Operators
    16. tr Escape Sequences
    17. readline Init File Syntax
    18. Table of ASCII Values
  22. Examples Included with bash
    1. bash Documentation and Examples
  23. Command-Line Processing
    1. Command-Line Processing Steps
  24. Revision Control
    1. Git
    2. Bazaar
    3. Mercurial
    4. Subversion
    5. Meld
    6. etckeeper
    7. Other
  25. Building bash from Source
    1. Obtaining bash
    2. Unpacking the Archive
    3. What’s in the Archive
    4. Who Do I Turn To?
  26. Index

Product information

  • Title: bash Cookbook, 2nd Edition
  • Author(s): Carl Albing, JP Vossen
  • Release date: October 2017
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781491975336