Classic Shell Scripting

Classic Shell Scripting

by Arnold Robbins, Nelson H. F. Beebe
Released May 2005
Publisher(s): O'Reilly Media, Inc.
ISBN: 9780596555269

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Shell scripting skills never go out of style. It's the shell that unlocks the real potential of Unix. Shell scripting is essential for Unix users and system administrators-a way to quickly harness and customize the full power of any Unix system. With shell scripts, you can combine the fundamental Unix text and file processing commands to crunch data and automate repetitive tasks. But beneath this simple promise lies a treacherous ocean of variations in Unix commands and standards. Classic Shell Scripting is written to help you reliably navigate these tricky waters.Writing shell scripts requires more than just a knowledge of the shell language, it also requires familiarity with the individual Unix programs: why each one is there, how to use them by themselves, and in combination with the other programs. The authors are intimately familiar with the tips and tricks that can be used to create excellent scripts, as well as the traps that can make your best effort a bad shell script. With Classic Shell Scripting you'll avoid hours of wasted effort. You'll learn not only write useful shell scripts, but how to do it properly and portably.The ability to program and customize the shell quickly, reliably, and portably to get the best out of any individual system is an important skill for anyone operating and maintaining Unix or Linux systems. Classic Shell Scripting gives you everything you need to master these essential skills.

Publisher resources

View/Submit Errata

Table of contents

  1. Classic Shell Scripting
    1. SPECIAL OFFER: Upgrade this ebook with O’Reilly
    2. A Note Regarding Supplemental Files
    3. Foreword
    4. Preface
      1. Intended Audience
      2. What You Should Already Know
      3. Chapter Summary
      4. Conventions Used in This Book
      5. Code Examples
      6. Unix Tools for Windows Systems
        1. Cygwin
        2. DJGPP
        3. MKS Toolkit
        4. AT&T UWIN
      7. Safari Enabled
      8. We'd Like to Hear from You
      9. Acknowledgments
    5. 1. Background
      1. 1.1. Unix History
      2. 1.2. Software Tools Principles
      3. 1.3. Summary
    6. 2. Getting Started
      1. 2.1. Scripting Languages Versus Compiled Languages
      2. 2.2. Why Use a Shell Script?
      3. 2.3. A Simple Script
      4. 2.4. Self-Contained Scripts: The #! First Line
      5. 2.5. Basic Shell Constructs
        1. 2.5.1. Commands and Arguments
        2. 2.5.2. Variables
        3. 2.5.3. Simple Output with echo
        4. 2.5.4. Fancier Output with printf
        5. 2.5.5. Basic I/O Redirection
          1. 2.5.5.1. Redirection and pipelines
          2. 2.5.5.2. Special files: /dev/null and /dev/tty
        6. 2.5.6. Basic Command Searching
      6. 2.6. Accessing Shell Script Arguments
      7. 2.7. Simple Execution Tracing
      8. 2.8. Internationalization and Localization
      9. 2.9. Summary
    7. 3. Searching and Substitutions
      1. 3.1. Searching for Text
        1. 3.1.1. Simple grep
      2. 3.2. Regular Expressions
        1. 3.2.1. What Is a Regular Expression?
          1. 3.2.1.1. POSIX bracket expressions
        2. 3.2.2. Basic Regular Expressions
          1. 3.2.2.1. Matching single characters
          2. 3.2.2.2. Backreferences
          3. 3.2.2.3. Matching multiple characters with one expression
          4. 3.2.2.4. Anchoring text matches
          5. 3.2.2.5. BRE operator precedence
        3. 3.2.3. Extended Regular Expressions
          1. 3.2.3.1. Matching single characters
          2. 3.2.3.2. Backreferences don't exist
          3. 3.2.3.3. Matching multiple regular expressions with one expression
          4. 3.2.3.4. Alternation
          5. 3.2.3.5. Grouping
          6. 3.2.3.6. Anchoring text matches
          7. 3.2.3.7. ERE operator precedence
        4. 3.2.4. Regular Expression Extensions
        5. 3.2.5. Which Programs Use Which Regular Expressions?
        6. 3.2.6. Making Substitutions in Text Files
        7. 3.2.7. Basic Usage
          1. 3.2.7.1. Substitution details
        8. 3.2.8. sed Operation
          1. 3.2.8.1. To print or not to print
        9. 3.2.9. Matching Specific Lines
        10. 3.2.10. How Much Text Gets Changed?
        11. 3.2.11. Lines Versus Strings
      3. 3.3. Working with Fields
        1. 3.3.1. Text File Conventions
        2. 3.3.2. Selecting Fields with cut
        3. 3.3.3. Joining Fields with join
        4. 3.3.4. Rearranging Fields with awk
          1. 3.3.4.1. Patterns and actions
          2. 3.3.4.2. Fields
          3. 3.3.4.3. Setting the field separators
          4. 3.3.4.4. Printing lines
          5. 3.3.4.5. Startup and cleanup actions
      4. 3.4. Summary
    8. 4. Text Processing Tools
      1. 4.1. Sorting Text
        1. 4.1.1. Sorting by Lines
        2. 4.1.2. Sorting by Fields
        3. 4.1.3. Sorting Text Blocks
        4. 4.1.4. Sort Efficiency
        5. 4.1.5. Sort Stability
        6. 4.1.6. Sort Wrap-Up
      2. 4.2. Removing Duplicates
      3. 4.3. Reformatting Paragraphs
      4. 4.4. Counting Lines, Words, and Characters
      5. 4.5. Printing
        1. 4.5.1. Evolution of Printing Technology
        2. 4.5.2. Other Printing Software
      6. 4.6. Extracting the First and Last Lines
      7. 4.7. Summary
    9. 5. Pipelines Can Do Amazing Things
      1. 5.1. Extracting Data from Structured Text Files
      2. 5.2. Structured Data for the Web
      3. 5.3. Cheating at Word Puzzles
      4. 5.4. Word Lists
      5. 5.5. Tag Lists
      6. 5.6. Summary
    10. 6. Variables, Making Decisions, and Repeating Actions
      1. 6.1. Variables and Arithmetic
        1. 6.1.1. Variable Assignment and the Environment
        2. 6.1.2. Parameter Expansion
          1. 6.1.2.1. Expansion operators
          2. 6.1.2.2. Positional parameters
          3. 6.1.2.3. Special variables
        3. 6.1.3. Arithmetic Expansion
      2. 6.2. Exit Statuses
        1. 6.2.1. Exit Status Values
        2. 6.2.2. if-elif-else-fi
        3. 6.2.3. Logical NOT, AND, and OR
        4. 6.2.4. The test Command
      3. 6.3. The case Statement
      4. 6.4. Looping
        1. 6.4.1. for Loops
        2. 6.4.2. while and until Loops
        3. 6.4.3. break and continue
        4. 6.4.4. shift and Option Processing
      5. 6.5. Functions
      6. 6.6. Summary
    11. 7. Input and Output, Files, and Command Evaluation
      1. 7.1. Standard Input, Output, and Error
      2. 7.2. Reading Lines with read
      3. 7.3. More About Redirections
        1. 7.3.1. Additional Redirection Operators
        2. 7.3.2. File Descriptor Manipulation
      4. 7.4. The Full Story on printf
      5. 7.5. Tilde Expansion and Wildcards
        1. 7.5.1. Tilde Expansion
        2. 7.5.2. Wildcarding
          1. 7.5.2.1. Hidden files
      6. 7.6. Command Substitution
        1. 7.6.1. Using sed for the head Command
        2. 7.6.2. Creating a Mailing List
        3. 7.6.3. Simple Math: expr
      7. 7.7. Quoting
      8. 7.8. Evaluation Order and eval
        1. 7.8.1. The eval Statement
        2. 7.8.2. Subshells and Code Blocks
      9. 7.9. Built-in Commands
        1. 7.9.1. The set Command
      10. 7.10. Summary
    12. 8. Production Scripts
      1. 8.1. Path Searching
      2. 8.2. Automating Software Builds
      3. 8.3. Summary
    13. 9. Enough awk to Be Dangerous
      1. 9.1. The awk Command Line
      2. 9.2. The awk Programming Model
      3. 9.3. Program Elements
        1. 9.3.1. Comments and Whitespace
        2. 9.3.2. Strings and String Expressions
        3. 9.3.3. Numbers and Numeric Expressions
        4. 9.3.4. Scalar Variables
        5. 9.3.5. Array Variables
        6. 9.3.6. Command-Line Arguments
        7. 9.3.7. Environment Variables
      4. 9.4. Records and Fields
        1. 9.4.1. Record Separators
        2. 9.4.2. Field Separators
        3. 9.4.3. Fields
      5. 9.5. Patterns and Actions
        1. 9.5.1. Patterns
        2. 9.5.2. Actions
      6. 9.6. One-Line Programs in awk
      7. 9.7. Statements
        1. 9.7.1. Sequential Execution
        2. 9.7.2. Conditional Execution
        3. 9.7.3. Iterative Execution
        4. 9.7.4. Array Membership Testing
        5. 9.7.5. Other Control Flow Statements
        6. 9.7.6. User-Controlled Input
        7. 9.7.7. Output Redirection
        8. 9.7.8. Running External Programs
      8. 9.8. User-Defined Functions
      9. 9.9. String Functions
        1. 9.9.1. Substring Extraction
        2. 9.9.2. Lettercase Conversion
        3. 9.9.3. String Searching
        4. 9.9.4. String Matching
        5. 9.9.5. String Substitution
        6. 9.9.6. String Splitting
        7. 9.9.7. String Reconstruction
        8. 9.9.8. String Formatting
      10. 9.10. Numeric Functions
      11. 9.11. Summary
    14. 10. Working with Files
      1. 10.1. Listing Files
        1. 10.1.1. Long File Listings
        2. 10.1.2. Listing File Metadata
      2. 10.2. Updating Modification Times with touch
      3. 10.3. Creating and Using Temporary Files
        1. 10.3.1. The $$ Variable
        2. 10.3.2. The mktemp Program
        3. 10.3.3. The /dev/random and /dev/urandom Special Files
      4. 10.4. Finding Files
        1. 10.4.1. Finding Files Quickly
        2. 10.4.2. Finding Where Commands Are Stored
        3. 10.4.3. The find Command
          1. 10.4.3.1. Using the find command
          2. 10.4.3.2. A simple find script
          3. 10.4.3.3. A complex find script
        4. 10.4.4. Finding Problem Files
      5. 10.5. Running Commands: xargs
      6. 10.6. Filesystem Space Information
        1. 10.6.1. The df Command
        2. 10.6.2. The du Command
      7. 10.7. Comparing Files
        1. 10.7.1. The cmp and diff Utilities
        2. 10.7.2. The patch Utility
        3. 10.7.3. File Checksum Matching
        4. 10.7.4. Digital Signature Verification
      8. 10.8. Summary
    15. 11. Extended Example: Merging User Databases
      1. 11.1. The Problem
      2. 11.2. The Password Files
      3. 11.3. Merging Password Files
        1. 11.3.1. Separating Users by Manageability
        2. 11.3.2. Managing UIDs
        3. 11.3.3. Creating User-Old UID-New UID Triples
      4. 11.4. Changing File Ownership
      5. 11.5. Other Real-World Issues
      6. 11.6. Summary
    16. 12. Spellchecking
      1. 12.1. The spell Program
      2. 12.2. The Original Unix Spellchecking Prototype
      3. 12.3. Improving ispell and aspell
        1. 12.3.1. Private Spelling Dictionaries
        2. 12.3.2. ispell and aspell
      4. 12.4. A Spellchecker in awk
        1. 12.4.1. Introductory Comments
        2. 12.4.2. Main Body
        3. 12.4.3. initialize( )
        4. 12.4.4. get_dictionaries( )
        5. 12.4.5. scan_options( )
        6. 12.4.6. load_dictionaries( )
        7. 12.4.7. load_suffixes( )
        8. 12.4.8. order_suffixes( )
        9. 12.4.9. spell_check_line( )
        10. 12.4.10. spell_check_word( )
        11. 12.4.11. strip_suffixes( )
        12. 12.4.12. report_exceptions( )
        13. 12.4.13. Retrospective on Our Spellchecker
        14. 12.4.14. Efficiency of awk Programs
      5. 12.5. Summary
    17. 13. Processes
      1. 13.1. Process Creation
      2. 13.2. Process Listing
      3. 13.3. Process Control and Deletion
        1. 13.3.1. Deleting Processes
        2. 13.3.2. Trapping Process Signals
      4. 13.4. Process System-Call Tracing
      5. 13.5. Process Accounting
      6. 13.6. Delayed Scheduling of Processes
        1. 13.6.1. sleep: Delay Awhile
        2. 13.6.2. at: Delay Until Specified Time
        3. 13.6.3. batch: Delay for Resource Control
        4. 13.6.4. crontab: Rerun at Specified Times
      7. 13.7. The /proc Filesystem
      8. 13.8. Summary
    18. 14. Shell Portability Issues and Extensions
      1. 14.1. Gotchas
      2. 14.2. The bash shopt Command
      3. 14.3. Common Extensions
        1. 14.3.1. The select Loop
        2. 14.3.2. Extended Test Facility
        3. 14.3.3. Extended Pattern Matching
        4. 14.3.4. Brace Expansion
        5. 14.3.5. Process Substitution
        6. 14.3.6. Indexed Arrays
        7. 14.3.7. Miscellaneous Extensions
      4. 14.4. Download Information
        1. 14.4.1. bash
        2. 14.4.2. ksh93
      5. 14.5. Other Extended Bourne-Style Shells
      6. 14.6. Shell Versions
      7. 14.7. Shell Initialization and Termination
        1. 14.7.1. Bourne Shell (sh) Startup
        2. 14.7.2. Korn Shell Startup
        3. 14.7.3. Bourne-Again Shell Startup and Termination
        4. 14.7.4. Z-Shell Startup and Termination
      8. 14.8. Summary
    19. 15. Secure Shell Scripts: Getting Started
      1. 15.1. Tips for Secure Shell Scripts
      2. 15.2. Restricted Shell
      3. 15.3. Trojan Horses
      4. 15.4. Setuid Shell Scripts: A Bad Idea
      5. 15.5. ksh93 and Privileged Mode
      6. 15.6. Summary
    20. A. Writing Manual Pages
      1. A.1. Manual Pages for pathfind
      2. A.2. Manual-Page Syntax Checking
      3. A.3. Manual-Page Format Conversion
      4. A.4. Manual-Page Installation
    21. B. Files and Filesystems
      1. B.1. What Is a File?
      2. B.2. How Are Files Named?
      3. B.3. What's in a Unix File?
      4. B.4. The Unix Hierarchical Filesystem
        1. B.4.1. Filesystem Structure
        2. B.4.2. Layered Filesystems
        3. B.4.3. Filesystem Implementation Overview
        4. B.4.4. Devices as Unix Files
      5. B.5. How Big Can Unix Files Be?
      6. B.6. Unix File Attributes
        1. B.6.1. File Ownership and Permissions
          1. B.6.1.1. Ownership
          2. B.6.1.2. Permissions
          3. B.6.1.3. Default permissions
          4. B.6.1.4. Permissions in action
          5. B.6.1.5. Directory permissions
        2. B.6.2. File Timestamps
        3. B.6.3. File Links
        4. B.6.4. File Size and Timestamp Variations
        5. B.6.5. Other File Metadata
      7. B.7. Unix File Ownership and Privacy Issues
      8. B.8. Unix File Extension Conventions
      9. B.9. Summary
    22. C. Important Unix Commands
      1. C.1. Shells and Built-in Commands
      2. C.2. Text Manipulation
      3. C.3. Files
      4. C.4. Processes
      5. C.5. Miscellaneous Programs
    23. 16. Bibliography
      1. 16.1. Unix Programmer's Manuals
      2. 16.2. Programming with the Unix Mindset
      3. 16.3. Awk and Shell
      4. 16.4. Standards
      5. 16.5. Security and Cryptography
      6. 16.6. Unix Internals
      7. 16.7. O'Reilly Books
      8. 16.8. Miscellaneous Books
    24. About the Authors
    25. Colophon
    26. SPECIAL OFFER: Upgrade this ebook with O’Reilly

Product information

  • Title: Classic Shell Scripting
  • Author(s): Arnold Robbins, Nelson H. F. Beebe
  • Release date: May 2005
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9780596555269