Python for DevOps

Python for DevOps

by Noah Gift, Kennedy Behrman, Alfredo Deza, Grig Gheorghiu
Released December 2019
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781492057642

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Much has changed in technology over the past decade. Data is hot, the cloud is ubiquitous, and many organizations need some form of automation. Throughout these transformations, Python has become one of the most popular languages in the world. This practical resource shows you how to use Python for everyday Linux systems administration tasks with today’s most useful DevOps tools, including Docker, Kubernetes, and Terraform.

Learning how to interact and automate with Linux is essential for millions of professionals. Python makes it much easier. With this book, you’ll learn how to develop software and solve problems using containers, as well as how to monitor, instrument, load-test, and operationalize your software. Looking for effective ways to "get stuff done" in Python? This is your guide.

  • Python foundations, including a brief introduction to the language
  • How to automate text, write command-line tools, and automate the filesystem
  • Linux utilities, package management, build systems, monitoring and instrumentation, and automated testing
  • Cloud computing, infrastructure as code, Kubernetes, and serverless
  • Machine learning operations and data engineering from a DevOps perspective
  • Building, deploying, and operationalizing a machine learning project

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. What Does DevOps Mean to the Authors?
    2. How to Use This Book
      1. Conceptual Topics
    3. Conventions Used in This Book
    4. Using Code Examples
    5. O’Reilly Online Learning
    6. How to Contact Us
    7. Acknowledgments
      1. Noah
      2. Kennedy
      3. Alfredo
      4. Grig
  2. 1. Python Essentials for DevOps
    1. Installing and Running Python
      1. The Python Shell
      2. Jupyter Notebooks
    2. Procedural Programming
      1. Variables
      2. Basic Math
      3. Comments
      4. Built-in Functions
      5. Print
      6. Range
    3. Execution Control
      1. if/elif/else
      2. for Loops
    4. while Loops
    5. Handling Exceptions
    6. Built-in Objects
      1. What Is an Object?
      2. Object Methods and Attributes
      3. Sequences
    7. Functions
      1. Anatomy of a Function
      2. Functions as Objects
      3. Anonymous Functions
    8. Using Regular Expressions
      1. Searching
      2. Character Sets
      3. Character Classes
      4. Groups
      5. Named Groups
      6. Find All
      7. Find Iterator
      8. Substitution
      9. Compiling
    9. Lazy Evaluation
      1. Generators
      2. Generator Comprehensions
    10. More IPython Features
      1. Using IPython to Run Unix Shell Commands
    11. Exercises
  3. 2. Automating Files and the Filesystem
    1. Reading and Writing Files
    2. Using Regular Expressions to Search Text
    3. Dealing with Large Files
    4. Encrypting Text
      1. Hashing with Hashlib
      2. Encryption with Cryptography
    5. The os Module
    6. Managing Files and Directories Using os.path
    7. Walking Directory Trees Using os.walk
    8. Paths as Objects with Pathlib
  4. 3. Working with the Command Line
    1. Working with the Shell
      1. Talking to the Interpreter with the sys Module
      2. Dealing with the Operating System Using the os Module
      3. Spawn Processes with the subprocess Module
    2. Creating Command-Line Tools
      1. Using sys.argv
      2. Using argparse
      3. Using click
      4. fire
      5. Implementing Plug-ins
    3. Case Study: Turbocharging Python with Command-Line Tools
      1. Using the Numba Just-in-Time (JIT) Compiler
      2. Using the GPU with CUDA Python
      3. Running True Multicore Multithreaded Python Using Numba
      4. KMeans Clustering
    4. Exercises
  5. 4. Useful Linux Utilities
    1. Disk Utilities
      1. Measuring Performance
      2. Partitions
      3. Retrieving Specific Device Information
    2. Network Utilities
      1. SSH Tunneling
      2. Benchmarking HTTP with Apache Benchmark (ab)
      3. Load Testing with molotov
    3. CPU Utilities
      1. Viewing Processes with htop
    4. Working with Bash and ZSH
      1. Customizing the Python Shell
      2. Recursive Globbing
      3. Searching and Replacing with Confirmation Prompts
      4. Removing Temporary Python Files
      5. Listing and Filtering Processes
      6. Unix Timestamp
    5. Mixing Python with Bash and ZSH
      1. Random Password Generator
      2. Does My Module Exist?
      3. Changing Directories to a Module’s Path
      4. Converting a CSV File to JSON
    6. Python One-Liners
      1. Debuggers
      2. How Fast Is this Snippet?
    7. strace
    8. Exercises
    9. Case Study Question
  6. 5. Package Management
    1. Why Is Packaging Important?
      1. When Packaging Might Not Be Needed
    2. Packaging Guidelines
      1. Descriptive Versioning
      2. The changelog
    3. Choosing a Strategy
    4. Packaging Solutions
      1. Native Python Packaging
      2. Debian Packaging
      3. RPM Packaging
    5. Management with systemd
      1. Long-Running Processes
      2. Setting It Up
      3. The systemd Unit File
    6. Installing the Unit
      1. Log Handling
    7. Exercises
    8. Case Study Question
  7. 6. Continuous Integration and Continuous Deployment
    1. Real-World Case Study: Converting a Poorly Maintained WordPress Site to Hugo
      1. Setting Up Hugo
      2. Converting WordPress to Hugo Posts
      3. Creating an Algolia Index and Updating It
      4. Orchestrating with a Makefile
      5. Deploying with AWS CodePipeline
    2. Real-World Case Study: Deploying a Python App Engine Application with Google Cloud Build
    3. Real-World Case Study: NFSOPS
  8. 7. Monitoring and Logging
    1. Key Concepts in Building Reliable Systems
    2. Immutable DevOps Principles
      1. Centralized Logging
      2. Case Study: Production Database Kills Hard Drives
      3. Did You Build It or Buy It?
      4. Fault Tolerance
    3. Monitoring
      1. Graphite
      2. StatsD
      3. Prometheus
    4. Instrumentation
      1. Naming Conventions
    5. Logging
      1. Why Is It Hard?
      2. The basicconfig
      3. Deeper Configuration
      4. Common Patterns
    6. The ELK Stack
      1. Logstash
      2. Elasticsearch and Kibana
    7. Exercises
    8. Case Study Question
  9. 8. Pytest for DevOps
    1. Testing Superpowers with pytest
    2. Getting Started with pytest
      1. Testing with pytest
      2. Differences with unittest
    3. pytest Features
      1. conftest.py
      2. The Amazing assert
      3. Parametrization
    4. Fixtures
      1. Getting Started
      2. Built-in Fixtures
    5. Infrastructure Testing
      1. What Is System Validation?
      2. Introduction to Testinfra
      3. Connecting to Remote Nodes
      4. Features and Special Fixtures
    6. Examples
    7. Testing Jupyter Notebooks with pytest
    8. Exercises
    9. Case Study Question
  10. 9. Cloud Computing
    1. Cloud Computing Foundations
    2. Types of Cloud Computing
    3. Types of Cloud Services
      1. Infrastructure as a Service
      2. Metal as a Service
      3. Platform as a Service
      4. Serverless Computing
      5. Software as a Service
    4. Infrastructure as Code
    5. Continuous Delivery
    6. Virtualization and Containers
      1. Hardware Virtualization
      2. Software Defined Networks
      3. Software Defined Storage
      4. Containers
    7. Challenges and Opportunities in Distributed Computing
    8. Python Concurrency, Performance, and Process Management in the Cloud Era
    9. Process Management
      1. Manage Processes with Subprocess
      2. Using Multiprocessing to Solve Problems
      3. Forking Processes with Pool()
      4. Function as a Service and Serverless
      5. High Performance Python with Numba
      6. Using Numba Just in Time Compiler
      7. Using High-Performance Servers
    10. Conclusion
    11. Exercises
    12. Case Study Questions
  11. 10. Infrastructure as Code
    1. A Classification of Infrastructure Automation Tools
    2. Manual Provisioning
    3. Automated Infrastructure Provisioning with Terraform
      1. Provisioning an S3 Bucket
      2. Provisioning an SSL Certificate with AWS ACM
      3. Provisioning an Amazon CloudFront Distribution
      4. Provisioning a Route 53 DNS Record
      5. Copying Static Files to S3
      6. Deleting All AWS Resources Provisioned with Terraform
    4. Automated Infrastructure Provisioning with Pulumi
      1. Creating a New Pulumi Python Project for AWS
      2. Creating Configuration Values for the Staging Stack
      3. Provisioning an ACM SSL Certificate
      4. Provisioning a Route 53 Zone and DNS Records
      5. Provisioning a CloudFront Distribution
      6. Provisioning a Route 53 DNS Record for the Site URL
      7. Creating and Deploying a New Stack
    5. Exercises
  12. 11. Container Technologies: Docker and Docker Compose
    1. What Is a Docker Container?
    2. Creating, Building, Running, and Removing Docker Images and Containers
    3. Publishing Docker Images to a Docker Registry
    4. Running a Docker Container with the Same Image on a Different Host
    5. Running Multiple Docker Containers with Docker Compose
    6. Porting the docker-compose Services to a New Host and Operating System
    7. Exercises
  13. 12. Container Orchestration: Kubernetes
    1. Short Overview of Kubernetes Concepts
    2. Using Kompose to Create Kubernetes Manifests from docker-compose.yaml
    3. Deploying Kubernetes Manifests to a Local Kubernetes Cluster Based on minikube
    4. Launching a GKE Kubernetes Cluster in GCP with Pulumi
    5. Deploying the Flask Example Application to GKE
    6. Installing Prometheus and Grafana Helm Charts
    7. Destroying the GKE Cluster
    8. Exercises
  14. 13. Serverless Technologies
    1. Deploying the Same Python Function to the “Big Three” Cloud Providers
      1. Installing Serverless Framework
      2. Deploying Python Function to AWS Lambda
      3. Deploying Python Function to Google Cloud Functions
      4. Deploying Python Function to Azure
    2. Deploying a Python Function to Self-Hosted FaaS Platforms
      1. Deploying Python Function to OpenFaaS
    3. Provisioning DynamoDB Table, Lambda Functions, and API Gateway Methods Using the AWS CDK
    4. Exercises
  15. 14. MLOps and Machine learning Engineering
    1. What Is Machine Learning?
      1. Supervised Machine Learning
      2. Modeling
    2. Python Machine learning Ecosystem
      1. Deep Learning with PyTorch
    3. Cloud Machine learning Platforms
    4. Machine learning Maturity Model
      1. Machine Learning Key Terminology
      2. Level 1: Framing, Scope Identification, and Problem Definition
      3. Level 2: Continuous Delivery of Data
      4. Level 3: Continuous Delivery of Clean Data
      5. Level 4: Continuous Delivery of Exploratory Data Analysis
      6. Level 5: Continuous Delivery of Traditional ML and AutoML
      7. Level 6: ML Operational Feedback Loop
    5. Sklearn Flask with Kubernetes and Docker
    6. Sklearn Flask with Kubernetes and Docker
      1. EDA
      2. Modeling
      3. Tune Scaled GBM
      4. Fit Model
      5. Evaluate
      6. adhoc_predict
      7. JSON Workflow
      8. Scale Input
      9. adhoc_predict from Pickle
      10. Scale Input
    7. Exercises
    8. Case Study Question
    9. Learning Assessments
  16. 15. Data Engineering
    1. Small Data
      1. Dealing with Small Data Files
    2. Write a File
    3. Read a File
    4. Generator Pipeline to Read and Process Lines
    5. Using YAML
    6. Big Data
    7. Big Data Tools, Components, and Platforms
      1. Data Sources
      2. Filesystems
      3. Data Storage
    8. Real-Time Streaming Ingestion
    9. Case Study: Building a Homegrown Data Pipeline
    10. Serverless Data Engineering
      1. Using AWS Lambda with CloudWatch Events
      2. Using Amazon CloudWatch Logging with AWS Lambda
      3. Using AWS Lambda to Populate Amazon Simple Queue Service
      4. Wiring Up CloudWatch Event Trigger
      5. Creating Event-Driven Lambdas
      6. Reading Amazon SQS Events from AWS Lambda
    11. Conclusion
    12. Exercises
    13. Case Study Question
  17. 16. DevOps War Stories and Interviews
    1. Film Studio Can’t Make Film
    2. Game Studio Can’t Ship Game
    3. Python Scripts Take 60 Seconds to Launch
    4. Putting Out a Fire with a Cache and Intelligent Instrumentation
    5. You’ll Automate Yourself Out of a Job!
    6. DevOps Antipatterns
      1. No Automated Build Server Antipattern
      2. Flying Blind
      3. Difficulties in Coordination as an Ongoing Accomplishment
      4. No Teamwork
    7. Interviews
      1. Glenn Solomon
      2. Andrew Nguyen
      3. Gabriella Roman
      4. Rigoberto Roche
      5. Jonathan LaCour
      6. Ville Tuulos
      7. Joseph Reis
      8. Teijo Holzer
      9. Matt Harrison
      10. Michael Foord
    8. Recommendations
    9. Exercises
    10. Challenges
    11. Capstone Project
  18. Index

Product information

  • Title: Python for DevOps
  • Author(s): Noah Gift, Kennedy Behrman, Alfredo Deza, Grig Gheorghiu
  • Release date: December 2019
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492057642