OpenStack Operations Guide

OpenStack Operations Guide

by Tom Fifield, Diane Fleming, Anne Gentle, Lorin Hochstein, Jonathan Proulx, Everett Toews, Joe Topjian
Released April 2014
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781491906293

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Design, deploy, and maintain your own private or public Infrastructure as a Service (IaaS), using the open source OpenStack platform. In this practical guide, experienced developers and OpenStack contributors show you how to build clouds based on reference architectures, as well as how to perform daily administration tasks.

Designed for horizontal scalability, OpenStack lets you build a cloud by integrating several technologies. This approach provides flexibility, but knowing which options to use can be bewildering. Once you complete this book, you’ll know the right questions to ask while you organize compute, storage, and networking resources. If you already know how to manage multiple Ubuntu machines and maintain MySQL, you’re ready to:

  • Set up automated deployment and configuration
  • Design a single-node cloud controller
  • Use metrics to improve scalability
  • Explore compute nodes, network design, and storage
  • Install OpenStack packages
  • Use an example architecture to help simplify decision-making
  • Build a working environment to explore an IaaS cloud
  • Manage users, projects, and quotas
  • Tackle maintenance, debugging, and network troubleshooting
  • Monitor, log, backup, and restore

Table of contents

  1. Preface
    1. Introduction to OpenStack
    2. Getting Started with OpenStack
      1. Using OpenStack
      2. Plug and Play OpenStack
      3. Roll Your Own OpenStack
    3. Who This Book Is For
      1. Further Reading
    4. How This Book Is Organized
    5. Why and How We Wrote This Book
    6. How to Contribute to This Book
    7. Conventions Used in This Book
    8. Using Code Examples
    9. Safari® Books Online
    10. How to Contact Us
    11. Acknowledgments
  2. I. Architecture
    1. 1. Example Architectures
      1. Example Architecture—Legacy Networking (nova)
        1. Overview
        2. Detailed Description
        3. Optional Extensions
      2. Example Architecture—OpenStack Networking
        1. Overview
        2. Detailed Description
        3. Example Component Configuration
      3. Parting Thoughts on Architectures
    2. 2. Provisioning and Deployment
      1. Automated Deployment
        1. Disk Partitioning and RAID
        2. Network Configuration
      2. Automated Configuration
      3. Remote Management
      4. Parting Thoughts for Provisioning and Deploying OpenStack
      5. Conclusion
    3. 3. Designing for Cloud Controllers and Cloud Management
      1. Hardware Considerations
      2. Separation of Services
      3. Database
      4. Message Queue
      5. Conductor Services
      6. Application Programming Interface (API)
      7. Extensions
      8. Scheduling
      9. Images
      10. Dashboard
      11. Authentication and Authorization
      12. Network Considerations
    4. 4. Compute Nodes
      1. Choosing a CPU
      2. Choosing a Hypervisor
      3. Instance Storage Solutions
        1. Off Compute Node Storage—Shared File System
        2. On Compute Node Storage—Shared File System
        3. On Compute Node Storage—Nonshared File System
        4. Issues with Live Migration
        5. Choice of File System
      4. Overcommitting
      5. Logging
      6. Networking
      7. Conclusion
    5. 5. Scaling
      1. The Starting Point
      2. Adding Cloud Controller Nodes
      3. Segregating Your Cloud
        1. Cells and Regions
        2. Availability Zones and Host Aggregates
      4. Scalable Hardware
        1. Hardware Procurement
        2. Capacity Planning
        3. Burn-in Testing
    6. 6. Storage Decisions
      1. Ephemeral Storage
      2. Persistent Storage
        1. Object Storage
        2. Block Storage
      3. OpenStack Storage Concepts
      4. Choosing Storage Backends
        1. Commodity Storage Backend Technologies
      5. Conclusion
    7. 7. Network Design
      1. Management Network
      2. Public Addressing Options
      3. IP Address Planning
      4. Network Topology
        1. VLAN Configuration Within OpenStack VMs
        2. Multi-NIC Provisioning
        3. Multi-Host and Single-Host Networking
      5. Services for Networking
        1. NTP
        2. DNS
      6. Conclusion
  3. II. Operations
    1. 8. Lay of the Land
      1. Using the OpenStack Dashboard for Administration
      2. Command-Line Tools
        1. Installing the Tools
        2. Administrative Command-Line Tools
        3. Getting Credentials
        4. Inspecting API Calls
        5. Servers and Services
        6. Diagnose Your Compute Nodes
      3. Network Inspection
      4. Users and Projects
      5. Running Instances
      6. Summary
    2. 9. Managing Projects and Users
      1. Projects or Tenants?
      2. Managing Projects
        1. Adding Projects
      3. Quotas
        1. Set Image Quotas
        2. Set Compute Service Quotas
        3. Set Object Storage Quotas
        4. Set Block Storage Quotas
      4. User Management
      5. Creating New Users
      6. Associating Users with Projects
        1. Customizing Authorization
        2. Users Who Disrupt Other Users
      7. Summary
    3. 10. User-Facing Operations
      1. Images
        1. Adding Images
        2. Sharing Images Between Projects
        3. Deleting Images
        4. Other CLI Options
        5. The Image Service and the Database
        6. Example Image Service Database Queries
      2. Flavors
        1. Private Flavors
        2. How Do I Modify an Existing Flavor?
      3. Security Groups
        1. General Security Groups Configuration
        2. End-User Configuration of Security Groups
      4. Block Storage
        1. Block Storage Creation Failures
      5. Instances
        1. Starting Instances
        2. Instance Boot Failures
        3. Using Instance-Specific Data
      6. Associating Security Groups
      7. Floating IPs
      8. Attaching Block Storage
      9. Taking Snapshots
        1. Live Snapshots
      10. Instances in the Database
      11. Good Luck!
    4. 11. Maintenance, Failures, and Debugging
      1. Cloud Controller and Storage Proxy Failures and Maintenance
        1. Planned Maintenance
        2. Rebooting a Cloud Controller or Storage Proxy
        3. After a Cloud Controller or Storage Proxy Reboots
        4. Total Cloud Controller Failure
      2. Compute Node Failures and Maintenance
        1. Planned Maintenance
        2. After a Compute Node Reboots
        3. Instances
        4. Inspecting and Recovering Data from Failed Instances
        5. Volumes
        6. Total Compute Node Failure
        7. /var/lib/nova/instances
      3. Storage Node Failures and Maintenance
        1. Rebooting a Storage Node
        2. Shutting Down a Storage Node
        3. Replacing a Swift Disk
      4. Handling a Complete Failure
      5. Configuration Management
      6. Working with Hardware
        1. Adding a Compute Node
        2. Adding an Object Storage Node
        3. Replacing Components
      7. Databases
        1. Database Connectivity
        2. Performance and Optimizing
      8. HDWMY
        1. Hourly
        2. Daily
        3. Weekly
        4. Monthly
        5. Quarterly
        6. Semiannually
      9. Determining Which Component Is Broken
        1. Tailing Logs
        2. Running Daemons on the CLI
      10. Uninstalling
    5. 12. Network Troubleshooting
      1. Using “ip a” to Check Interface States
      2. Visualizing nova-network Traffic in the Cloud
      3. Visualizing OpenStack Networking Service Traffic in the Cloud
      4. Finding a Failure in the Path
      5. tcpdump
      6. iptables
      7. Network Configuration in the Database for nova-network
        1. Manually Deassociating a Floating IP
      8. Debugging DHCP Issues with nova-network
      9. Debugging DNS Issues
      10. Troubleshooting Open vSwitch
      11. Dealing with Network Namespaces
      12. Summary
    6. 13. Logging and Monitoring
      1. Where Are the Logs?
      2. Reading the Logs
      3. Tracing Instance Requests
      4. Adding Custom Logging Statements
      5. RabbitMQ Web Management Interface or rabbitmqctl
      6. Centrally Managing Logs
        1. rsyslog Client Configuration
        2. rsyslog Server Configuration
      7. StackTach
      8. Monitoring
        1. Process Monitoring
        2. Resource Alerting
        3. Metering and Telemetry with Ceilometer
        4. OpenStack-Specific Resources
        5. Intelligent Alerting
        6. Trending
      9. Summary
    7. 14. Backup and Recovery
      1. What to Back Up
      2. Database Backups
      3. File System Backups
        1. Compute
        2. Image Catalog and Delivery
        3. Identity
        4. Block Storage
        5. Object Storage
      4. Recovering Backups
      5. Summary
    8. 15. Customization
      1. Create an OpenStack Development Environment
      2. Customizing Object Storage (Swift) Middleware
      3. Customizing the OpenStack Compute (nova) Scheduler
      4. Customizing the Dashboard (Horizon)
      5. Conclusion
    9. 16. Upstream OpenStack
      1. Getting Help
      2. Reporting Bugs
        1. Confirming and Prioritizing
        2. Bug Fixing
        3. After the Change Is Accepted
      3. Join the OpenStack Community
      4. How to Contribute to the Documentation
      5. Security Information
      6. Finding Additional Information
    10. 17. Advanced Configuration
      1. Differences Between Various Drivers
      2. Implementing Periodic Tasks
      3. Specific Configuration Topics
        1. Security Configuration for Compute, Networking, and Storage
        2. High Availability
        3. Enabling IPv6 Support
        4. Periodic Task Frequency for Compute
        5. Geographical Considerations for Object Storage
    11. 18. Upgrades
      1. Pre-Upgrade Testing Environment
      2. Preparing for a Rollback
      3. Upgrades
      4. How to Perform an Upgrade from Grizzly to Havana—Ubuntu
        1. Impact on Users
        2. Upgrade Considerations
        3. Perform a Backup
        4. Manage Repositories
        5. Update Configuration Files
        6. Upgrade Packages on the Controller Node
        7. Stop Services, Update Database Schemas, and Restart Services on the Controller Node
        8. Upgrade Packages and Restart Services on the Compute Nodes
        9. Upgrade Packages and Restart Services on the Block Storage Nodes
      5. How to Perform an Upgrade from Grizzly to Havana—Red Hat Enterprise Linux and Derivatives
        1. Impact on Users
        2. Upgrade Considerations
        3. Perform a Backup
        4. Manage Repositories
        5. Update Configuration Files
        6. Upgrade Packages on the Controller Node
        7. Stop Services, Update Database Schemas, and Restart Services on the Controller Node
        8. Upgrade Packages and Restart Services on the Compute Nodes
        9. Upgrade Packages and Restart Services on the Block Storage Nodes
      6. Cleaning Up and Final Configuration File Updates
      7. Rolling Back a Failed Upgrade
  4. A. Use Cases
    1. NeCTAR
      1. Deployment
      2. Resources
    2. MIT CSAIL
      1. Deployment
      2. Resources
    3. DAIR
      1. Deployment
      2. Resources
    4. CERN
      1. Deployment
      2. Resources
  5. B. Tales From the Cryp^H^H^H^H Cloud
    1. Double VLAN
    2. The Issue
    3. Disappearing Images
    4. The Valentine’s Day Compute Node Massacre
    5. Down the Rabbit Hole
    6. Havana Haunted by the Dead
  6. C. Working with Roadmaps
    1. Information Available to You
    2. Influencing the Roadmap
    3. Aspects to Watch
      1. Driver Quality Improvements
      2. Easier Upgrades
      3. Deprecation of Nova Network
    4. Replacement of Open vSwitch Plug-in with Modular Layer 2
    5. Compute V3 API
    6. OpenStack on OpenStack (TripleO)
    7. Data Processing (Sahara)
    8. Bare-Metal Deployment (Ironic)
    9. Database as a Service (Trove)
    10. Messaging as a Service (Marconi)
    11. Scheduler Improvements
      1. Block Storage Improvements
      2. Toward a Python SDK
  7. D. Icehouse Preview
    1. Block Storage (cinder)
    2. Common (oslo)
    3. Compute (nova)
    4. Database Service (trove)
    5. Identity (keystone)
    6. Image Service (glance)
    7. Networking (neutron)
    8. Object Storage (swift)
    9. OpenStack dashboard (horizon)
    10. Orchestration (heat)
    11. Telemetry (ceilometer)
  8. E. Resources
    1. OpenStack
    2. Cloud (General)
    3. Python
    4. Networking
    5. Systems Administration
    6. Virtualization
    7. Configuration Management
  9. Glossary
  10. Index

Product information

  • Title: OpenStack Operations Guide
  • Author(s): Tom Fifield, Diane Fleming, Anne Gentle, Lorin Hochstein, Jonathan Proulx, Everett Toews, Joe Topjian
  • Release date: April 2014
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781491906293