Hands-On Software Engineering with Golang

Hands-On Software Engineering with Golang

by Achilleas Anagnostopoulos
Released January 2020
Publisher(s): Packt Publishing
ISBN: 9781838554491

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Explore software engineering methodologies, techniques, and best practices in Go programming to build easy-to-maintain software that can effortlessly scale on demand

Key Features

  • Apply best practices to produce lean, testable, and maintainable Go code to avoid accumulating technical debt
  • Explore Go's built-in support for concurrency and message passing to build high-performance applications
  • Scale your Go programs across machines and manage their life cycle using Kubernetes

Book Description

Over the last few years, Go has become one of the favorite languages for building scalable and distributed systems. Its opinionated design and built-in concurrency features make it easy for engineers to author code that efficiently utilizes all available CPU cores.

This Golang book distills industry best practices for writing lean Go code that is easy to test and maintain, and helps you to explore its practical implementation by creating a multi-tier application called Links 'R' Us from scratch. You'll be guided through all the steps involved in designing, implementing, testing, deploying, and scaling an application. Starting with a monolithic architecture, you'll iteratively transform the project into a service-oriented architecture (SOA) that supports the efficient out-of-core processing of large link graphs. You'll learn about various cutting-edge and advanced software engineering techniques such as building extensible data processing pipelines, designing APIs using gRPC, and running distributed graph processing algorithms at scale. Finally, you'll learn how to compile and package your Go services using Docker and automate their deployment to a Kubernetes cluster.

By the end of this book, you'll know how to think like a professional software developer or engineer and write lean and efficient Go code.

What you will learn

  • Understand different stages of the software development life cycle and the role of a software engineer
  • Create APIs using gRPC and leverage the middleware offered by the gRPC ecosystem
  • Discover various approaches to managing package dependencies for your projects
  • Build an end-to-end project from scratch and explore different strategies for scaling it
  • Develop a graph processing system and extend it to run in a distributed manner
  • Deploy Go services on Kubernetes and monitor their health using Prometheus

Who this book is for

This Golang programming book is for developers and software engineers looking to use Go to design and build scalable distributed systems effectively. Knowledge of Go programming and basic networking principles is required.

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. Hands-On Software Engineering with Golang
  3. Dedication
  4. About Packt
    1. Why subscribe?
  5. Contributors
    1. About the author
    2. About the reviewer
    3. Packt is searching for authors like you
  6. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the example code files
      2. Code in Action
      3. Download the color images
      4. Conventions used
    4. Get in touch
      1. Reviews
  7. Section 1: Software Engineering and the Software Development Life Cycle
  8. A Bird's-Eye View of Software Engineering
    1. What is software engineering?
    2. Types of software engineering roles
      1. The role of the software engineer (SWE)
      2. The role of the software development engineer in test (SDET)
      3. The role of the site reliability engineer (SRE)
      4. The role of the release engineer (RE)
      5. The role of the system architect
    3. A list of software development models that all engineers should know
      1. Waterfall
      2. Iterative enhancement
      3. Spiral
      4. Agile
        1. Lean
          1. Eliminate waste
          2. Create knowledge
          3. Defer commitment
          4. Build in quality
          5. Deliver fast
          6. Respect and empower people
          7. See and optimize the whole
        2. Scrum
          1. Scrum roles
          2. Essential Scrum events
        3. Kanban
      5. DevOps
        1. The CAMS model
        2. The three ways model
    4. Summary
    5. Questions
    6. Further reading
  9. Section 2: Best Practices for Maintainable and Testable Go Code
  10. Best Practices for Writing Clean and Maintainable Go Code
    1. The SOLID principles of object-oriented design
      1. Single responsibility
      2. Open/closed principle
      3. Liskov substitution
      4. Interface segregation
      5. Dependency inversion
      6. Applying the SOLID principles
    2. Organizing code into packages
      1. Naming conventions for Go packages
      2. Circular dependencies
        1. Breaking circular dependencies via implicit interfaces
        2. Sometimes, code repetition is not a bad idea!
    3. Tips and tools for writing lean and easy-to-maintain Go code
      1. Optimizing function implementations for readability
      2. Variable naming conventions
      3. Using Go interfaces effectively
      4. Zero values are your friends
      5. Using tools to analyze and manipulate Go programs
        1. Taking care of formatting and imports (gofmt, goimports)
        2. Refactoring code across packages (gorename, gomvpkg, fix)
        3. Improving code quality metrics with the help of linters
    4. Summary
    5. Questions
    6. Further reading
  11. Dependency Management
    1. What's all the fuss about software versioning?
      1. Semantic versioning
        1. Comparing semantic versions
        2. Applying semantic versioning to Go packages
      2. Managing the source code for multiple package versions
        1. Single repository with versioned folders
        2. Single repository – multiple branches
    2. Vendoring – the good, the bad, and the ugly
      1. Benefits of vendoring dependencies
      2. Is vendoring always a good idea?
    3. Strategies and tools for vendoring dependencies
      1. The dep tool
        1. The Gopkg.toml file
        2. The Gopkg.lock file
      2. Go modules – the way forward
      3. Fork packages
    4. Summary
    5. Questions
    6. Further reading
  12. The Art of Testing
    1. Technical requirements
    2. Unit testing
      1. Mocks, stubs, fakes, and spies – commonalities and differences
        1. Stubs and spies!
        2. Mocks
          1. Introducing gomock
          2. Exploring the details of the project we want to write tests for
          3. Leveraging gomock to write a unit test for our application
        3. Fake objects
      2. Black-box versus white-box testing for Go packages – an example
        1. The services behind the facade
        2. Writing black-box tests
        3. Boosting code coverage via white-box tests
      3. Table-driven tests versus subtests
        1. Table-driven tests
        2. Subtests
        3. The best of both worlds
      4. Using third-party testing frameworks
    3. Integration versus functional testing
      1. Integration tests
      2. Functional tests
      3. Functional tests part deux – testing in production!
    4. Smoke tests
    5. Chaos testing – breaking your systems in fun and interesting ways!
    6. Tips and tricks for writing tests
      1. Using environment variables to set up or skip tests
      2. Speeding up testing for local development
      3. Excluding classes of tests via build flags
      4. This is not the output you are looking for – mocking calls to external binaries
      5. Testing timeouts is easy when you have all the time in the world!
    7. Summary
    8. Questions
    9. Further reading
  13. Section 3: Designing and Building a Multi-Tier System from Scratch
  14. The Links 'R'; Us Project
    1. System overview – what are we going to be building?
    2. Selecting an SDLC model for our project
      1. Iterating faster using an Agile framework
      2. Elephant carpaccio – how to iterate even faster!
    3. Requirements analysis
      1. Functional requirements
        1. User story – link submission
        2. User story – search
        3. User story – crawl link graph
        4. User story – calculate PageRank scores
        5. User story – monitor Links 'R' Us health
      2. Non-functional requirements
        1. Service-level objectives
        2. Security considerations
        3. Being good netizens
    4. System component modeling
      1. The crawler
        1. The link filter
        2. The link fetcher
        3. The content extractor
        4. The link extractor
      2. The content indexer
      3. The link provider
      4. The link graph
      5. The PageRank calculator
      6. The metrics store
      7. The frontend
    5. Monolith or microservices? The ultimate question
    6. Summary
    7. Questions
    8. Further reading
  15. Building a Persistence Layer
    1. Technical requirements
      1. Running tests that require CockroachDB
      2. Running tests that require Elasticsearch
    2. Exploring a taxonomy of database systems
      1. Key-value stores
      2. Relational databases
      3. NoSQL databases
      4. Document databases
    3. Understanding the need for a data layer abstraction
    4. Designing the data layer for the link graph component
      1. Creating an ER diagram for the link graph store
      2. Listing the required set of operations for the data access layer
      3. Defining a Go interface for the link graph
        1. Partitioning links and edges for processing the graph in parallel
        2. Iterating Links and Edges
      4. Verifying graph implementations using a shared test suite
      5. Implementing an in-memory graph store
        1. Upserting links
        2. Upserting edges
        3. Looking up links
        4. Iterating links/edges
        5. Removing stale edges
        6. Setting up a test suite for the graph implementation
      6. Scaling across with a CockroachDB-backed graph implementation
        1. Dealing with DB migrations
        2. An overview of the DB schema for the CockroachDB implementation
        3. Upserting links
        4. Upserting edges
        5. Looking up links
        6. Iterating links/edges
        7. Removing stale edges
        8. Setting up a test suite for the CockroachDB implementation
    5. Designing the data layer for the text indexer component
      1. A model for indexed documents
      2. Listing the set of operations that the text indexer needs to support
        1. Defining the Indexer interface
      3. Verifying indexer implementations using a shared test suite
      4. An in-memory Indexer implementation using bleve
        1. Indexing documents
        2. Looking up documents and updating their PageRank score
        3. Searching the index
        4. Iterating the list of search results
        5. Setting up a test suite for the in-memory indexer
      5. Scaling across an Elasticsearch indexer implementation
        1. Creating a new Elasticsearch indexer instance
        2. Indexing and looking up documents
        3. Performing paginated searches
        4. Updating the PageRank score for a document
        5. Setting up a test suite for the Elasticsearch indexer
    6. Summary
    7. Questions
    8. Further reading
  16. Data-Processing Pipelines
    1. Technical requirements
    2. Building a generic data-processing pipeline in Go
      1. Design goals for the pipeline package
      2. Modeling pipeline payloads
      3. Multistage processing
        1. Stageless pipelines – is that even possible?
      4. Strategies for handling errors
        1. Accumulating and returning all errors
        2. Using a dead-letter queue
        3. Terminating the pipeline's execution if an error occurs
      5. Synchronous versus asynchronous pipelines
        1. Synchronous pipelines
        2. Asynchronous pipelines
      6. Implementing a stage worker for executing payload processors
        1. FIFO
        2. Fixed and dynamic worker pools
        3. 1-to-N broadcasting
      7. Implementing the input source worker
      8. Implementing the output sink worker
      9. Putting it all together – the pipeline API
    3. Building a crawler pipeline for the Links 'R' Us project
      1. Defining the payload for the crawler
      2. Implementing a source and a sink for the crawler
      3. Fetching the contents of graph links 
      4. Extracting outgoing links from retrieved webpages
      5. Extracting the title and text from retrieved web pages
      6. Inserting discovered outgoing links to the graph
      7. Indexing the contents of retrieved web pages
      8. Assembling and running the pipeline
    4. Summary
    5. Questions
    6. Further reading
  17. Graph-Based Data Processing
    1. Technical requirements
    2. Exploring the Bulk Synchronous Parallel model
    3. Building a graph processing system in Go
      1. Queueing and delivering messages
        1. The Message interface
        2. Queues and message iterators
        3. Implementing an in-memory, thread-safe queue
      2. Modeling the vertices and edges of graphs
        1. Defining the Vertex and Edge types
        2. Inserting vertices and edges into the graph
      3. Sharing global graph state through data aggregation
        1. Defining the Aggregator interface
        2. Registering and looking up aggregators
        3. Implementing a lock-free accumulator for float64 values
      4. Sending and receiving messages
      5. Implementing graph-based algorithms using compute functions
        1. Achieving vertical scaling by executing compute functions in parallel
        2. Orchestrating the execution of super-steps
      6. Creating and managing Graph instances
    4. Solving interesting graph problems
      1. Searching graphs for the shortest path
        1. The sequential Dijkstra algorithm
        2. Leveraging a gossip protocol to run Dijkstra in parallel
      2. Graph coloring
        1. A sequential greedy algorithm for coloring undirected graphs
        2. Exploiting parallelism for undirected graph coloring
      3. Calculating PageRank scores
        1. The model of the random surfer
        2. An iterative approach to PageRank score calculation
        3. Reaching convergence – when should we stop iterating?
        4. Web graphs in the real world – dealing with dead ends
        5. Defining an API for the PageRank calculator
        6. Implementing a compute function to calculate PageRank scores
    5. Summary
    6. Further reading
  18. Communicating with the Outside World
    1. Technical requirements
    2. Designing robust, secure, and backward-compatible REST APIs
      1. Using human-readable paths for RESTful resources
      2. Controlling access to API endpoints
        1. Basic HTTP authentication
        2. Securing TLS connections from eavesdropping
        3. Authenticating to external service providers using OAuth2
      3. Dealing with API versions
        1. Including the API version as a route prefix
        2. Negotiating API versions via HTTP Accept headers
      4. Building RESTful APIs in Go
    3. Building RPC-based APIs with the help of gRPC
      1. Comparing gRPC to REST
      2. Defining messages using protocol buffers
        1. Defining messages
        2. Versioning message definitions
        3. Representing collections
        4. Modeling field unions
        5. The Any type
      3. Implementing RPC services
        1. Unary RPCs
        2. Server-streaming RPCs
        3. Client-streaming RPCs
        4. Bi-directional streaming RPCs
      4. Security considerations for gRPC APIs
    4. Decoupling Links 'R' Us components from the underlying data stores
      1. Defining RPCs for accessing a remote link-graph instance
      2. Defining RPCs for accessing a text-indexer instance
      3. Creating high-level clients for accessing data stores over gRPC 
    5. Summary
    6. Questions
    7. Further reading
  19. Building, Packaging, and Deploying Software
    1. Technical requirements
    2. Building and packaging Go services using Docker
      1. Benefits of containerization
      2. Best practices for dockerizing Go applications
      3. Selecting a suitable base container for your application
    3. A gentle introduction to Kubernetes
      1. Peeking under the hood
      2. Summarizing the most common Kubernetes resource types
      3. Running a Kubernetes cluster on your laptop!
    4. Building and deploying a monolithic version of Links 'R' Us
      1. Distributing computation across application instances
        1. Carving the UUID space into non-overlapping partitions
        2. Assigning a partition range to each pod
      2. Building wrappers for the application services
      3. The crawler service
      4. The PageRank calculator service
      5. Serving a fully functioning frontend to users
        1. Specifying the endpoints for the frontend application
        2. Performing searches and paginating results
        3. Generating convincing summaries for search results
        4. Highlighting search keywords
      6. Orchestrating the execution of individual services
      7. Putting everything together
        1. Terminating the application in a clean way
        2. Dockerizing and starting a single instance of the monolith
      8. Deploying and scaling the monolith on Kubernetes
        1. Setting up the required namespaces
        2. Deploying CockroachDB and Elasticsearch using Helm
        3. Deploying Links 'R' Us
    5. Summary
    6. Questions
    7. Further reading
  20. Section 4: Scaling Out to Handle a Growing Number of Users
  21. Splitting Monoliths into Microservices
    1. Technical requirements
    2. Monoliths versus service-oriented architectures
      1. Is there something inherently wrong with monoliths?
      2. Microservice anti-patterns and how to deal with them
    3. Monitoring the state of your microservices
      1. Tracing requests through distributed systems
        1. The OpenTracing project
        2. Stepping through a distributed tracing example
          1. The provider service
          2. The aggregator service
          3. The gateway
          4. Putting it all together
        3. Capturing and visualizing traces using Jaeger
      2. Making logging your trusted ally
        1. Logging best practices
        2. The devil is in the (logging) details
        3. Shipping and indexing logs inside Kubernetes
          1. Running a log collector on each Kubernetes node
          2. Using a sidecar container to collect logs
          3. Shipping logs directly from the application
      3. Introspecting live Go services
    4. Building a microservice-based version of Links 'R' Us
      1. Decoupling access to the data stores
      2. Breaking down the monolith into distinct services
      3. Deploying the microservices that comprise the Links 'R' Us project
        1. Deploying the link-graph and text-indexer API services
        2. Deploying the web crawler
        3. Deploying the PageRank service
        4. Deploying the frontend service
      4. Locking down access to our Kubernetes cluster using network policies
    5. Summary
    6. Questions
    7. Further reading
  22. Building Distributed Graph-Processing Systems
    1. Technical requirements
    2. Introducing the master/worker model
      1. Ensuring that masters are highly available
        1. The leader-follower configuration
        2. The multi-master configuration
      2. Strategies for discovering nodes
      3. Recovering from errors
    3. Out-of-core distributed graph processing
      1. Describing the system architecture, requirements, and limitations
      2. Modeling a state machine for executing graph computations
      3. Establishing a communication protocol between workers and masters
        1. Defining a job queue RPC service
        2. Establishing protocol buffer definitions for worker payloads
        3. Establishing protocol buffer definitions for master payloads
      4. Defining abstractions for working with bi-directional gRPC streams
        1. Remote worker stream
        2. Remote master stream
      5. Creating a distributed barrier for the graph execution steps
        1. Implementing a step barrier for individual workers
        2. Implementing a step barrier for the master
      6. Creating custom executor factories for wrapping existing graph instances
        1. The workers' executor factory
        2. The master's executor factory
      7. Coordinating the execution of a graph job
        1. Simplifying end user interactions with the dbspgraph package
        2. The worker job coordinator
          1. Running a new job
          2. Transitioning through the stages of the graph's state machine
          3. Handling incoming payloads from the master
          4. Using the master as an outgoing message relay
        3. The master job coordinator
          1. Running a new job
          2. Transitioning through the stages for the graph's state machine
          3. Handling incoming worker payloads
          4. Relaying messages between workers
      8. Defining package-level APIs for working with master and worker nodes
        1. Instantiating and operating worker nodes
        2. Instantiating and operating master nodes
          1. Handling incoming gRPC connections
          2. Running a new job
    4. Deploying a distributed version of the Links 'R' Us PageRank calculator
      1. Retrofitting master and worker capabilities to the PageRank calculator service
        1. Serializing PageRank messages and aggregator values
        2. Defining job runners for the master and the worker
          1. Implementing the job runner for master nodes
          2. The worker job runner
        3. Deploying the final Links 'R' Us version to Kubernetes
    5. Summary
    6. Questions
    7. Further reading
  23. Metrics Collection and Visualization
    1. Technical requirements
    2. Monitoring from the perspective of a site reliability engineer
      1. Service-level indicators (SLIs)
      2. Service-level objectives (SLOs)
      3. Service-level agreements (SLAs)
    3. Exploring options for collecting and aggregating metrics
      1. Comparing push versus pull systems
      2. Capturing metrics using Prometheus
        1. Supported metric types
        2. Automating the detection of scrape targets
          1. Static and file-based scrape target configuration
          2. Querying the underlying cloud provider
          3. Leveraging the API exposed by Kubernetes
        3. Instrumenting Go code
          1. Registering metrics with Prometheus
          2. Vector-based metrics
          3. Exporting metrics for scraping
    4. Visualizing collected metrics using Grafana
    5. Using Prometheus as an end-to-end solution for alerting
      1. Using Prometheus as a source for alert events
      2. Handling alert events
        1. Grouping alerts together
        2. Selectively muting alerts
        3. Configuring alert receivers
        4. Routing alerts to receivers
    6. Summary
    7. Questions
    8. Further reading
  24. Epilogue
  25. Assessments
    1. Chapter 1
    2. Chapter 2
    3. Chapter 3
    4. Chapter 4
    5. Chapter 5
    6. Chapter 6
    7. Chapter 7
    8. Chapter 8
    9. Chapter 9
    10. Chapter 10
    11. Chapter 11
    12. Chapter 12
    13. Chapter 13
  26. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Hands-On Software Engineering with Golang
  • Author(s): Achilleas Anagnostopoulos
  • Release date: January 2020
  • Publisher(s): Packt Publishing
  • ISBN: 9781838554491