Mastering Embedded Linux Programming - Third Edition

Mastering Embedded Linux Programming - Third Edition

by Frank Vasquez, Chris Simmonds
Released May 2021
Publisher(s): Packt Publishing
ISBN: 9781789530384

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Harness the power of Linux to create versatile and robust embedded solutions

Key Features

  • Learn how to develop and configure robust embedded Linux devices
  • Explore the new features of Linux 5.4 and the Yocto Project 3.1 (Dunfell)
  • Discover different ways to debug and profile your code in both user space and the Linux kernel

Book Description

If you’re looking for a book that will demystify embedded Linux, then you’ve come to the right place. Mastering Embedded Linux Programming is a fully comprehensive guide that can serve both as means to learn new things or as a handy reference.

The first few chapters of this book will break down the fundamental elements that underpin all embedded Linux projects: the toolchain, the bootloader, the kernel, and the root filesystem. After that, you will learn how to create each of these elements from scratch and automate the process using Buildroot and the Yocto Project. As you progress, the book will show you how to implement an effective storage strategy for flash memory chips and install updates to a device remotely once it’s deployed.

You’ll also learn about the key aspects of writing code for embedded Linux, such as how to access hardware from apps, the implications of writing multi-threaded code, and techniques to manage memory in an efficient way. The final chapters demonstrate how to debug your code, whether it resides in apps or in the Linux kernel itself.

You’ll also cover the different tracers and profilers that are available for Linux so that you can quickly pinpoint any performance bottlenecks in your system.

By the end of this Linux book, you’ll be able to create efficient and secure embedded devices using Linux.

What you will learn

  • Use Buildroot and the Yocto Project to create embedded Linux systems
  • Troubleshoot BitBake build failures and streamline your Yocto development workflow
  • Update IoT devices securely in the field using Mender or balena
  • Prototype peripheral additions by reading schematics, modifying device trees, soldering breakout boards, and probing pins with a logic analyzer
  • Interact with hardware without having to write kernel device drivers
  • Divide your system up into services supervised by BusyBox runit
  • Debug devices remotely using GDB and measure the performance of systems using tools such as perf, ftrace, eBPF, and Callgrind

Who this book is for

If you’re a systems software engineer or system administrator who wants to learn how to implement Linux on embedded devices, then this book is for you. It's also aimed at embedded systems engineers accustomed to programming for low-power microcontrollers, who can use this book to help make the leap to high-speed systems on chips that can run Linux. Anyone who develops hardware that needs to run Linux will find something useful in this book – but before you get started, you'll need a solid grasp on POSIX standard, C programming, and shell scripting.

Table of contents

  1. Mastering Embedded Linux Programming Third Edition
  2. Contributors
  3. About the authors
  4. About the reviewers
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Download the color images
    6. Conventions used
    7. Get in touch
    8. Reviews
  6. Section 1: Elements of Embedded Linux
  7. Chapter 1: Starting Out
    1. Choosing Linux
    2. When not to choose Linux
    3. Meeting the players
    4. Moving through the project life cycle
      1. The four elements of embedded Linux
    5. Navigating open source
      1. Licenses
    6. Selecting hardware for embedded Linux
    7. Obtaining the hardware for this book
      1. The Raspberry Pi 4
      2. The BeagleBone Black
      3. QEMU
    8. Provisioning your development environment
    9. Summary
  8. Chapter 2: Learning about Toolchains
    1. Technical requirements
    2. Introducing toolchains
      1. Types of toolchains
      2. CPU architectures
      3. Choosing the C library
    3. Finding a toolchain
    4. Building a toolchain using crosstool-NG
      1. Installing crosstool-NG
      2. Building a toolchain for BeagleBone Black
      3. Building a toolchain for QEMU
    5. Anatomy of a toolchain
      1. Finding out about your cross compiler
      2. The sysroot, library, and header files
      3. Other tools in the toolchain
      4. Looking at the components of the C library
    6. Linking with libraries – static and dynamic linking
      1. Static libraries
      2. Shared libraries
    7. The art of cross-compiling
      1. Simple makefiles
      2. Autotools
      3. Package configuration
      4. Problems with cross-compiling
      5. CMake
    8. Summary
    9. Further reading
  9. Chapter 3: All about Bootloaders
    1. Technical requirements
    2. What does a bootloader do?
    3. The boot sequence
      1. Phase 1 – ROM code
      2. Phase 2 – secondary program loader
      3. Phase 3 – TPL
    4. Moving from the bootloader to a kernel
    5. Introducing device trees
      1. Device tree basics
      2. The reg property
      3. Labels and interrupts
      4. Device tree include files
      5. Compiling a device tree
    6. U-Boot
      1. Building U-Boot
      2. Installing U-Boot
      3. Using U-Boot
      4. Booting Linux
      5. Porting U-Boot to a new board
      6. Building and testing
      7. Falcon mode
    7. Summary
  10. Chapter 4: Configuring and Building the Kernel
    1. Technical requirements
    2. What does the kernel do?
    3. Choosing a kernel
      1. Kernel development cycle
      2. Stable and long-term support releases
    4. Building the kernel
      1. Getting the source
      2. Understanding kernel configuration – Kconfig
      3. Using LOCALVERSION to identify your kernel
      4. When to use kernel modules
    5. Compiling – Kbuild
      1. Finding out which kernel target to build
      2. Build artifacts
      3. Compiling device trees
      4. Compiling modules
      5. Cleaning kernel sources
      6. Building a 64-bit kernel for the Raspberry Pi 4
      7. Building a kernel for the BeagleBone Black
      8. Building a kernel for QEMU
    6. Booting the kernel
      1. Booting the Raspberry Pi 4
      2. Booting the BeagleBone Black
      3. Booting QEMU
      4. Kernel panic
      5. Early user space
      6. Kernel messages
      7. The kernel command line
    7. Porting Linux to a new board
      1. A new device tree
      2. Setting the board's compatible property
    8. Summary
    9. Additional reading
  11. Chapter 5: Building a Root Filesystem
    1. Technical requirements
    2. What should be in the root filesystem?
      1. The directory layout
      2. The staging directory
      3. POSIX file access permissions
      4. File ownership permissions in the staging directory
      5. Programs for the root filesystem
      6. Libraries for the root filesystem
      7. Device nodes
      8. The proc and sysfs filesystems
      9. Kernel modules
    3. Transferring the root filesystem to the target
    4. Creating a boot initramfs
      1. Standalone initramfs
      2. Booting the initramfs
      3. Booting with QEMU
      4. Booting the BeagleBone Black
      5. Building an initramfs into the kernel image
      6. Building an initramfs using a device table
      7. The old initrd format
    5. The init program
      1. Starting a daemon process
    6. Configuring user accounts
      1. Adding user accounts to the root filesystem
    7. A better way of managing device nodes
      1. An example using devtmpfs
      2. An example using mdev
      3. Are static device nodes so bad after all?
    8. Configuring the network
      1. Network components for glibc
    9. Creating filesystem images with device tables
      1. Booting the BeagleBone Black
    10. Mounting the root filesystem using NFS
      1. Testing with QEMU
      2. Testing with the BeagleBone Black
      3. Problems with file permissions
    11. Using TFTP to load the kernel
    12. Summary
    13. Further reading
  12. Chapter 6: Selecting a Build System
    1. Technical requirements
    2. Comparing build systems
    3. Distributing binaries
    4. Introducing Buildroot
      1. Background
      2. Stable releases and long-term support
      3. Installing
      4. Configuring
      5. Running
      6. Targeting real hardware
      7. Creating a custom BSP
      8. Adding your own code
      9. License compliance
    5. Introducing the Yocto Project
      1. Background
      2. Stable releases and supports
      3. Installing the Yocto Project
      4. Configuring
      5. Building
      6. Running the QEMU target
      7. Layers
      8. Customizing images via local.conf
      9. Writing an image recipe
      10. Creating an SDK
      11. The license audit
    6. Summary
    7. Further reading
  13. Chapter 7: Developing with Yocto
    1. Technical requirements
    2. Building on top of an existing BSP
      1. Building an existing BSP
      2. Controlling Wi-Fi
      3. Controlling Bluetooth
      4. Adding a custom layer
    3. Capturing changes with devtool
      1. Development workflows
      2. Creating a new recipe
      3. Modifying the source built by a recipe
      4. Upgrading a recipe to a newer version
    4. Building your own distro
      1. When and when not to
      2. Creating a new distro layer
      3. Configuring your distro
      4. Adding more recipes to your distro
      5. Runtime package management
    5. Provisioning a remote package server
    6. Summary
    7. Further reading
  14. Chapter 8: Yocto Under the Hood
    1. Technical requirements
    2. Decomposing Yocto's architecture and workflow
      1. Metadata
      2. Build tasks
      3. Image generation
    3. Separating metadata into layers
    4. Troubleshooting build failures
      1. Isolating errors
      2. Dumping the environment
      3. Reading the task log
      4. Adding more logging
      5. Running commands from devshell
      6. Graphing dependencies
    5. Understanding BitBake syntax and semantics
      1. Tasks
      2. Dependencies
      3. Variables
      4. Functions
      5. RDEPENDS revisited
    6. Summary
    7. Further reading
  15. Section 2: System Architecture and Design Decisions
  16. Chapter 9: Creating a Storage Strategy
    1. Technical requirements
    2. Storage options
      1. NOR flash
      2. NAND flash
    3. Accessing flash memory from the bootloader
      1. U-Boot and NOR flash
      2. U-Boot and NAND flash
      3. U-Boot and MMC, SD, and eMMC
    4. Accessing flash memory from Linux
      1. Memory technology devices
      2. The MMC block driver
    5. Filesystems for flash memory
      1. Flash translation layers
    6. Filesystems for NOR and NAND flash memory
      1. JFFS2
      2. YAFFS2
      3. UBI and UBIFS
    7. Filesystems for managed flash
      1. Flashbench
      2. Discard and TRIM
      3. Ext4
      4. F2FS
      5. FAT16/32
    8. Read-only compressed filesystems
      1. SquashFS
    9. Temporary filesystems
    10. Making the root filesystem read-only
    11. Filesystem choices
    12. Summary
    13. Further reading
  17. Chapter 10: Updating Software in the Field
    1. Technical requirements
    2. From where do updates originate?
    3. What to update
      1. Bootloader
      2. Kernel
      3. Root filesystem
      4. System applications
      5. Device-specific data
      6. Components that need to be updated
    4. The basics of software updates
      1. Making updates robust
      2. Making updates fail-safe
      3. Making updates secure
    5. Types of update mechanism
      1. Symmetric image update
      2. Asymmetric image update
      3. Atomic file updates
    6. OTA updates
    7. Using Mender for local updates
      1. Building the Mender client
    8. Installing an update
    9. Using Mender for OTA updates
    10. Using balena for local updates
      1. Creating an account
      2. Creating an application
      3. Adding a device
      4. Installing the CLI
      5. Pushing a project
    11. Summary
  18. Chapter 11: Interfacing with Device Drivers
    1. Technical requirements
    2. The role of device drivers
    3. Character devices
    4. Block devices
    5. Network devices
    6. Finding out about drivers at runtime
      1. Getting information from sysfs
    7. Finding the right device driver
    8. Device drivers in user space
      1. GPIO
      2. LEDs
      3. I2C
      4. SPI
    9. Writing a kernel device driver
      1. Designing a character driver interface
      2. The anatomy of a device driver
      3. Compiling kernel modules
      4. Loading kernel modules
    10. Discovering the hardware configuration
      1. Device trees
      2. The platform data
      3. Linking hardware with device drivers
    11. Summary
    12. Further reading
  19. Chapter 12: Prototyping with Breakout Boards
    1. Technical requirements
    2. Mapping schematics to the device tree's source
      1. Reading schematics and data sheets
      2. Installing Debian on the BeagleBone Black
      3. Enabling spidev
      4. Customizing the device tree
    3. Prototyping with breakout boards
      1. Closing the SPI jumper
      2. Attaching the GNSS antenna
      3. Attaching the SPI header
      4. Connecting the SPI jumper wires
    4. Probing SPI signals with a logic analyzer
    5. Receiving NMEA messages over SPI
    6. Summary
    7. Further reading
  20. Chapter 13: Starting Up – The init Program
    1. Technical requirements
    2. After the kernel has booted
    3. Introducing the init programs
    4. BusyBox init
      1. Buildroot init scripts
    5. System V init
      1. inittab
      2. The init.d scripts
      3. Adding a new daemon
      4. Starting and stopping services
    6. systemd
      1. Building systemd with the Yocto Project and Buildroot
      2. Introducing targets, services, and units
      3. How systemd boots the system
      4. Adding your own service
      5. Adding a watchdog
      6. Implications for embedded Linux
    7. Summary
    8. Further reading
  21. Chapter 14: Starting with BusyBox runit
    1. Technical requirements
    2. Getting BusyBox runit
    3. Creating service directories and files
      1. Service directory layout
      2. Service configuration
    4. Service supervision
      1. Controlling services
    5. Depending on other services
      1. Start dependencies
      2. Custom start dependencies
      3. Putting it all together
    6. Dedicated service logging
      1. How does it work?
      2. Adding dedicated logging to a service
      3. Log rotation
    7. Signaling a service
    8. Summary
    9. Further reading
  22. Chapter 15: Managing Power
    1. Technical requirements
    2. Measuring power usage
    3. Scaling the clock frequency
      1. The CPUFreq driver
      2. Using CPUFreq
    4. Selecting the best idle state
      1. The CPUIdle driver
      2. Tickless operation
    5. Powering down peripherals
    6. Putting the system to sleep
      1. Power states
      2. Wakeup events
      3. Timed wakeups from the real-time clock
    7. Summary
    8. Further reading
  23. Section 3: Writing Embedded Applications
  24. Chapter 16: Packaging Python
    1. Technical requirements
      1. Getting Docker
    2. Retracing the origins of Python packaging
      1. distutils
      2. setuptools
      3. setup.py
    3. Installing Python packages with pip
      1. requirements.txt
    4. Managing Python virtual environments with venv
    5. Installing precompiled binaries with conda
      1. Environment management
      2. Package management
    6. Deploying Python applications with Docker
      1. The anatomy of a Dockerfile
      2. Building a Docker image
      3. Running a Docker image
      4. Fetching a Docker image
      5. Publishing a Docker image
      6. Cleaning up
    7. Summary
    8. Further reading
  25. Chapter 17: Learning about Processes and Threads
    1. Technical requirements
    2. Process or thread?
    3. Processes
      1. Creating a new process
      2. Terminating a process
      3. Running a different program
      4. Daemons
      5. Inter-process communication
    4. Threads
      1. Creating a new thread
      2. Terminating a thread
      3. Compiling a program with threads
      4. Inter-thread communication
      5. Mutual exclusion
      6. Changing conditions
      7. Partitioning the problem
    5. ZeroMQ
      1. Getting pyzmq
      2. Messaging between processes
      3. Messaging within processes
    6. Scheduling
      1. Fairness versus determinism
      2. Time-shared policies
      3. Real-time policies
      4. Choosing a policy
      5. Choosing a real-time priority
    7. Summary
    8. Further reading
  26. Chapter 18: Managing Memory
    1. Technical requirements
    2. Virtual memory basics
    3. Kernel space memory layout
      1. How much memory does the kernel use?
    4. User space memory layout
    5. The process memory map
    6. Swapping
      1. Swapping to compressed memory (zram)
    7. Mapping memory with mmap
      1. Using mmap to allocate private memory
      2. Using mmap to share memory
      3. Using mmap to access device memory
    8. How much memory does my application use?
    9. Per-process memory usage
      1. Using top and ps
      2. Using smem
      3. Other tools to consider
    10. Identifying memory leaks
      1. mtrace
      2. Valgrind
    11. Running out of memory
    12. Summary
    13. Further reading
  27. Section 4: Debugging and Optimizing Performance
  28. Chapter 19: Debugging with GDB
    1. Technical requirements
    2. The GNU debugger
    3. Preparing to debug
    4. Debugging applications
      1. Remote debugging using gdbserver
      2. Setting up the Yocto Project for remote debugging
      3. Setting up Buildroot for remote debugging
      4. Starting to debug
      5. Native debugging
    5. Just-in-time debugging
    6. Debugging forks and threads
    7. Core files
      1. Using GDB to look at core files
    8. GDB user interfaces
      1. Terminal User Interface
      2. Data Display Debugger
      3. Visual Studio Code
    9. Debugging kernel code
      1. Debugging kernel code with kgdb
      2. A sample debug session
      3. Debugging early code
      4. Debugging modules
      5. Debugging kernel code with kdb
      6. Looking at an Oops
      7. Preserving the Oops
    10. Summary
    11. Further reading
  29. Chapter 20: Profiling and Tracing
    1. Technical requirements
    2. The observer effect
      1. Symbol tables and compile flags
    3. Beginning to profile
    4. Profiling with top
    5. The poor man's profiler
    6. Introducing perf
      1. Configuring the kernel for perf
      2. Building perf with the Yocto Project
      3. Building perf with Buildroot
      4. Profiling with perf
      5. Call graphs
      6. perf annotate
    7. Tracing events
    8. Introducing Ftrace
      1. Preparing to use Ftrace
      2. Using Ftrace
      3. Dynamic Ftrace and trace filters
      4. Trace events
    9. Using LTTng
      1. LTTng and the Yocto Project
      2. LTTng and Buildroot
      3. Using LTTng for kernel tracing
    10. Using BPF
      1. Configuring the kernel for BPF
      2. Building a BCC toolkit with Buildroot
      3. Using BPF tracing tools
    11. Using Valgrind
      1. Callgrind
      2. Helgrind
    12. Using strace
    13. Summary
    14. Further reading
  30. Chapter 21: Real-Time Programming
    1. Technical requirements
    2. What is real time?
    3. Identifying sources of non-determinism
    4. Understanding scheduling latency
    5. Kernel preemption
      1. The real-time Linux kernel (PREEMPT_RT)
      2. Threaded interrupt handlers
    6. Preemptible kernel locks
      1. Getting the PREEMPT_RT patches
      2. The Yocto Project and PREEMPT_RT
    7. High-resolution timers
    8. Avoiding page faults
    9. Interrupt shielding
    10. Measuring scheduling latencies
      1. cyclictest
      2. Using Ftrace
      3. Combining cyclictest and Ftrace
    11. Summary
    12. Further reading
    13. Why subscribe?
  31. Other Books You May Enjoy
    1. Packt is searching for authors like you
    2. Leave a review - let other readers know what you think

Product information

  • Title: Mastering Embedded Linux Programming - Third Edition
  • Author(s): Frank Vasquez, Chris Simmonds
  • Release date: May 2021
  • Publisher(s): Packt Publishing
  • ISBN: 9781789530384