Federated Learning with Python

Federated Learning with Python

by Kiyoshi Nakayama PhD, George Jeno
Released October 2022
Publisher(s): Packt Publishing
ISBN: 9781803247106

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Learn the essential skills for building an authentic federated learning system with Python and take your machine learning applications to the next level

Key Features

  • Design distributed systems that can be applied to real-world federated learning applications at scale
  • Discover multiple aggregation schemes applicable to various ML settings and applications
  • Develop a federated learning system that can be tested in distributed machine learning settings

Book Description

Federated learning (FL) is a paradigm-shifting technology in AI that enables and accelerates machine learning (ML), allowing you to work on private data. It has become a must-have solution for most enterprise industries, making it a critical part of your learning journey. This book helps you get to grips with the building blocks of FL and how the systems work and interact with each other using solid coding examples.

FL is more than just aggregating collected ML models and bringing them back to the distributed agents. This book teaches you about all the essential basics of FL and shows you how to design distributed systems and learning mechanisms carefully so as to synchronize the dispersed learning processes and synthesize the locally trained ML models in a consistent manner. This way, you'll be able to create a sustainable and resilient FL system that can constantly function in real-world operations. This book goes further than simply outlining FL's conceptual framework or theory, as is the case with the majority of research-related literature.

By the end of this book, you'll have an in-depth understanding of the FL system design and implementation basics and be able to create an FL system and applications that can be deployed to various local and cloud environments.

What you will learn

  • Discover the challenges related to centralized big data ML that we currently face along with their solutions
  • Understand the theoretical and conceptual basics of FL
  • Acquire design and architecting skills to build an FL system
  • Explore the actual implementation of FL servers and clients
  • Find out how to integrate FL into your own ML application
  • Understand various aggregation mechanisms for diverse ML scenarios
  • Discover popular use cases and future trends in FL

Who this book is for

This book is for machine learning engineers, data scientists, and artificial intelligence (AI) enthusiasts who want to learn about creating machine learning applications empowered by federated learning. You'll need basic knowledge of Python programming and machine learning concepts to get started with this book.

Table of contents

  1. Federated Learning with Python
  2. Acknowledgments
  3. Contributors
  4. About the authors
  5. About the reviewer
  6. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Download the color images
    6. Conventions used
    7. Get in touch
    8. Share Your Thoughts
    9. Download a free PDF copy of this book
  7. Part 1 Federated Learning – Conceptual Foundations
  8. Chapter 1: Challenges in Big Data and Traditional AI
    1. Understanding the nature of big data
      1. Definition of big data
      2. Big data now
      3. Triple-A mindset for big data
    2. Data privacy as a bottleneck
      1. Risks in handling private data
      2. Increased data protection regulations
      3. From privacy by design to data minimalism
    3. Impacts of training data and model bias
      1. Expensive training of big data
      2. Model bias and training data
    4. Model drift and performance degradation
      1. How models can stop working
      2. Continuous monitoring – the price of letting causation go
    5. FL as the main solution for data problems
    6. Summary
    7. Further reading
  9. Chapter 2: What Is Federated Learning?
    1. Understanding the current state of ML
      1. What is a model?
      2. ML – automating the model creation process
      3. Deep learning
    2. Distributed learning nature – toward scalable AI
      1. Distributed computing
      2. Distributed ML
      3. Edge inference
      4. Edge training
    3. Understanding FL
      1. Defining FL
      2. The FL process
    4. FL system considerations
      1. Security for FL systems
      2. Decentralized FL and blockchain
    5. Summary
    6. Further reading
  10. Chapter 3: Workings of the Federated Learning System
    1. FL system architecture
      1. Cluster aggregators
      2. Distributed agents
      3. Database servers
      4. Intermediate servers for low computational agent devices
    2. Understanding the FL system flow – from initialization to continuous operation
      1. Initialization of the database, aggregator, and agent
      2. Initial model upload process by initial agent
      3. Overall FL cycle and process of the FL system
      4. Synchronous and asynchronous FL
      5. The aggregator-side FL cycle and process
      6. The agent-side local retraining cycle and process
      7. Model interpretation based on deviation from baseline outputs
    3. Basics of model aggregation
      1. What exactly does it mean to aggregate models?
      2. FedAvg – Federated averaging
    4. Furthering scalability with horizontal design
      1. Horizontal design with semi-global model
      2. Distributed database
      3. Asynchronous agent participation in a multiple-aggregator scenario
      4. Semi-global model synthesis
    5. Summary
    6. Further reading
  11. Part 2 The Design and Implementation of the Federated Learning System
  12. Chapter 4: Federated Learning Server Implementation with Python
    1. Technical requirements
    2. Main software components of the aggregator 
and database
      1. Aggregator-side codes
      2. lib/util codes
      3. Database-side code
      4. Toward the configuration of the aggregator
    3. Implementing FL server-side functionalities
      1. Importing libraries for the FL server
      2. Defining the FL Server class
      3. Initializing the FL server
      4. Registration function of agents
      5. The server for handling messages from local agents
      6. The global model synthesis routine
      7. Functions to send the global models to the agents
      8. Functions to push the local and global models to the database
    4. Maintaining models for aggregation with the 
state manager
      1. Importing the libraries of the state manager
      2. Defining the state manager class
      3. Initializing the state manager
      4. Initializing a global model
      5. Checking the aggregation criteria
      6. Buffering the local models
      7. Clearing the saved models
      8. Adding agents
      9. Incrementing the FL round
    5. Aggregating local models
      1. Importing the libraries for the aggregator
      2. Defining and initializing the aggregator class
      3. Defining the aggregate_local_models function
      4. The FedAvg function
    6. Running the FL server
    7. Implementing and running the database server
      1. Toward the configuration of the database
      2. Defining the database server
      3. efining the database with SQLite
      4. Running the database server
    8. Potential enhancements to the FL server
      1. Redesigning the database
      2. Automating the registry of an initial model
      3. Performance metrics for local and global models
      4. Fine-tuned aggregation
    9. Summary
  13. Chapter 5: Federated Learning Client-Side Implementation
    1. Technical requirements
    2. An overview of FL client-side components
      1. Distributed agent-side code
      2. Configuration of an agent
    3. Implementing FL client-side main functionalities
      1. Importing libraries for an agent
      2. Defining the Client class
      3. Initializing the client
      4. Agent participation in an FL cycle
      5. Model exchange synchronization
      6. Push and polling implementation
    4. Designing FL client libraries
      1. Starting FL client core threads
      2. Saving global models
      3. Manipulating client state
      4. Sending local models to aggregator
      5. Waiting for global models from an aggregator
    5. Local ML engine integration into an FL system
      1. Importing libraries for a local ML engine
      2. Defining the ML models, training, and test functions
      3. Integration of client libraries into your local ML engine
    6. An example of integrating image classification 
into an FL system
      1. Integration of client libraries into the IC example
    7. Summary
  14. Chapter 6: Running the Federated Learning System and Analyzing the Results
    1. Technical requirements
    2. Configuring and running the FL system
      1. Installing the FL environment
      2. Configuring the FL system with JSON files for each component
      3. Running the database and aggregator on the FL server
      4. Running a minimal example with the FL client
      5. Data and database folders
      6. Databases with SQLite
    3. Understanding what happens when the minimal example runs
      1. Running just one minimal agent
      2. Running two minimal agents
    4. Running image classification and analyzing the results
      1. Preparing the CIFAR-10 dataset
      2. The ML model used for FL with image classification
      3. How to run the image classification example with CNN
      4. Evaluation of running the image classification with CNN
      5. Running five agents
    5. Summary
  15. Chapter 7: Model Aggregation
    1. Technical requirements
    2. Revisiting aggregation
    3. Understanding FedAvg
      1. Dataset distributions
      2. Computational power distributions
      3. Protecting against adversarial agents
    4. Modifying aggregation for non-ideal cases
      1. Handling heterogeneous computational power
      2. Adversarial agents
      3. Non-IID datasets
    5. Summary
  16. Part 3 Moving Toward the Production of Federated Learning Applications
  17. Chapter 8: Introducing Existing Federated Learning Frameworks
    1. Technical requirements
      1. TensorFlow Federated
      2. OpenFL
      3. IBM FL
      4. Flower
      5. STADLE
    2. Introduction to FL frameworks
      1. Flower
      2. TensorFlow Federated (TFF)
      3. OpenFL
      4. IBM FL
      5. STADLE
      6. PySyft
    3. Example – the federated training of an NLP model
      1. Defining the sentiment analysis model
      2. Creating the data loader
      3. Training the model
      4. Adopting an FL training approach
      5. Integrating TensorFlow Federated for SST-2
      6. Integrating OpenFL for SST-2
      7. Integrating IBM FL for SST-2
      8. Integrating Flower for SST-2
      9. Integrating STADLE for SST-2
    4. Example – the federated training of an image classification model on non-IID data
      1. Skewing the CIFAR-10 dataset
      2. Integrating OpenFL for CIFAR-10
      3. Integrating IBM FL for CIFAR-10
      4. Integrating Flower for CIFAR-10
      5. Integrating STADLE for CIFAR-10
    5. Summary
  18. Chapter 9: Case Studies with Key Use Cases of Federated Learning Applications
    1. Applying FL to the healthcare sector
      1. Challenges in healthcare
      2. Medical imaging
      3. Drug discovery
      4. EHRs
    2. Applying FL to the financial sector
      1. Anti-Money Laundering (AML)
      2. Proposed solutions to the existing AML approach
      3. Demo of FL in the AML space
      4. Benefits of FL for risk detection systems
    3. FL meets edge computing
      1. Edge computing with IoT over 5G
      2. Edge FL example – object detection
      3. Making autonomous driving happen with FL
      4. Applying FL to robotics
    4. Moving toward the Internet of Intelligence
      1. Introducing the IoFT
      2. Understanding the role of FL in Web 3.0
    5. Applying FL to distributed learning for big data
    6. Summary
    7. References
    8. Further reading
  19. Chapter 10: Future Trends and Developments
    1. Looking at future AI trends
      1. The limitation of centralized ML
      2. Revisiting the benefits of FL
      3. Toward distributed learning for further privacy and training efficiency
    2. Ongoing research and developments in FL
      1. Exploring various FL types and approaches
      2. Understanding enhanced distributed learning frameworks with FL
    3. Journeying on to collective intelligence
      1. Intelligence-centric era with collective intelligence
      2. Internet of Intelligence
      3. Crowdsourced learning with FL
    4. Summary
    5. Further reading
  20. Appendix: Exploring Internal Libraries
    1. Technical requirements
    2. Overview of the internal libraries for the FL system
      1. states.py
      2. communication_handler.py
      3. data_struc.py
      4. helpers.py
      5. messengers.py
    3. Enumeration classes for implementing the FL system
      1. Importing libraries to define the enumeration classes
      2. IDPrefix defining the FL system components
      3. Client state classes
      4. List of classes defining the types of ML models and messages
      5. List of state classes defining message location
    4. Understanding communication handler functionalities
      1. Importing libraries for the communication handler
      2. Functions of the communication handler
    5. Understanding the data structure handler class
      1. Importing libraries for the data structure handler
      2. The LimitedDict class
    6. Understanding helper and supporting libraries
      1. Importing libraries for helper libraries
      2. Functions of the helper library
    7. Messengers to generate communication payloads
      1. Importing libraries for messengers
      2. Functions of messengers
    8. Summary
  21. Index
    1. Why subscribe?
  22. Other Books You May Enjoy
    1. Packt is searching for authors like you
    2. Share Your Thoughts
    3. Download a free PDF copy of this book

Product information

  • Title: Federated Learning with Python
  • Author(s): Kiyoshi Nakayama PhD, George Jeno
  • Release date: October 2022
  • Publisher(s): Packt Publishing
  • ISBN: 9781803247106