Stochastic Optimization for Large-scale Machine Learning

Stochastic Optimization for Large-scale Machine Learning

by Vinod Kumar Chauhan
Released November 2021
Publisher(s): CRC Press
ISBN: 9781000505610

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Stochastic Optimization for Large-scale Machine Learning identifies different areas of improvement and recent research directions to tackle the challenge. Developed optimisation techniques are also explored to improve machine learning algorithms based on data access and on first and second order optimisation methods.

Table of contents

  1. Cover Page
  2. Half-Title Page
  3. Title Page
  4. Copyright Page
  5. Dedication Page
  6. Contents
  7. List of Figures
  8. List of Tables
  9. Preface
  10. SECTION I BACKGROUND
    1. CHAPTER 1 ◾ Introduction
      1. 1.1 LARGE-SCALE MACHINE LEARNING
      2. 1.2 OPTIMIZATION PROBLEMS
      3. 1.3 LINEAR CLASSIFICATION
        1. 1.3.1 Support Vector Machine (SVM)
        2. 1.3.2 Logistic Regression
        3. 1.3.3 First and Second Order Methods
          1. 1.3.3.1 First Order Methods
          2. 1.3.3.2 Second Order Methods
      4. 1.4 STOCHASTIC APPROXIMATION APPROACH
      5. 1.5 COORDINATE DESCENT APPROACH
      6. 1.6 DATASETS
      7. 1.7 ORGANIZATION OF BOOK
    2. CHAPTER 2 ◾ Optimization Problem, Solvers, Challenges and Research Directions
      1. 2.1 INTRODUCTION
        1. 2.1.1 Contributions
      2. 2.2 LITERATURE
      3. 2.3 PROBLEM FORMULATIONS
        1. 2.3.1 Hard Margin SVM (1992)
        2. 2.3.2 Soft Margin SVM (1995)
        3. 2.3.3 One-versus-Rest (1998)
        4. 2.3.4 One-versus-One (1999)
        5. 2.3.5 Least Squares SVM (1999)
        6. 2.3.6 ν-SVM (2000)
        7. 2.3.7 Smooth SVM (2001)
        8. 2.3.8 Proximal SVM (2001)
        9. 2.3.9 Crammer Singer SVM (2002)
        10. 2.3.10 Eν-SVM (2003)
        11. 2.3.11 Twin SVM (2007)
        12. 2.3.12 Capped lp-norm SVM (2017)
      4. 2.4 PROBLEM SOLVERS
        1. 2.4.1 Exact Line Search Method
        2. 2.4.2 Backtracking Line Search
        3. 2.4.3 Constant Step Size
        4. 2.4.4 Lipschitz and Strong Convexity Constants
        5. 2.4.5 Trust Region Method
        6. 2.4.6 Gradient Descent Method
        7. 2.4.7 Newton Method
        8. 2.4.8 Gauss-Newton Method
        9. 2.4.9 Levenberg-Marquardt Method
        10. 2.4.10 Quasi-Newton Method
        11. 2.4.11 Subgradient Method
        12. 2.4.12 Conjugate Gradient Method
        13. 2.4.13 Truncated Newton Method
        14. 2.4.14 Proximal Gradient Method
        15. 2.4.15 Recent Algorithms
      5. 2.5 COMPARATIVE STUDY
        1. 2.5.1 Results from Literature
        2. 2.5.2 Results from Experimental Study
          1. 2.5.2.1 Experimental Setup and Implementation Details
          2. 2.5.2.2 Results and Discussions
      6. 2.6 CURRENT CHALLENGES AND RESEARCH DIRECTIONS
        1. 2.6.1 Big Data Challenge
        2. 2.6.2 Areas of Improvement
          1. 2.6.2.1 Problem Formulations
          2. 2.6.2.2 Problem Solvers
          3. 2.6.2.3 Problem Solving Strategies/Approaches
          4. 2.6.2.4 Platforms/Frameworks
        3. 2.6.3 Research Directions
          1. 2.6.3.1 Stochastic Approximation Algorithms
          2. 2.6.3.2 Coordinate Descent Algorithms
          3. 2.6.3.3 Proximal Algorithms
          4. 2.6.3.4 Parallel/Distributed Algorithms
          5. 2.6.3.5 Hybrid Algorithms
      7. 2.7 CONCLUSION
  11. SECTION II FIRST ORDER METHODS
    1. CHAPTER 3 ◾ Mini-batch and Block-coordinate Approach
      1. 3.1 INTRODUCTION
        1. 3.1.1 Motivation
        2. 3.1.2 Batch Block Optimization Framework (BBOF)
        3. 3.1.3 Brief Literature Review
        4. 3.1.4 Contributions
      2. 3.2 STOCHASTIC AVERAGE ADJUSTED GRADIENT (SAAG) METHODS
      3. 3.3 ANALYSIS
      4. 3.4 NUMERICAL EXPERIMENTS
        1. 3.4.1 Experimental Setup
        2. 3.4.2 Convergence against Epochs
        3. 3.4.3 Convergence against Time
      5. 3.5 CONCLUSION AND FUTURE SCOPE
    2. CHAPTER 4 ◾ Variance Reduction Methods
      1. 4.1 INTRODUCTION
        1. 4.1.1 Optimization Problem
        2. 4.1.2 Solution Techniques for Optimization Problem
        3. 4.1.3 Contributions
      2. 4.2 NOTATIONS AND RELATED WORK
        1. 4.2.1 Notations
        2. 4.2.2 Related Work
      3. 4.3 SAAG-I, II AND PROXIMAL EXTENSIONS
      4. 4.4 SAAG-III AND IV ALGORITHMS
      5. 4.5 ANALYSIS
      6. 4.6 EXPERIMENTAL RESULTS
        1. 4.6.1 Experimental Setup
        2. 4.6.2 Results with Smooth Problem
        3. 4.6.3 Results with Non-smooth Problem
        4. 4.6.4 Mini-batch Block-coordinate versus Mini-batch setting
        5. 4.6.5 Results with SVM
      7. 4.7 CONCLUSION
    3. CHAPTER 5 ◾ Learning and Data Access
      1. 5.1 INTRODUCTION
        1. 5.1.1 Optimization Problem
        2. 5.1.2 Literature Review
        3. 5.1.3 Contributions
      2. 5.2 SYSTEMATIC SAMPLING
        1. 5.2.1 Definitions
        2. 5.2.2 Learning Using Systematic Sampling
      3. 5.3 ANALYSIS
      4. 5.4 EXPERIMENTS
        1. 5.4.1 Experimental Setup
        2. 5.4.2 Implementation Details
        3. 5.4.3 Results
      5. 5.5 CONCLUSION
  12. SECTION III SECOND ORDER METHODS
    1. CHAPTER 6 ◾ Mini-batch Block-coordinate Newton Method
      1. 6.1 INTRODUCTION
        1. 6.1.1 Contributions
      2. 6.2 MBN
      3. 6.3 EXPERIMENTS
        1. 6.3.1 Experimental Setup
        2. 6.3.2 Comparative Study
      4. 6.4 CONCLUSION
    2. CHAPTER 7 ◾ Stochastic Trust Region Inexact Newton Method
      1. 7.1 INTRODUCTION
        1. 7.1.1 Optimization Problem
        2. 7.1.2 Solution Techniques
        3. 7.1.3 Contributions
      2. 7.2 LITERATURE REVIEW
      3. 7.3 TRUST REGION INEXACT NEWTON METHOD
        1. 7.3.1 Inexact Newton Method
        2. 7.3.2 Trust Region Inexact Newton Method
      4. 7.4 STRON
        1. 7.4.1 Complexity
        2. 7.4.2 Analysis
      5. 7.5 EXPERIMENTAL RESULTS
        1. 7.5.1 Experimental Setup
        2. 7.5.2 Comparative Study
        3. 7.5.3 Results with SVM
      6. 7.6 EXTENSIONS
        1. 7.6.1 PCG Subproblem Solver
        2. 7.6.2 Stochastic Variance Reduced Trust Region Inexact Newton Method
      7. 7.7 CONCLUSION
  13. SECTION IV CONCLUSION
    1. CHAPTER 8 ◾ Conclusion and Future Scope
      1. 8.1 FUTURE SCOPE
  14. Bibliography
  15. Index

Product information

  • Title: Stochastic Optimization for Large-scale Machine Learning
  • Author(s): Vinod Kumar Chauhan
  • Release date: November 2021
  • Publisher(s): CRC Press
  • ISBN: 9781000505610