Probability, Random Variables, and Random Processes: Theory and Signal Processing Applications

Probability, Random Variables, and Random Processes: Theory and Signal Processing Applications

by John J. Shynk
Released November 2012
Publisher(s): Wiley-Interscience
ISBN: 9780470242094

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Probability, Random Variables, and Random Processes is a comprehensive textbook on probability theory for engineers that provides a more rigorous mathematical framework than is usually encountered in undergraduate courses. It is intended for first-year graduate students who have some familiarity with probability and random variables, though not necessarily of random processes and systems that operate on random signals. It is also appropriate for advanced undergraduate students who have a strong mathematical background.

The book has the following features:

  • Several appendices include related material on integration, important inequalities and identities, frequency-domain transforms, and linear algebra. These topics have been included so that the book is relatively self-contained. One appendix contains an extensive summary of 33 random variables and their properties such as moments, characteristic functions, and entropy.

  • Unlike most books on probability, numerous figures have been included to clarify and expand upon important points. Over 600 illustrations and MATLAB plots have been designed to reinforce the material and illustrate the various characterizations and properties of random quantities.

  • Sufficient statistics are covered in detail, as is their connection to parameter estimation techniques. These include classical Bayesian estimation and several optimality criteria: mean-square error, mean-absolute error, maximum likelihood, method of moments, and least squares.

  • The last four chapters provide an introduction to several topics usually studied in subsequent engineering courses: communication systems and information theory; optimal filtering (Wiener and Kalman); adaptive filtering (FIR and IIR); and antenna beamforming, channel equalization, and direction finding. This material is available electronically at the companion website.

    • Probability, Random Variables, and Random Processes is the only textbook on probability for engineers that includes relevant background material, provides extensive summaries of key results, and extends various statistical techniques to a range of applications in signal processing.

    Note: The ebook version does not provide access to the companion files.

    Table of contents

    1. Cover
    2. Title Page
    3. Copyright
    4. Dedication
    5. Preface
    6. Notation
    7. Chapter 1: Overview and Background
      1. 1.1 INTRODUCTION
      2. 1.2 DETERMINISTIC SIGNALS AND SYSTEMS
      3. 1.3 STATISTICAL SIGNAL PROCESSING WITH MATLAB®
      4. PROBLEMS
      5. FURTHER READING
    8. Part I: Probability, Random Variables, and Expectation
      1. Chapter 2: Probability Theory
        1. 2.1 INTRODUCTION
        2. 2.2 SETS AND SAMPLE SPACES
        3. 2.3 SET OPERATIONS
        4. 2.4 EVENTS AND FIELDS
        5. 2.5 SUMMARY OF A RANDOM EXPERIMENT
        6. 2.6 MEASURE THEORY
        7. 2.7 AXIOMS OF PROBABILITY
        8. 2.8 BASIC PROBABILITY RESULTS
        9. 2.9 CONDITIONAL PROBABILITY
        10. 2.10 INDEPENDENCE
        11. 2.11 BAYES' FORMULA
        12. 2.12 TOTAL PROBABILITY
        13. 2.13 DISCRETE SAMPLE SPACES
        14. 2.14 CONTINUOUS SAMPLE SPACES
        15. 2.15 NONMEASURABLE SUBSETS OF R
        16. PROBLEMS
        17. FURTHER READING
      2. Chapter 3: Random Variables
        1. 3.1 INTRODUCTION
        2. 3.2 FUNCTIONS AND MAPPINGS
        3. 3.3 DISTRIBUTION FUNCTION
        4. 3.4 PROBABILITY MASS FUNCTION
        5. 3.5 PROBABILITY DENSITY FUNCTION
        6. 3.6 MIXED DISTRIBUTIONS
        7. 3.7 PARAMETRIC MODELS FOR RANDOM VARIABLES
        8. 3.8 CONTINUOUS RANDOM VARIABLES
        9. 3.9 DISCRETE RANDOM VARIABLES
        10. PROBLEMS
        11. FURTHER READING
      3. Chapter 4: Multiple Random Variables
        1. 4.1 INTRODUCTION
        2. 4.2 RANDOM VARIABLE APPROXIMATIONS
        3. 4.3 JOINT AND MARGINAL DISTRIBUTIONS
        4. 4.4 INDEPENDENT RANDOM VARIABLES
        5. 4.5 CONDITIONAL DISTRIBUTION
        6. 4.6 RANDOM VECTORS
        7. 4.7 GENERATING DEPENDENT RANDOM VARIABLES
        8. 4.8 RANDOM VARIABLE TRANSFORMATIONS
        9. 4.9 IMPORTANT FUNCTIONS OF TWO RANDOM VARIABLES
        10. 4.10 TRANSFORMATIONS OF RANDOM VARIABLE FAMILIES
        11. 4.11 TRANSFORMATIONS OF RANDOM VECTORS
        12. 4.12 SAMPLE MEAN X AND SAMPLE VARIANCE S_2
        13. 4.13 MINIMUM, MAXIMUM, AND ORDER STATISTICS
        14. 4.14 MIXTURES
        15. PROBLEMS
        16. FURTHER READING
      4. Chapter 5: Expectation and Moments
        1. 5.1 INTRODUCTION
        2. 5.2 EXPECTATION AND INTEGRATION
        3. 5.3 INDICATOR RANDOM VARIABLE
        4. 5.4 SIMPLE RANDOM VARIABLE
        5. 5.5 EXPECTATION FOR DISCRETE SAMPLE SPACES
        6. 5.6 EXPECTATION FOR CONTINUOUS SAMPLE SPACES
        7. 5.7 SUMMARY OF EXPECTATION
        8. 5.8 FUNCTIONAL VIEW OF THE MEAN
        9. 5.9 PROPERTIES OF EXPECTATION
        10. 5.10 EXPECTATION OF A FUNCTION
        11. 5.11 CHARACTERISTIC FUNCTION
        12. 5.12 CONDITIONAL EXPECTATION
        13. 5.13 PROPERTIES OF CONDITIONAL EXPECTATION
        14. 5.14 LOCATION PARAMETERS: MEAN, MEDIAN, AND MODE
        15. 5.15 VARIANCE, COVARIANCE, AND CORRELATION
        16. 5.16 FUNCTIONAL VIEW OF THE VARIANCE
        17. 5.17 EXPECTATION AND THE INDICATOR FUNCTION
        18. 5.18 CORRELATION COEFFICIENTS
        19. 5.19 ORTHOGONALITY
        20. 5.20 CORRELATION AND COVARIANCE MATRICES
        21. 5.21 HIGHER ORDER MOMENTS AND CUMULANTS
        22. 5.22 FUNCTIONAL VIEW OF SKEWNESS
        23. 5.23 FUNCTIONAL VIEW OF KURTOSIS
        24. 5.24 GENERATING FUNCTIONS
        25. 5.25 FOURTH-ORDER GAUSSIAN MOMENT
        26. 5.26 EXPECTATIONS OF NONLINEAR TRANSFORMATIONS
        27. PROBLEMS
        28. FURTHER READING
    9. Part II: Random Processes, Systems, and Parameter Estimation
      1. Chapter 6: Random Processes
        1. 6.1 INTRODUCTION
        2. 6.2 CHARACTERIZATIONS OF A RANDOM PROCESS
        3. 6.3 CONSISTENCY AND EXTENSION
        4. 6.4 TYPES OF RANDOM PROCESSES
        5. 6.5 STATIONARITY
        6. 6.6 INDEPENDENT AND IDENTICALLY DISTRIBUTED
        7. 6.7 INDEPENDENT INCREMENTS
        8. 6.8 MARTINGALES
        9. 6.9 MARKOV SEQUENCE
        10. 6.10 MARKOV PROCESS
        11. 6.11 RANDOM SEQUENCES
        12. 6.12 RANDOM PROCESSES
        13. PROBLEMS
        14. FURTHER READING
      2. Chapter 7: Stochastic Convergence, Calculus, and Decompositions
        1. 7.1 INTRODUCTION
        2. 7.2 STOCHASTIC CONVERGENCE
        3. 7.3 LAWS OF LARGE NUMBERS
        4. 7.4 CENTRAL LIMIT THEOREM
        5. 7.5 STOCHASTIC CONTINUITY
        6. 7.6 DERIVATIVES AND INTEGRALS
        7. 7.7 DIFFERENTIAL EQUATIONS
        8. 7.8 DIFFERENCE EQUATIONS
        9. 7.9 INNOVATIONS AND MEAN-SQUARE PREDICTABILITY
        10. 7.10 DOOB–MEYER DECOMPOSITION
        11. 7.11 KARHUNEN–LOÈVE EXPANSION
        12. PROBLEMS
        13. FURTHER READING
      3. Chapter 8: Systems, Noise, and Spectrum Estimation
        1. 8.1 INTRODUCTION
        2. 8.2 CORRELATION REVISITED
        3. 8.3 ERGODICITY
        4. 8.4 EIGENFUNCTIONS OF R_XX(τ)
        5. 8.5 POWER SPECTRAL DENSITY
        6. 8.6 POWER SPECTRAL DISTRIBUTION
        7. 8.7 CROSS-POWER SPECTRAL DENSITY
        8. 8.8 SYSTEMS WITH RANDOM INPUTS
        9. 8.9 PASSBAND SIGNALS
        10. 8.10 WHITE NOISE
        11. 8.11 BANDWIDTH
        12. 8.12 SPECTRUM ESTIMATION
        13. 8.13 PARAMETRIC MODELS
        14. 8.14 SYSTEM IDENTIFICATION
        15. PROBLEMS
        16. FURTHER READING
      4. Chapter 9: Sufficient Statistics and Parameter Estimation
        1. 9.1 INTRODUCTION
        2. 9.2 STATISTICS
        3. 9.3 SUFFICIENT STATISTICS
        4. 9.4 MINIMAL SUFFICIENT STATISTIC
        5. 9.5 EXPONENTIAL FAMILIES
        6. 9.6 LOCATION-SCALE FAMILIES
        7. 9.7 COMPLETE STATISTIC
        8. 9.8 RAO–BLACKWELL THEOREM
        9. 9.9 LEHMANN–SCHEFFÉ THEOREM
        10. 9.10 BAYES ESTIMATION
        11. 9.11 MEAN-SQUARE-ERROR ESTIMATION
        12. 9.12 MEAN-ABSOLUTE-ERROR ESTIMATION
        13. 9.13 ORTHOGONALITY CONDITION
        14. 9.14 PROPERTIES OF ESTIMATORS
        15. 9.15 MAXIMUM A POSTERIORI ESTIMATION
        16. 9.16 MAXIMUM LIKELIHOOD ESTIMATION
        17. 9.17 LIKELIHOOD RATIO TEST
        18. 9.18 EXPECTATION–MAXIMIZATION ALGORITHM
        19. 9.19 METHOD OF MOMENTS
        20. 9.20 LEAST-SQUARES ESTIMATION
        21. 9.21 PROPERTIES OF LS ESTIMATORS
        22. 9.22 BEST LINEAR UNBIASED ESTIMATION
        23. 9.23 PROPERTIES OF BLU ESTIMATORS
        24. PROBLEMS
        25. FURTHER READING
        26. A NOTE ON PART III OF THE BOOK
    10. Appendices
      1. Appendix A: Summaries of Univariate Parametric Distributions
        1. A.1 NOTATION
        2. A.2 FURTHER READING
        3. A.3 CONTINUOUS RANDOM VARIABLES
        4. A.4 DISCRETE RANDOM VARIABLES
      2. Appendix B: Functions and Properties
        1. B.1 CONTINUITY AND BOUNDED VARIATION
        2. B.2 SUPREMUM AND INFIMUM
        3. B.3 ORDER NOTATION
        4. B.4 FLOOR AND CEILING FUNCTIONS
        5. B.5 CONVEX AND CONCAVE FUNCTIONS
        6. B.6 EVEN AND ODD FUNCTIONS
        7. B.7 SIGNUM FUNCTION
        8. B.8 DIRAC DELTA FUNCTION
        9. B.9 KRONECKER DELTA FUNCTION
        10. B.10 UNIT-STEP FUNCTIONS
        11. B.11 RECTANGLE FUNCTIONS
        12. B.12 TRIANGLE AND RAMP FUNCTIONS
        13. B.13 INDICATOR FUNCTIONS
        14. B.14 SINC FUNCTION
        15. B.15 LOGARITHM FUNCTIONS
        16. B.16 GAMMA FUNCTIONS
        17. B.17 BETA FUNCTIONS
        18. B.18 BESSEL FUNCTIONS
        19. B.19 Q-FUNCTION AND ERROR FUNCTIONS
        20. B.20 MARCUM Q-FUNCTION
        21. B.21 ZETA FUNCTION
        22. B.22 RISING AND FALLING FACTORIALS
        23. B.23 LAGUERRE POLYNOMIALS
        24. B.24 HYPERGEOMETRIC FUNCTIONS
        25. B.25 BERNOULLI NUMBERS
        26. B.26 HARMONIC NUMBERS
        27. B.27 EULER–MASCHERONI CONSTANT
        28. B.28 DIRICHLET FUNCTION
        29. FURTHER READING
      3. Appendix C: Frequency-Domain Transforms and Properties
        1. C.1 LAPLACE TRANSFORM
        2. C.2 CONTINUOUS-TIME FOURIER TRANSFORM
        3. C.3 z-TRANSFORM
        4. C.4 DISCRETE-TIME FOURIER TRANSFORM
        5. FURTHER READING
      4. Appendix D: Integration and Integrals
        1. D.1 REVIEW OF RIEMANN INTEGRAL
        2. D.2 RIEMANN–STIELTJES INTEGRAL
        3. D.3 LEBESGUE INTEGRAL
        4. D.4 PDF INTEGRALS
        5. D.5 INDEFINITE AND DEFINITE INTEGRALS
        6. D.6 INTEGRAL FORMULAS
        7. D.7 DOUBLE INTEGRALS OF SPECIAL FUNCTIONS
        8. FURTHER READING
      5. Appendix E: Identities and Infinite Series
        1. E.1 ZERO AND INFINITY
        2. E.2 MINIMUM AND MAXIMUM
        3. E.3 TRIGONOMETRIC IDENTITIES
        4. E.4 STIRLING’S FORMULA
        5. E.5 TAYLOR SERIES
        6. E.6 SERIES EXPANSIONS AND CLOSED-FORM SUMS
        7. E.7 VANDERMONDE'S IDENTITY
        8. E.8 PMF SUMS AND FUNCTIONAL FORMS
        9. E.9 COMPLETING THE SQUARE
        10. E.10 SUMMATION BY PARTS
        11. FURTHER READING
      6. Appendix F: Inequalities and Bounds for Expectations
        1. F.1 CAUCHY–SCHWARZ AND HÖLDER INEQUALITIES
        2. F.2 TRIANGLE AND MINKOWSKI INEQUALITIES
        3. F.3 BIENAYMÉ, CHEBYSHEV, AND MARKOV INEQUALITIES
        4. F.4 CHERNOFF'S INEQUALITY
        5. F.5 JENSEN'S INEQUALITY
        6. F.6 CRAMÉR–RAO INEQUALITY
        7. FURTHER READING
      7. Appendix G: Matrix and Vector Properties
        1. G.1 BASIC PROPERTIES
        2. G.2 FOUR FUNDAMENTAL SUBSPACES
        3. G.3 EIGENDECOMPOSITION
        4. G.4 LU, LDU, AND CHOLESKY DECOMPOSITIONS
        5. G.5 JACOBIAN MATRIX AND THE JACOBIAN
        6. G.6 KRONECKER AND SCHUR PRODUCTS
        7. G.7 PROPERTIES OF TRACE AND DETERMINANT
        8. G.8 MATRIX INVERSION LEMMA
        9. G.9 CAUCHY–SCHWARZ INEQUALITY
        10. G.10 DIFFERENTIATION
        11. G.11 COMPLEX DIFFERENTIATION
        12. FURTHER READING
    11. Glossary
      1. SUMMARY OF NOTATION
      2. GENERAL SYMBOLS
      3. GREEK SYMBOLS
      4. CALLIGRAPHIC SYMBOLS
      5. MATHEMATICAL SYMBOLS
      6. ABBREVIATIONS
    12. References
    13. Index

    Product information

    • Title: Probability, Random Variables, and Random Processes: Theory and Signal Processing Applications
    • Author(s): John J. Shynk
    • Release date: November 2012
    • Publisher(s): Wiley-Interscience
    • ISBN: 9780470242094