Creating Augmented and Virtual Realities

Creating Augmented and Virtual Realities

by Erin Pangilinan, Steve Lukas, Vasanth Mohan
Released April 2019
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781492044192

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

Despite popular forays into augmented and virtual reality in recent years, spatial computing still sits on the cusp of mainstream use. Developers, artists, and designers looking to enter this field today have few places to turn for expert guidance. In this book, Erin Pangilinan, Steve Lukas, and Vasanth Mohan examine the AR and VR development pipeline and provide hands-on practice to help you hone your skills.

Through step-by-step tutorials, you’ll learn how to build practical applications and experiences grounded in theory and backed by industry use cases. In each section of the book, industry specialists, including Timoni West, Victor Prisacariu, and Nicolas Meuleau, join the authors to explain the technology behind spatial computing.

In three parts, this book covers:

  • Art and design: Explore spatial computing and design interactions, human-centered interaction and sensory design, and content creation tools for digital art
  • Technical development: Examine differences between ARKit, ARCore, and spatial mapping-based systems; learn approaches to cross-platform development on head-mounted displays
  • Use cases: Learn how data and machine learning visualization and AI work in spatial computing, training, sports, health, and other enterprise applications

Publisher resources

View/Submit Errata

Table of contents

  1. Foreword
  2. Preface
    1. A Note on Terminology
    2. Why We Wrote This Book
    3. What We Didn’t Cover in This Book
    4. How This Book Is Organized
    5. Conventions Used in This Book
    6. Using Code Examples
    7. O’Reilly Online Learning
    8. How to Contact Us
    9. Acknowledgments
  3. I. Design and Art Across Digital Realities
  4. 1. How Humans Interact with Computers
    1. Common Term Definition
    2. Introduction
    3. Modalities Through the Ages: Pre-Twentieth Century
    4. Modalities Through the Ages: Through World War II
    5. Modalities Through the Ages: Post-World War II
    6. Modalities Through the Ages: The Rise of Personal Computing
    7. Modalities Through the Ages: Computer Miniaturization
    8. Why Did We Just Go Over All of This?
      1. Types of Common HCI Modalities
    9. New Modalities
    10. The Current State of Modalities for Spatial Computing Devices
    11. Current Controllers for Immersive Computing Systems
      1. Body Tracking Technologies
    12. A Note on Hand Tracking and Hand Pose Recognition
      1. Voice, Hands, and Hardware Inputs over the Next Generation
  5. 2. Designing for Our Senses, Not Our Devices
    1. Envisioning a Future
    2. Sensory Technology Explained
    3. So, Who Are We Building This Future For?
      1. The Role of Women in AI
    4. Sensory Design
      1. An Introduction
    5. Five Sensory Principles
      1. 1. Intuitive Experiences Are Multisensory
      2. 2. 3D Will Be Normcore
      3. 3. Designs Become Physical Nature
      4. 4. Design for the Uncontrollable
      5. 5. Unlock the Power of Spatial Collaboration
    6. Adobe’s AR Story
    7. Conclusion
  6. II. How eXtended Reality Is Changing Digital Art
  7. 3. Virtual Reality for Art
    1. A More Natural Way of Making 3D Art
    2. VR for Animation
  8. 4. 3D Art Optimization
    1. Introduction
      1. Options to Consider
      2. Ideal Solution
      3. Topology
      4. Baking
    2. Draw Calls
    3. Using VR Tools for Creating 3D Art
    4. Acquiring 3D Models Versus Making Them from Scratch
    5. Summary
  9. III. Hardware, SLAM, Tracking
  10. 5. How the Computer Vision That Makes Augmented Reality Possible Works
    1. Who Are We?
    2. A Brief History of AR
    3. How and Why to Select an AR Platform
      1. I’m a Developer, What Platform Should I Use and Why?
      2. Performance Is Statistics
      3. Integrating Hardware and Software
      4. Optical Calibration
      5. Inertial Calibration
      6. The Future of Tracking
      7. The Future of AR Computer Vision
    4. Mapping
      1. How Does Multiplayer AR Work?
      2. What’s the Difficult Part?
      3. How Does Relocalization Work?
      4. What’s the State of the Art in Research (and Coming Soon to Consumer)?
      5. Can the Relocalization Problem Really Be Solved for Consumers?
      6. Can’t Google or Apple Just Do This?
      7. Relocalization != Multiplayer; It’s Also Critical for…
      8. How Is Relocalization Really Being Done Today in Apps?
    5. Platforms
      1. Apple’s ARKit
      2. Some Mysteries Explained
      3. Isn’t ARCore Just Tango-Lite?
      4. So, Should I Build on ARCore Now?
      5. What About Tango, Hololens, Vuforia, and Others?
    6. Other Development Considerations
      1. Lighting
      2. Multiplayer AR—Why It’s Quite Difficult
      3. How Do People Connect Through AR?
      4. How Do AR Apps Connect to the World and Know Where They Really Are?
      5. How Do AR Apps Understand and Connect to Things in the Real World?
    7. The AR Cloud
      1. What I Envision When I Think About the AR Cloud
      2. How Big a Deal Is the AR Cloud?
      3. So, Just What Is This AR Cloud?
      4. Why All the “Meh” from the Press for ARKit and ARCore?
      5. What’s Missing to Make a Great AR App?
      6. Is Today’s Mobile Cloud up to the Job?
      7. Is ARKit (or ARCore) Useless Without the AR Cloud?
      8. The Dawn of the AR Cloud
      9. The Bigger Picture—Privacy and AR Cloud Data
    8. Glossary
  11. IV. Creating Cross-Platform Augmented Reality and Virtual Reality
  12. 6. Virtual Reality and Augmented Reality: Cross-Platform Theory
    1. Why Cross-Platform?
    2. The Role of Game Engines
    3. Understanding 3D Graphics
      1. The Virtual Camera
      2. Degrees of Freedom
    4. Portability Lessons from Video Game Design
    5. Simplifying the Controller Input
      1. Development Step 1: Designing the Base Interface
      2. Development Step 2: Platform Integration
    6. Summary
  13. 7. Virtual Reality Toolkit: Open Source Framework for the Community
    1. What Is VRTK and Why People Use It?
    2. The History of VRTK
    3. Welcome to the SteamVR Unity Toolkit
    4. VRTK v4
    5. The Future of VRTK
    6. The Success of VRTK
      1. Getting Started with VRTK 4
  14. 8. Three Virtual Reality and Augmented Reality Development Best Practices
    1. Developing for Virtual Reality and Augmented Reality Is Difficult
    2. Handling Locomotion
      1. Locomotion in VR
      2. Locomotion in AR
    3. Effective Use of Audio
      1. Audio in VR
      2. Audio in AR
    4. Common Interactions Paradigms
      1. Inventory for VR
      2. Augmented Reality Raycasts
    5. Conclusion
  15. V. Enhancing Data Representation: Data Visualization and Artificial Intelligence in Spatial Computing
  16. 9. Data and Machine Learning Visualization Design and Development in Spatial Computing
    1. Introduction
    2. Understanding Data Visualization
    3. Principles for Data and Machine Learning Visualization in Spatial Computing
    4. Why Data and Machine Learning Visualization Works in Spatial Computing
      1. The Evolution of Data Visualization Design with the Emergence of XR
      2. 2D and 3D Data Represented in XR
    5. 2D Data Visualizations versus 3D Data Visualization in Spatial Computing
    6. Interactivity in Data Visualizations in Spatial Computing
    7. Animation
    8. Failures in Data Visualization Design
    9. Good Data Visualization Design Optimize 3D Spaces
    10. “Savings in Time Feel Like Simplicity”
    11. Data Representations, Infographics, and Interactions
      1. What Qualifies as Data Visualization?
    12. Defining Distinctions in Data Visualization and Big Data or Machine Learning Visualizations
    13. How to Create Data Visualization: Data Visualization Creation Pipeline
    14. WebXR: Building Data Visualizations for the Web
    15. Data Visualization Challenges in XR
    16. Data Visualization Industry Use Case Examples of Data Visualizations
    17. 3D Reconstruction and Direct Manipulation of Real-World Data: Anatomical Structures in XR
      1. A Closer Look at Glass Brain
      2. TVA Surg Medical Imaging VR Module
    18. Data Visualization Is for Everyone: Open Source–Based Data Visualization in XR
      1. Protein Data Visualization
    19. Hands-On Tutorials: How to Create Data Visualization in Spatial Computing
    20. How to Create Data Visualization: Resources
    21. Conclusion
    22. References
    23. Resources
      1. Data Visualization Tools
      2. Machine Learning Visualization Tools
      3. Data Journalism Visualizations
    24. Glossary
  17. 10. Character AI and Behaviors
    1. Introduction
    2. Behaviors
    3. Current Practice: Reactive AI
      1. Adaptability
      2. Complexity and Universality
      3. Feasibility
    4. More Intelligence in the System: Deliberative AI
    5. Machine Learning
      1. Reinforcement Learning
      2. Deep Reinforcement Learning
      3. Imitation Learning
      4. Combining Automated Planning and Machine Learning
      5. Applications
    6. Conclusion
    7. References
  18. VI. Use Cases in Embodied Reality
  19. 11. The Virtual and Augmented Reality Health Technology Ecosystem
    1. VR/AR Health Technology Application Design
    2. Standard UX Isn’t Intuitive
      1. Pick a Calm Environment
      2. Convenience
    3. Tutorial: Insight Parkinson’s Experiment
      1. What Insight Does
      2. How It Was Built
    4. Companies
      1. Planning and Guidance
      2. Experiences Designed for Medical Education
      3. Experiences Designed for Use by Patients
      4. Proactive Health
    5. Case Studies from Leading Academic Institutions
  20. 12. The Fan Experience: SportsXR
    1. Introduction
    2. Part 1: Five Key Principles of AR and VR for Sports
      1. Nothing Is Live
    3. Part 2: The Next Evolution of Sports Experiences
    4. Part 3: Making the Future
      1. Ownership
      2. Final Thought
    5. Conclusion
  21. 13. Virtual Reality Enterprise Training Use Cases
    1. Introduction: The Importance of Enterprise Training
    2. Does VR Training Work?
    3. Use Case: Flood House Training
    4. What Is VR Training Good for? R.I.D.E.
    5. What Makes Good VR Training?
    6. Spherical Video
      1. The Benefits of Spherical Video
      2. The Challenges of Spherical Video
      3. Interactions with Spherical Video
    7. Use Case: Factory Floor Training
    8. The Role of Narrative
    9. Use Case: Store Robbery Training
    10. The Future of XR Training: Beyond Spherical Video
      1. Computer Graphics
    11. Use Case: Soft Skills Training
    12. The Future: Photogrammetry
    13. The Future: Light Fields
    14. The Future: AR Training
    15. The Future: Voice Recognition
    16. The Future: The Ideal Training Scenario
    17. References
  22. Afterword
  23. Index

Product information

  • Title: Creating Augmented and Virtual Realities
  • Author(s): Erin Pangilinan, Steve Lukas, Vasanth Mohan
  • Release date: April 2019
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492044192