Data Wrangling with SQL

Data Wrangling with SQL

by Raghav Kandarpa, Shivangi Saxena
Released July 2023
Publisher(s): Packt Publishing
ISBN: 9781837630028

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Become a data wrangling expert and make well-informed decisions by effectively utilizing and analyzing raw unstructured data in a systematic manner Purchase of the print or Kindle book includes a free PDF eBook

Key Features

  • Implement query optimization during data wrangling using the SQL language with practical use cases
  • Master data cleaning, handle the date function and null value, and write subqueries and window functions
  • Practice self-assessment questions for SQL-based interviews and real-world case study rounds

Book Description

The amount of data generated continues to grow rapidly, making it increasingly important for businesses to be able to wrangle this data and understand it quickly and efficiently. Although data wrangling can be challenging, with the right tools and techniques you can efficiently handle enormous amounts of unstructured data.

The book starts by introducing you to the basics of SQL, focusing on the core principles and techniques of data wrangling. You’ll then explore advanced SQL concepts like aggregate functions, window functions, CTEs, and subqueries that are very popular in the business world. The next set of chapters will walk you through different functions within SQL query that cause delays in data transformation and help you figure out the difference between a good query and bad one. You’ll also learn how data wrangling and data science go hand in hand. The book is filled with datasets and practical examples to help you understand the concepts thoroughly, along with best practices to guide you at every stage of data wrangling.

By the end of this book, you’ll be equipped with essential techniques and best practices for data wrangling, and will predominantly learn how to use clean and standardized data models to make informed decisions, helping businesses avoid costly mistakes.

What you will learn

  • Build time series models using data wrangling
  • Discover data wrangling best practices as well as tips and tricks
  • Find out how to use subqueries, window functions, CTEs, and aggregate functions
  • Handle missing data, data types, date formats, and redundant data
  • Build clean and efficient data models using data wrangling techniques
  • Remove outliers and calculate standard deviation to gauge the skewness of data

Who this book is for

This book is for data analysts looking for effective hands-on methods to manage and analyze large volumes of data using SQL. The book will also benefit data scientists, product managers, and basically any role wherein you are expected to gather data insights and develop business strategies using SQL as a language. If you are new to or have basic knowledge of SQL and databases and an understanding of data cleaning practices, this book will give you further insights into how you can apply SQL concepts to build clean, standardized data models for accurate analysis.

Table of contents

  1. Data Wrangling with SQL
  2. Acknowledgements
  3. Dedication
  4. Contributors
  5. About the authors
  6. About the reviewers
  7. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Conventions used
    6. Get in touch
    7. Share Your Thoughts
    8. Download a free PDF copy of this book
  8. Part 1:Data Wrangling Introduction
  9. Chapter 1: Database Introduction
    1. Getting started
      1. Establishing the foundation
      2. Efficient data organization
      3. Data integrity and consistency
    2. Technical requirements
    3. Decoding database structures – relational and non-relational
      1. What is a database?
      2. Types of databases
    4. Tables and relationships
      1. The SQL CREATE DATABASE statement
      2. The SQL CREATE TABLE statement
      3. SQL DROP TABLE versus TRUNCATE TABLE
      4. SQL ALTER TABLE
      5. SQL constraints
      6. SQL keys
      7. Database relationships
    5. Comparing database normalization and denormalization
      1. Normalization
      2. Types of normalization
      3. Denormalization
      4. When to apply denormalization
      5. Disadvantages of denormalization
    6. Summary
    7. Practical exercises
      1. Practical exercise 1
      2. Practical exercise 2
      3. Practical exercise 3
      4. Practical exercise 4
  10. Chapter 2: Data Profiling and Preparation before Data Wrangling
    1. What is data wrangling?
      1. Data wrangling steps
      2. The importance of data wrangling
      3. Benefits of data wrangling
      4. Data wrangling use cases
      5. Business use cases
    2. Data capture
      1. How does data get captured?
      2. Data-capturing techniques
      3. Web scraping
      4. Structured versus unstructured data
    3. Paid-for versus free data-wrangling tools
    4. Data profiling
      1. Data profiling types
      2. Data profiling techniques
    5. Practical exercise
      1. Step 1 – Discovery
      2. Step 2 – Structuring
      3. Step 3 – Cleaning
      4. Step 4 – Enriching
      5. Step 5 – Validating
      6. Step 6 – Publishing
    6. Summary
  11. Part 2:Data Wrangling Techniques Using SQL
  12. Chapter 3: Data Wrangling on String Data Types
    1. SQL data types
      1. Numeric data types
      2. Date and time data types
      3. String data type
    2. SQL string functions
      1. RIGHT()
      2. LEFT()
      3. LEN()
      4. TRIM()
      5. RTRIM()
      6. LTRIM()
      7. RPAD()
      8. LPAD()
      9. REPLACE()
      10. REVERSE()
      11. SUBSTRING()
      12. CAST()
      13. CONCATENATE()
      14. CONCATENATE_WS()
      15. UPPER function
      16. LOWER function
      17. INITCAP function
      18. INSTR function
    3. Summary
    4. Practical exercises
      1. Practical exercise 1
      2. Practical exercise 2
      3. Practical exercise 3
      4. Practical exercise 4
  13. Chapter 4: Data Wrangling on the DATE Data Type
    1. SQL DATE data type functions
      1. EXTRACT
      2. DATEDIFF()
      3. TIMEDIFF()
      4. DATE_ADD()
      5. DATE_SUB()
      6. DATE_FORMAT()
      7. STR_TO_DATE()
      8. Extracting the current date and time
    2. Summary
  14. Chapter 5: Handling NULL Values
    1. The impact of missing data and NULL values on data analysis
    2. Understanding the importance of data validation and cleaning before analyzing data
    3. Identifying NULL/missing values
    4. NULL values versus zero values
    5. Using the IS NULL and IS NOT NULL operators to filter and select data with NULL values
      1. IS NULL() and IS NOT NULL() – scenario
    6. Using the COALESCE and IFNULL functions to replace NULL values with a default value
      1. IFNULL()
      2. COALESCE()
      3. IS NULL versus = NULL
    7. Summary
  15. Chapter 6: Pivoting Data Using SQL
    1. SQL Transpose – rows to columns
      1. Use case scenario
    2. SQL Cross Tab – columns to rows
      1. Use case scenario
      2. Unpivoting data in SQL
      3. Analytical workflow – from SQL to business intelligence – transforming data into actionable insights
    3. Summary
  16. Part 3:SQL Subqueries, Aggregate And Window Functions
  17. Chapter 7: Subqueries and CTEs
    1. Introduction to subqueries
      1. Simple subqueries
      2. Correlated subqueries
      3. Using subqueries in SELECT statements
      4. Using subqueries in FROM statements
      5. Using subqueries in WHERE statements
      6. Nested subqueries
      7. Correlated subqueries
      8. Using subqueries in INSERT, UPDATE, and DELETE statements
    2. Managing and maintaining subqueries
      1. Common table expressions
      2. Performance considerations for subqueries and CTEs
      3. Subquery versus CTEs
    3. Summary
  18. Chapter 8: Aggregate Functions
    1. Overview of aggregate functions in SQL
      1. Using GROUP BY
      2. COUNT()
      3. SUM()
      4. AVG()
      5. MIN() and MAX()
      6. COUNT(DISTINCT)
      7. Case scenario – using all aggregate functions
    2. Summary
  19. Chapter 9: SQL Window Functions
    1. The importance of SQL window functions
    2. SQL aggregate functions
    3. SQL window functions versus aggregate functions
      1. Window functions versus aggregate functions – an example to illustrate the differences
    4. Window functions
      1. SUM()
      2. COUNT()
      3. AVG()
      4. ROW_NUMBER()
      5. RANK() and DENSE_RANK()
      6. Lead() and Lag()
      7. NTILE()
    5. Summary
  20. Part 4:Optimizing Query Performance
  21. Chapter 10: Optimizing Query Performance
    1. Introduction to query optimization
    2. Query execution plan
    3. Query optimization techniques
      1. Example
      2. Caching
      3. Normalization
    4. Query monitoring and troubleshooting
      1. Query profiling
      2. Query logging
      3. Database monitoring
    5. Tips and tricks for writing efficient queries
    6. Summary
    7. In the next chapter, we will learn about descriptive statistics using SQL, which will provide us with insights into the distribution, central tendency, and variability of data, which can, in turn, help us identify outliers and anomalies. Common SQL functions and statements used for descriptive statistics include COUNT, AVG, MIN, MAX, and GROUP BY. By using SQL to analyze data, researchers and analysts can efficiently extract and summarize information from large datasets.
  22. Part 5:Data Science And Wrangling
  23. Chapter 11: Descriptive Statistics with SQL
    1. Calculating descriptive statistics with SQL
      1. Mean
      2. Median
      3. Mode
      4. Standard deviation
      5. Variance
      6. Variability
    2. Summary
    3. In the next chapter, we will learn how SQL can be used for time series analysis.
  24. Chapter 12: Time Series with SQL
    1. Running totals
      1. Case scenario
    2. Lead and lag for time series analysis
      1. Case scenario
      2. Key KPIs
    3. Percentage change
      1. Case scenario
      2. Key KPIs
    4. Moving averages
      1. Case scenario
      2. Key KPIs
    5. Rank for time series analysis
      1. Case scenario
      2. Key KPIs
    6. CTE for time series analysis
      1. Importance of using CTEs while performing time series analysis
    7. Forecasting with linear regression
      1. Case scenario
      2. Key KPIs
    8. Summary
    9. In the next chapter, we will learn different methods to find outliers in the data easily. Outlier detection is an important aspect of data analysis as it helps determine if the data is correct, looks at the skewness of the data, and removes any unexpected values.
  25. Chapter 13: Outlier Detection
    1. Measures of central tendency and dispersion
      1. Case scenario
      2. Key KPIs
    2. Methods for detecting outliers
      1. Box plot method
    3. Handling outliers
      1. Case scenario
      2. Key points to keep in mind while handling outliers
    4. Applying outlier detection
      1. Challenges and limitations
      2. Best practices
    5. Summary
  26. Index
    1. Why subscribe?
  27. Other Books You May Enjoy
    1. Packt is searching for authors like you
    2. Share Your Thoughts
    3. Download a free PDF copy of this book

Product information

  • Title: Data Wrangling with SQL
  • Author(s): Raghav Kandarpa, Shivangi Saxena
  • Release date: July 2023
  • Publisher(s): Packt Publishing
  • ISBN: 9781837630028