Chapter 14. Automation and Scaling
Youâve scraped large amounts of data from APIs and websites, youâve cleaned and organized your data, and youâve run statistical analysis and produced visual reports. Now itâs time to let Python take the wheel and automate your data wrangling. In this chapter, weâll cover how to automate your data analysis, collection, and publication. We will learn how to create proper logging and alerting so you can fully automate your scripts and get notifications of success, failure, and any issues your work encounters along the way.
We will also take a look at scaling your automation using Python libraries designed to help you execute many tasks and monitor their success and failure. Weâll analyze some libraries and helper tools for fully scaling your data in the cloud.
Python gives us plenty of options for automation and scaling. There are some simple, straightforward tasks that lend themselves to Python automation on almost any machine without much setup, and there are some larger, more complex ways to automate. Weâll cover examples of both, as well as how to scale your data automation as a data wrangler.
Why Automate?
Automation gives you a way to easily run your scripts without needing to do so on your local machineâor even be awake! The ability to automate means you can spend time working on other more thought-intensive projects. If you have a well-written script to perform data cleanup for you, you can focus on working with the data ...
Get Data Wrangling with Python now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.