Learn to effectively manage data and execute data science projects from start to finish using Python.

Preface

“BeĴer than any statistician at computer science and beĴer at statistics than any computer scientist” – this is a phrase I’ve heard said about data scientists since I started my official data science training. It might be true, but data science has grown to incorporate so many different fields and technologies that it might not be able to be captured with such a simple statement anymore. Not to mention that statistics, and especially computer science, cover a lot of ground, too. But as a quick-and-dirty way to describe data science in three words, “statistics + computer science” works.

Many people learn data science to improve their lives. For me, I wanted to transition out of the physical sciences, which are bound by physical locations, and have more freedom to travel around the world. Working in a digital space like data science allows for that, while high-tech manufacturing doesn’t. For others, the increase in pay is alluring. For many of us, we see the stories about data scientists being happy and highly paid and are immediately interested in learning more. Some people learn data science due to their intellectual curiosity and the fun of it. In any case, if you want to be a data scientist, you’d beĴer enjoy working with computers and data!

I wrote this book for a few reasons, and one good reason to create teaching materials or even teach courses is you will learn the materials beĴer by teaching it. So, one thing I’d recommend doing if you want to really learn is to create some teaching materials. An easy way to do this is to write a blog post about using data science tosolve a problem. It could be any dataset from Kaggle, for example, or some data you’ve got access to and are allowed to share.

In the book, we use Python to carry out data science. However, there are a plethora of tools for doing data science, so don’t feel like Python is the only way. There is a debate among data scientists whether or not a data scientist must be able to program. On the one hand, being able to code enables us to use cuĴing-edge tools and integrate into other software products more easily.

On the other hand, not all data science work is the same, and some doesn’t have to be done with code. Many people doing data science use R and other tools (such as GUIs) to carry out their work. However, Python seems to be the top choice and integrates nicely
into software stacks at companies. Python, like any other skill, requires practice and dedication to master. This book is here to get you started, and I hope you have fun learning Python and data science and are excited to continue your data science journey well beyond this book.

Key Features

• Understand and utilize data science tools in Python, such as specialized machine learning algorithms and statistical modeling
• Build a strong data science foundation with the best data science tools available in Python
• Add value to yourself, your organization, and society by extracting actionable insights from raw data

Book Description

Practical Data Science with Python teaches you core data science concepts, with real-world and realistic examples, and strengthens your grip on the basic as well as advanced principles of data preparation and storage, statistics, probability theory, machine learning, and Python programming, helping you build a solid foundation to gain proficiency in data science.

The book starts with an overview of basic Python skills and then introduces foundational data science techniques, followed by a thorough explanation of the Python code needed to execute the techniques. You’ll understand the code by working through the examples. The code has been broken down into small chunks (a few lines or a function at a time) to enable thorough discussion.

As you progress, you will learn how to perform data analysis while exploring the functionalities of key data science Python packages, including pandas, SciPy, and scikit-learn. Finally, the book covers ethics and privacy concerns in data science and suggests resources for improving data science skills, as well as ways to stay up to date on new data science developments.

By the end of the book, you should be able to comfortably use Python for basic data science projects and should have the skills to execute the data science process on any data source.

What you will learn

• Use Python data science packages effectively
• Clean and prepare data for data science work, including feature engineering and feature selection
• Data modeling, including classic statistical models (such as t-tests), and essential machine learning algorithms, such as random forests and boosted models
• Evaluate model performance
• Compare and understand different machine learning methods
• Interact with Excel spreadsheets through Python
• Create automated data science reports through Python
• Get to grips with text analytics techniques

Who this book is for

The book is intended for beginners, including students starting or about to start a data science, analytics, or related program (e.g. Bachelor’s, Master’s, bootcamp, online courses), recent college graduates who want to learn new skills to set them apart in the job market, professionals who want to learn hands-on data science techniques in Python, and those who want to shift their career to data science.

The book requires basic familiarity with Python. A “getting started with Python” section has been included to get complete novices up to speed.

---

What this book covers

Part I, An Introduction and the Basics

Chapter 1, Introduction to Data Science, gives an overview of data science, including the history, top skills and tools used in the field, specializations and related fields, and best practices for data science projects.

Chapter 2, GeĴing Started with Python, explains installing Python and Python distributions (specifically, Anaconda), editing and running code with code editors, IPython, Jupyter Notebooks, basic use of the command line, installing Python packages and using virtual environments, Python programming basics, how to deal with errors and use documentation, and software engineering best practices (including Git and GitHub).

Part II, Dealing with Data

Chapter 3, SQL and Built-In File Handling Modules in Python, covers loading data from ordinary text files using built-in Python capabilities, using Python’s built-in sqlite3 module for databases, basic SQL commands, and the SQLAlchemy package in Python.

Chapter 4, Loading and Wrangling Data with Pandas and NumPy, explains how to use the pandas and NumPy packages in Python. Using pandas, we learn how to load and save data with several different data source types (CSV, Excel files, and so on), how to carry out some basic exploratory data analysis (EDA), how to prepare and clean data for later use, and some essential data wrangling tools withpandas and NumPy. We also learn how pandas uses NumPy and a few NumPy basics.

Chapter 5, Exploratory Data Analysis and Visualization, covers EDA and visualization packages in Python, such as pandas-profiling, seaborn, plotly, and more. We also cover visualization best practices.

Chapter 6, Data Wrangling Documents and Spreadsheets, shows how to use Python packages to load data from Microsoft Word and PDF documents, along with some basic preparation, cleaning, and analysis of text data. We also cover reading, writing, and extracting data from Microsoft Excel files.

Chapter 7, Web Scraping, demonstrates the basics of web scraping with base Python and Python packages. We learn about the basic structure of the internet and web pages, and how to parse web pages. Use of web application programming interfaces (APIs) is also covered. Finally, we wrap up with the ethics and legality of web scraping.

Part III, Statistics for Data Science

Chapter 8, Probability, Distributions, and Sampling, explains foundational probability concepts, common probability distributions in data science, and useful sampling techniques for data science.

Chapter 9, Statistical Testing for Data Science, covers some useful statistical tests, such as t- and z-tests, ANOVA and post-hoc tests, distribution testing, outlier testing, and testing for relationships between variables.

Part IV, Machine Learning

Chapter 10, Preparing Data for Machine Learning: Feature Selection, Feature Engineering, and Dimensionality Reduction, explains feature selection methods, including univariate statistical methods, such as correlation, mutual information score, chi-squared, and other feature selection methods. We also cover feature engineering methods for categorical, datetime, and outlier data. Mathematical transformations for feature transformation such as Yeo-Johnson are covered as well. Finally, dimensionality reduction using principal component analysis (PCA) is covered, and other options for dimensionality reduction are presented.

Chapter 11, Machine Learning for Classification, covers using Python for machine learning classification algorithms, including binary, multiclass, and multi-label classification. The algorithms covered include logistic regression, Naïve Bayes, and k-nearest neighbors (KNN).

Chapter 12, Evaluating Machine Learning Classification Models and Sampling for Classification, is about performance metrics for classification, such as accuracy, Cohen’s Kappa, confusion matrices, and more. We also cover sampling imbalanced data to improve machine learning classification performance.

Chapter 13, Machine Learning with Regression, covers implementing and interpreting linear regression with the scikit-learn and statsmodels Python packages, as well as regularization of linear regression models. KNN and other models are also covered. Evaluation of regression models with metrics such as the coefficient of determination (R2) and information criteria (such as the Akaike information criterion, AIC) is covered as well.

Chapter 14, Optimizing Models and Using AutoML, demonstrates hyperparameter optimization for ML models using random, grid,and Bayesian searches. The various packages in Python for optimizing models are discussed. We learn how to use learning curves to optimize the amount of data for an ML model. Optimizing the number of features using recursive feature selection is covered. Finally, we cover some different options for AutoML in Python and learn how to use the pycaret AutoML package.

Chapter 15, Tree-Based Machine Learning Models, explains how trees work in ML algorithms, and we learn how to use some of the most advanced tree-based ML models, including random forests, XGBoost, LightGBM, and CatBoost. We also cover feature importances from tree-based methods.

Chapter 16, Support Vector Machine (SVM) Machine Learning Models, covers the basic theory behind SVMs and how to use them for classification and regression in Python as well as tuning SVM hyperparameters.

Part V, Text Analysis and Reporting

Chapter 17, Clustering with Machine Learning, explains the theory and use of some common clustering algorithms for unsupervised learning: k-means clustering, DBSCAN, and hierarchical clustering. We also look at what other options exist for clustering.

Chapter 18, Working with Text, covers the basics of text analysis and natural language processing (NLP). We start with pre-processing and cleaning of text, then cover basic analysis and statistical methods for text. Then we cover unsupervised learning for text, including topic modeling. We also cover supervised learning for classification with text, and finally, sentiment analysis.

Part VI, Wrapping Up

Chapter 19, Data Storytelling and Automated Reporting/Dashboarding, explains how to string together our analysis and data into a captivating story, and best practices for communicating data and results of data science work. We also learn about dashboarding to display our analysis for monitoring results and how to use the streamlit package in Python to create a dashboard.

Chapter 20, Ethics and Privacy, covers the ethical and privacy concerns in data science, including bias in machine learning algorithms, data privacy concerns in data preparation and analysis, data privacy laws and regulations, and using data science for the common good. We cover k-anonymity, l-diversity, and t-closeness to measure the level of privacy in datasets along with an example.

Chapter 21, Staying Up to Date and the Future of Data Science, discusses ways to keep on top of the ever-changing field of data science, and suggests some resources for staying up to date. We also briefly discuss some of the topics we didn’t cover in the book and talk about where the future of data science might be going.

To get the most out of this book

• Readers should be interested in computing, using Python, and doing data science
• This book is intended for beginners and intermediates to Python and data science, though more advanced practitioners can benefit by reading it as well (for example, to review and maybe learn some new things)
• Readers should have some basic knowledge of how to use a computer and the internet
• Installation of Python is required, but is covered in the book

Download the example code files

The code bundle for the book is hosted on GitHub at https://github.com/PacktPublishing/Practical-DataScience-with-Python. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!