Look before you model

Before building any model, you explore the data. You cannot model what you do not understand.

The first time you Run code here, your browser downloads Python and the data science libraries (numpy, pandas, scikit-learn). This is a large one time download, around thirty megabytes, then it is cached and every run after is fast, even offline. Be patient on that first run.

We explore data with pandas, the main data tool in Python, usually imported as pd. Its key object is the DataFrame, a table of rows and named columns, much like a sheet in Excel or a table in a database.

In real projects you load data from a file with pd.read_csv("yourfile.csv"). Here we use a small made-up sample so it runs instantly in your browser. You can explore real African datasets on the RiseAfrica datasets page. We will use this one student dataset throughout the course.

Reading what describe() tells you

describe() returns eight numbers for every numeric column. Learning to read them is a real skill:

• count: how many rows actually have a value. If this is lower than the other columns, that column has missing data, which you will deal with later.
• mean: the average value.
• std: the standard deviation, a measure of how spread out the values are. A large std means the values vary widely.
• min: the smallest value.
• 25%: if you sorted the column from low to high, this is the value a quarter of the way up. A quarter of the rows fall below it. It is called the 25th percentile.
• 50%: the middle value, also called the median. Half the rows fall below it.
• 75%: three quarters of the way up, the 75th percentile.
• max: the largest value.

Read together, these tell you a lot before you model. If the mean sits far from the median (the 50%), your data is lopsided. If the min or max look extreme, you may have errors or outliers worth checking. Always read this summary before building anything.

Exercise