Statistics can be our friend!

Subscribe to my newsletter and never miss my upcoming articles

Statistics can be our friend, but I have struggled to make statistics meaningful or rather to use statistics. In this short article, I will illustrate some uses of statistics in the Internet of Things and streaming measurements from Environment sensors. To introduce the techniques, I have to provide some insight into the capture device and then move onto the actual statistics. So let us start with Data Capture and then move to the Analytics.

Data Capture

My YouTube channel contains an entire video serious on this topic. The image below will give you a broad perspective of the data capture I am using.

IMG_20200728_110318-COLLAGE(1).jpg

My home environment is instrumented, and I am measuring things like Room Temperature, Room Pressure, Room Humidity, Gases present, Particulates in the Air, movement, and noise level. With data streaming in, every 20minutes, for months, we can turn to statistics to derive meaning and understanding from the measurements.

Statistics can be our friend

Statistics can be our friend, but we have to know how to use them, and it has to make sense. The first step is to fetch the data, loading the data into a rectangular shape, and then perform the various tests and measures.

Fetching

Since I mentioned that I use MongoDB, I can just run a Python script to retrieve that data.

cursor = db.readings.find({})
docs = []
for document in cursor:
    docs.append(document)

The code produces a JSON object - In Python talk that is a list of dictionaries.

Screenshot from 2020-08-23 14-00-35.png

Next, we want to convert that JSON object into a Data Frame.

Loading

Naturally, we use the Pandas library and can easily confirm the JSON structure into a Data Frame.

Screenshot from 2020-08-23 14-05-37.png

Even using .info() gives us a good view of the quality of our data, the features involved, and the size of the set. We certainly have missing values as all the variables don't have the same amount of observations.

Statistics

With Statistics, I try to use Visual charts as opposed to pure metrics. Perhaps I am more visually orientated, but graphs make more sense to me than numbers on a page. Let's look at a Box Plot to see where we are. We might need to do some cleaning!

Screenshot from 2020-08-23 14-09-29.png Wow! We not only have missing values, from above, but we have some extreme values. Those extreme values we call outliers. -10 Degrees Centigrade would be unheard of for internal temperatures, and I believe 40 degrees internally would be a heat stroke event. These are not correct readings and might represent inaccurate sensor readings.

First, let us fix the missing values for Temperature. Setting missing values we call Imputation, and that is an entire body of knowledge. For the purpose of illustration, let us just fill in all missing values with a 0. Not a great idea, but it will be illustrative. A better way would be to impute missing values from readings in proximity to the missing measurement. Let us say we are missing an entry at 12 noon but have an observation for 11 am and for 1 pm. So the 12 Noon reading can be the average of the neighbours.

Hello Pandas and .fillna(0)

Screenshot from 2020-08-23 14-16-43.png

After the .fillna(0) command, we can see that the Room Temperature feature no longer has any missing values. You can see through looking at the variable '_id' and comparing that count with the Room Temperature count.

Now re-draw the Box Plot and take a look at the dramatic changes in the Visual.

Screenshot from 2020-08-23 14-17-05.png

Our median has shifted from 22 to 19 using a dirty imputation strategy of just filling in the missing data points with zero. So you need to be extremely careful how you deal with missing values, or you will get a completely different answer.

Screenshot from 2020-08-23 14-25-59.png

More old favourites - the mean, median, max and minimum are showing the estimates of location. Suddenly we are talking like a statistician. But steady on!

We have a long way to go as we are only getting started. Statistics can be your friend, but you need to be careful and do the work.

No Comments Yet