Switching to AT&T M2X

Around August of last year, the time-series database I was using, TempoIQ, announced that they were not offering their data ingestion and storage plans any more. During the 2 week grace period until their server-shutdown, I looked through other alternatives to TempoIQ, such as Influx Data, RethinkDB, and finally AT&T M2X. I eventually picked AT&T M2X because I was able to write to the database via http as well as https, which is useful for low powered microcontroller platforms. Furthermore, it was the most economical solution offered.

I use the Python client library found here to interact with the AT&T M2X API for my data analysis (pulling from M2X and pushing to plot.ly), and the Arduino client library found here in order to push my data from the Particle Photon to the cloud. It took some time to familiarize myself with the new API, but I found it to be quite intuitive and useful!

The latest version of the PICKLE is comprised of a Plantower PMS5003 laser air quality detection sensor module (14$) and a Particle Photon (the microcontroller with built-in Wifi and a cloud service). As the data (LPO) is fed into the Photon, the microcontroller pushes the data to the M2X database as well as the Particle cloud. The Particle cloud plays an integral role in this setup because it makes it very trivial to manage multiple monitors simultaneously. For example, if I have 20 of these modules deployed in the field and I want to make a change to the program on each of the microcontrollers, rather than painstakingly tracking down each monitor and updating it, the Particle cloud allows you to manage all of the nodes—all through the cloud.

AT&T M2X console: Screen Shot 2017-06-13 at 3.18.53 PM

Particle console: Screen Shot 2017-06-13 at 3.15.40 PM
Another neat feature of AT&T M2X is that you can opt to make your databases public; that is, anyone can view your data without having explicit permission. For example, to view live updates from the monitors in your console (terminal), simply run

curl https://api.particle.io/v1/devices/events?access_token=e31c562d43dd3dd36b71297a0d22ccc579901829

Here is a live stream from one of the monitors in my house (averaged over 10 minutes for the past 24 hours). Note that the time is in GMT.

I am currently interning at the Bay Area Air Quality Management District (BAAQMD) for the summer of 2017; more to come on that later!

Posted in Uncategorized | 1 Comment

Incense Spike

This past weekend I decided to test the latest model of the PICKLE—with the Plantower 5003 and the Particle Electron—by lighting an incense stick near the monitor. As you can tell, the PM2.5 levels jump from roughly 3.0 to about 130, which was quite alarming. However, in the grand scale of things, the air quality when burning the incense stick is still almost 8 times better than that of Beijing and New Dehli.



Posted in Uncategorized | Leave a comment

Circuit Diagram for the Pickle 42R Red

I thought I’d learn Fritzing and post the circuit diagram of the modified Shinyei PPD42NS. The circuit is really quite simple. In the 42R, the Shinyei is connected to the Particle Photon which is a microcontroller with built-in Wifi and a cloud service. The Photon is shown below on the breadboard.

pickle photon ppd42ns_bb

I played with the resistor value quite a bit, until I settled on 10K. In the Shinyei, there are two internal 82K resistors and so adding the external 10K, reduces the threshold to 0.47V

Modified Shinyei ppd42ns_schem

This modified threshold is measured at A4 in the Photon. So if you are reading 0.47V at A4, everything is OK. Else something is off. Reducing R3 to make the sensor more sensitive does not work well, there is a lot of noise.

Hope this helps!

Posted in Uncategorized | 2 Comments

More Pickle Jr. monitors using the PPD42

It’s been a while and school intervened and I did not get to post to the blog. Since then there have been a bunch of questions and so first I want to show a few more Pickle Jr. PM Monitors I have built. I used laser cut acrylic at the local TechShop (love the place!) to make the PPD42 and the Arduino board fit together, rather than cut the box as I previously did with the Dremel and rotary cutters. So here they are:

42R RF Photon 20160515_153521

The one the left is a 42R i.e. with the sensitivity resistor added but without the fan (more on this later). The one in the center is 42RF i.e. with the resistor and a fan. Unlike the fan I added before. I mounted this fan on the opening of the PPD42. The last one (red) is a 42R with a Particle Photon. I showed the 42R Red at the SolidCon 2015 last year.

Here is another view that shows that there are three different boards in each Pickle.

42R RF Photon side 20160515_153822

The 42R on the left has an Arduino Uno, the 42RF has an Arduino Ethernet and the 42R Red has the Particle Photon.

So now on to how to connect the 10K resistor. I’ll describe this using the 42R Red. This is the smallest one I have built since the Photon is smaller than an Arduino Uno, the size is slightly larger than the PPD42 itself! The black tabs are the acrylic from the other pieces showing thru the red acrylic. It all fits together without any glue!

42R Photon 20160515_154422

Now lets open the box by removing the front face,

So now you can see that the 10K resistor is connected between A4 on the Photon, which is simply Pin5 on the PPD42 and GND. The 42R has only one cable and that is a USB cable for power. The PPD42 gets power from the VIN on the Photon and since there is no fan that is all that is needed. The Photon has wifi on board and so connected to the Photon Cloud.

Hope you liked these, I’ll return later this week and post some more results.


Posted in Uncategorized | 27 Comments

Measuring the Pickle Jr. – a modified PPD42 with an attached fan.

In the previous post I talked about how I built Pickle Jr – which is my name for the networked air pollution monitor that comprises the Shinyei PPD42 with a small Sunon fan fused to the 42. I first built the Pickle described earlier which was the networked air pollution monitor that comprised the PPD60PV-T2 (aka T2) with a small fan fused to it and an Arduino Ethernet.

I decided to do make the Pickle Jr.  hoping that the same improvement in noise and sensitivity I got with the Pickle (T2) would also apply to the 42.

For those who have worked with the 42 or one of its cousins (the DSM501 and the Sharp GP2Y thingy) know that the data gives many many zeros. As in this blog post from Darrell Tan in Singapore. http://irq5.io/2013/07/24/testing-the-shinyei-ppd42ns/, and in this blog from David Holstius http://www.davidholstius.com.

That is even when there is known air pollution, the sensor reads zero. If you wait long enough or average over a long time then there is some data. But waiting for an hour, or in some cases the entire day is not that great so I wanted to see results with a minute or so. So I added the fan to the 42 and measured it against the 42 without the fan, and the T2 and the AES-1, the zeros improved, but unfortunately I still got many of them (this is with PM2.5 less than approx 25 ug/m3.). The issue with the zeros is that the data cannot be scaled because a zero is a zero 🙂 So I have seen other folks use a previous non-zero value to guess what that zero should have been. But what if you could reduce the number of zeros significantly in the first place ?

So how do I reduce the number of zeros ?? The 42 has two controls on the board that are set at the Shinyei factory for calibration. I did not want to mess with those if I could help it because it would be impossible to know precisely what I had set them to in case I (or others) wanted to do the same thing. Then I found this brilliant note by Tracy Allen and he has figured out how the 42 works!


After reading and re-reading the note (with help from others), I understood that P1 is what most of us measure and that P2 is less sensitive. But, we can make P2 more sensitive by decreasing the threshold voltage at Pin5. I did not know quite how to do that right away but after some trial and error, I attached a 10 KΩ resistor between Pin5 and Ground and then measured the voltage at that Pin. Why 10 KΩ ? Because I wanted to reduce the threshold to half from what it was for the P1. I measured 1.35V as the threshold for P1 and with the 10 KΩ resistor I measured 0.491V at Pin5. Without the resistor I measured 2.48V at Pin5.

So now with the fan and with the resistor I measured both P1 and P2.

Screen Shot 2014-12-14 at 3.15.25 PM

The sensor 42:00012 is the 42 with Fan and resistor and 42:00004 is the vanilla 42. As before I store all the data in tempoiq and make all the plots with plotly. I have written the code to store and extract info from tempoiq and then post process in plotly. The y-axis is just the P1 and P2 voltages scaled by a straight line curve from the Shinyei data sheet measured every minute. The x-axis is the date. So now the number of zeros has gone down dramatically! But obviously the calibration would be quite off. However, since I did not muck around with the two controls, if all of us used the same resistor value then we could come up with a *new* calibration curve (more on this later).

Now how good is this 42 with fan and resistor, let me call this 42FR compared to the T2 with fan or the Dylos ? I wanted to do this with the AES-1 as well, but the windows computer it is connected to has some issues. So here is the 42FR, 42 and Dylos with measurements every minute:

Screen Shot 2014-12-14 at 3.41.25 PM

The 42FR is still quite noisy but has a response that generally traces the Dylos over this per minute interval. Changing the interval to 10 minutes  results in

Screen Shot 2014-12-14 at 3.42.57 PM

Now the respond of the 42FR vs the 42 is much clearer wrt the Dylos. Adding the T2 with fan to the mix for 1 minute intervals gives:

Screen Shot 2014-12-14 at 4.00.03 PM

Since I used the left y-axis for both the T2 and 42’s, the latter look “compressed” and, of course, the T2 with fan tracks quite closely to the Dylos. BTW, this is not the calibrated particle counts, I have not done that yet as I said before. Here is the same data over 10 minute intervals

Screen Shot 2014-12-14 at 4.10.10 PM

Interestingly the vanilla 42 can barely be seen wrt the 42FR. And now the proof of the pudding, that is how is the regression of the 42FR and the vanilla 42 vs the T2 with fan and the Dylos. First the comparisons with the T2

Screen Shot 2014-12-14 at 3.56.47 PM

That is a pretty decent R2 of 0.62 for the 42FR compared to an R2 of 0.02 for the vanilla 42! Across 10 minutes it becomes better

Screen Shot 2014-12-14 at 4.07.03 PM

R2 of 0.81 for the 42Fr compared to an R2 of 0.19 for the vanilla 42. But the number of points are not enough, I need to continue this for a longer period. Next is the comparisons to the Dylos, first the per minute

Screen Shot 2014-12-14 at 3.52.55 PM

A lower R2 than for the T2 but still oodles better than the vanilla 42. And finally the same over 10 minute intervals

Screen Shot 2014-12-14 at 4.07.18 PM

A decent R2 of 0.55 for the 42FR vs 0.21 for the vanilla 42.

I absolutely love tempoiq and plotly. I am so thankful to the tempoiq guys who have just rocked for this project! And the plotly tool is the greatest too.

Work still to be done (likely in the holiday break): compare with much higher PM2.5 concentrations, and then produce a calibration curve of the 42FR vs Dylos and T2 that anyone can use if they want to modify their 42’s. As a summary, the mods are to add a fan to the plastic housing and the resistor to the threshold pin Pin5.

Posted in Uncategorized | 22 Comments

Building Pickle Jr. – the low-cost networked PM2.5 monitor…Part 2…

The last monitor I built, the Pickle was based on the PPD60PV-T2. It was super cool to see that adding the fan fused to the T2 would improve the performance so much that it worked pretty close to the AES-1 from Shinyei.

However, the T2 is kinda expensive and I wanted to see if the same fan idea would work with the cheaper sensors, the PPD42 in particular. As a result, I built the Pickle Jr – which is the 42 with the fan connected to the same Arduino Ethernet I used for the T2. I did the same thing as with the T2 i.e. added the L-shaped connector from Rain Drip and then fitted the fan on top.


So the Rain Drip is cut just like for the T2 and then glued with black epoxy to the 42. The challenge with the 42 is that there is no circular hole at the bottom of the plastic enclosure so I had to cut a ⅜” hole thru the plastic. It was tough as the top lid kept wanting to come off. But it worked at the end. Then I snipped off the heater resistor as well. The T2 had different connectors for the heater but the 42 does not.


It looks slanted, but it is not, this is due to the weight of the fan+Raindrip.



The fan is a Sunon that is rated for 5v, but I plugged it into the spare 3.3v connection on the Arduino.

So air flows into the top opening of the 42 and is sucked out by the fan at the bottom. It is pretty airtight, as a next step I am going to remove the cable tie as well – I did not use the cable tie for the T2.

Next…Measuring and Calibrating the 42 with fan….

Posted in Uncategorized | 9 Comments

Building Pickle – the networked PM2.5 monitor….Part 1

The monitor I built to calibrate the Shinyei vs the PM2.5 AES-1, I called Pickle. Why the name Pickle ? Because I love crunchy pickles and I thought it would be a cool name. Can also stand for:

Particle Counter in the Kloud 🙂

I wanted to build a monitor that hooked up directly to the network with an ethernet cable so that there are no other gadgets like a phone or computer needed to send the data to the cloud. The monitor would be in places in India and elsewhere where power could go on and off and also where a person may not be present at all times. After some trial and error in playing with the Raspberry Pi, Arduino Yun, Arduino Uno with the Ethernet shield and the Arduino Ethernet board, I finally settled for the last choice aka the Arduino Ethernet. While it costs $45 (vs $35 for the Pi), it was very stable across power outages and loss of network connectivity after I added code that tested the Ethernet connection every 5 minutes.

Here is the completed Pickle using the PPD60PV-T2 – only missing item is the led.

Pickle front view

Pickle is made from:

6 pieces of Acrylic from TAP Plastics

Shinyei PPD60PV-T2

Sunon or Adda DC Fan. Currently I am using Sunon GM0502PFV2-8 25mm 5V brushless fan.

Arduino Ethernet board

Screws, standoffs, Rain Bird sprinkler elbow, Sugru and black resin epoxy from TAP Plastics.

The tricky part for me was adding a fan to the monitor. I wanted a way for air to come in but not light! After much trial and error including using a huge USB fan in a cardboard box, I figured out a way to directly attach the fan to the T2 and that is what I have been using since.

The key part is the Rain Bird sprinkler elbow I purchased from Home Depot:

Rain Bird Sprinkler elbow

The Sunon 25mmx25mm fan site on top of this elbow at the spout, and the horizontal section (after cutting) attaches to the T2 at the bottom. The T2 has an opening at the bottom for the heater resistor. This is where I attach the elbow for the air to get sucked out. Pictures are better than words, so:

Rain drip with horizontal section cut

Now how this attaches to the T2.

Rain Drip attaches to T2

And the fan goes on top:

T2 with Rain drip and fan

The Rain Bird attaches to the T2 and the Fan attaches to the Rain Bird with the epoxy and with black sugru.

In the next post (Part 2) I’ll walk thru in some more detail.

Posted in Uncategorized | 7 Comments

Calibrating the Shinyei PPD Sensors…..Part 2

In Part 1, I covered how I added an air-tight fan (well I have not yet published the mechanical details) to the PPD60PV-T2 and that reduced the variability of the particle count by a large amount.

In this post, I will compare the output of the T2 with and without the fan to the Shinyei AES1 (aka PMS1). The latter is a PM2.5 sensor i.e. gives output in ug/m3. The T2’s output is in volts or pulse occupancy which can be translated into PCS which is particles per 0.01 cu ft.

So I connected the 42 and the T2 to an Arduino Uno and used the interrupt function to measure the low pulse occupancy and the A->D function of the Arduino to measure the analog voltage (only for the T2). Every 30 seconds I printed these measures to the input USB port of a raspberry Pi which then posted to the net. In the case of the T2 with the fan, I built an acrylic case, housed the sensor+air tight fan and sent the data directly to the net using an Arduino Ethernet. The AES-1 has its own Ethernet board which also connected to the net, but then it sent the data only to a Windows computer (I had to borrow one from my Dad as I has not really seen one before).

So, here is the plot of the T2, T2 with Fan, Dylos (ignore for now) and the AES. This was for one day. The vertical axis is really a straight line scaling of the raw output from the T2, except for the Dylos when it is the actual reading of the small particles.

The plots are using plot.ly which reads the data from TempoDB via a python script.

Shinyei PPD60PV-T2 with and without Fan vs AES-1 every 30 secs

Shinyei PPD60PV-T2 with and without Fan vs AES-1 every 30 secs

And here are the correlation plots, first without the fan.

Shinyei PPD60PV-T2 without fan vs AES-1

Shinyei PPD60PV-T2 without fan vs AES-1

The R2 is 0.57, not too bad but not great. So from here (ignoring the y-intercept), one can take the T2 reading and divide by 27 to get the PM2.5 reading.

But things significantly improve with the T2 and fan.

Shinyei PPD60PV-T2 with fan vs AES-1

Shinyei PPD60PV-T2 with fan vs AES-1

Now the R2 is 0.96! So the simple addition of a fan made the T2 become a much better sensor for measuring PM2.5 by measuring the scaled voltage and dividing by 60 (ignoring the y intercept). Also the T2 with fan is more sensitive than without the fan.

Next I will post the mechanical details of how I added the fan.

Comments and criticisms welcome!

Posted in Uncategorized | Tagged , , , , , | 4 Comments

Calibrating the Shinyei PPD Sensors…..Part 1

I started down the air pollution measurement path with the “MQ” series of sensors and then after getting widely different results under the same conditions, I decided to switch to measuring particle counts. Like a few others I have read about, I excited about doing air pollution work because I suffered from respiratory attacks that were (finally) chased down to air pollution as the root cause.

However, the cheap PM sensors also showed variability and came with instructions of converting pulse durations or voltages into particle counts, but not into PM2.5 or PM10 measures. So I raised money ($4K) for a school project and purchased a PM2.5 sensor from Shinyei the AES-1 or PMS-1 – this was the least expensive one I could find – still cost me $1000! Shinyei says it is calibrated to the SHARP5030, an instrument that costs $15K, using the “Federal Equivalence Method”.


I hooked up the PPD42NS and the PPD60PV-T2 (courtesy Michael Heimbinder) to a Raspberry-Pi and the AES to the net and made measurements. While the correlation was not great, the T2 was much better than the 42. These plots are all taken from TempoDB, an excellent time-series cloud database that is super easy to write to and read from. All my data is stored there.

Shinyei PPD42NS every 30 sec.

Shinyei PPD42NS every 30 sec.

Shinyei PPD60PV-T2 every 30 sec.

Shinyei PPD60PV-T2 every 30 sec.

I got the idea of adding a fan from Tim Dye. After much trial and error I found that as airtight a seal between the sensor and the fan is needed to get better results. But when I did that successfully (more on this later), the results were much much better than relying on the heater in the sensor. Below is the plot of the T2 with the fan compare to the T2 without the fan above.

Shinyei PPD60PV-T2 with fan every 30 sec.

Shinyei PPD60PV-T2 with fan every 30 sec.

The vertical axis is the PCS count reported by either mapping the analog voltage or the low pulse occupancy from the Shinyei data sheets. A lot of the variability (called “noise”) in the T2 without the fan has gone away when the fan was added!

Now what remains is to compare the T2 with and without the fan to the AES1 PM2.5 sensor.

Next…..Calibrating the T2 with the fan with the Shinyei AES-1

Posted in Air Pollution | Tagged , , , , | 8 Comments

A Monster Calls

One hundred and fifty years ago, the monster began, this country had become a place of industry. Factories grew on the landscape like weeds. Trees fell, fields were up-ended, rivers blackened. The sky choked on smoke and ash, and the people did, too, spending their days coughing and itching, their eyes turned forever toward the ground. Villages grew into town, towns into cities. And people began to live on the earth rather than within it.” – Patrick Ness, A Monster Calls

Posted in Uncategorized | Leave a comment