Data Analysis: SCD30 - ICOS Sensors

[keenanjohnson]

ICOS project lead: [[email protected]](mailto:[email protected])

Zurich deployment: [[email protected]](mailto:[email protected])

Intro meeting on June 15, 2023

Goal of ICOS project is to provide CO2 measurement observation data in major cities across Europe to cross-check with estimation. Data can be used to validate effectiveness of climate policies over time.

Zurich has 3 tiers of sensors already deployed:

• high precision (Piccaro)
• Mid-range (Valaiso)
• Low cost (SCD8): 160 sensors deployed.

Findings about low cost sensors (NDIR): no two sensors are the same. best practice is to deploy at least 2 sensors co-located to detect outliers. Drift is not stable. co-locate near weather stations. calibration near mid-range sensor does not guarantee that the other sensors nearby can be calibrated accordingly. best practice is to perform chamber calibration of all the sensors before deploying in the field.

3 frog sensors (SCD30) were deployed at the Zurich location, next to a high precision sensor.

Next step: data analysis for the 3 months of data collected at the Zurich site: compute offset and drift.

Sensor data from reference sensor (Picarro): https://drive.google.com/file/d/1aYCM4r6pxls_DKS3venl55GPf9jlnlo4/view?usp=share_link

Sensor data from frog sensor: download from https://dashboard.ribbitnetwork.org/

Recommendations from Pascal Rubli on approach to normalize min by min datasets:

We are using the mean value of 1 h data in our datasets and specify the time start and time end of the aggregation interval to be completely transparent about the aggregation intervals (these questions will arise at some point!).
I suggest (as you will also have to aggregate your Ribbit Sensor's data anyway) to apply the same aggregation intervals and methods to the sensor data and the reference instrument data, so you have full control over your processes. We experienced it to be crucial be in control of these processes to come to conclusions about sensor performance, otherwise some open questions might arise around simple mathematics which can be avoided this way.

[keenanjohnson]

Questions from the Ribbit Network Discord. from Achim (@airgradienthq)

Question: If we download the data from ribbit as (hourly) how exactly is the time stamp defined. Start/end or middle of data?
** Answer: ** You can see we are using a pretty basic pandas resample here. The default is that the timestamp goes to the start (left) of the mean grouping. However, if you pull the 2 seconds frequency data, it should be at the default rate, so there won't be any averaging.

Question: How are the averages made? Just arithmetic averages of all measurements in that time span without any adjustments?
Answer: Yes it is an arithmetic mean. You can see the DB queries we make here.

[airgradienthq]

Sensor data from reference sensor

@keenanjohnson I think this file only contains meterological data and the CO2 data is missing. Please check.

[airgradienthq]

Sensor data from frog sensor: download

@keenanjohnson Also it seems that I am not able to download the ribbit data.

Would it be possible you attach the csv data of the three ribbits (1min and 60min buckets)?

[keenanjohnson]

Here is one of the datasets from the Frog sensor.
frog_data.csv

[keenanjohnson]

Sorry I also missed your note on the different time buckets. Will pull that and update here shortly.

[keenanjohnson]

Here is a bit more reference information and data.

Reference Instrument Data
The reference data is split into two files one with the CO2 concentrations and one with environmental data.

Both data files are linked below:

duebendorf_empas_co2_analyser_observations_from_late_august_2023.csv
duebendorf_empas_meteorological_observations_from_late_august_2023.csv

Frog Sensor Data Files
There are datasets from 3 different Frog Sensors.

All data from each frog is contained withinin one file.

I have attached both the 1 minute and 60 minute averages here.

icos-frog-sensor-1-data-mean-1-min.csv
icos-frog-sensor-1-data-mean-60-min.csv
icos-frog-sensor-2-data-mean-1-min.csv
icos-frog-sensor-2-data-mean-60-min.csv
icos-frog-sensor-3-data-mean-1-min.csv
icos-frog-sensor-3-data-mean-60-min.csv

Early Analysis Results

Our quick look at the data reveals obvious differences in the sensors. Sensors 1 & 2 both seem to be wildly out of family with the reference data, which sensor 3 appears to be closer.

For analysis, I would recommend we focus on comparing sensor 3 to the reference source.

[keenanjohnson]

The EPA has a good guide for low cost sensor comparison / analysis here: https://www.epa.gov/sites/default/files/2018-01/documents/collocation_instruction_guide.pdf

[keenanjohnson]

Here is a photo of the sensor locations. You can see the reference sensor inlet and the Frog sensors on the right.

[keenanjohnson]

Achim from Airgradient shared the analytic scripts they use at Airgradient in case it is useful for reference. https://drive.google.com/drive/folders/1HVlJKfHY0ZP0wkzYi9uR0UCRzgIwi5Tf?usp=drive_link

[keenanjohnson]

The excellent folks at Air Gradient have done some initial analysis that I am attaching here.

Some of the key takeaways:

• There is a very different and consistent bias between the three Frog sensors as noted above. It seems that the Sensirion ABC algorithm is not compensating for the bias as intended as the bias is never corrected.
• The sensors have a very high degree of noise compared to the Airgradient sensor ( the senseair S8)
  • The increased noise could be one reason why the ABC algorithm is not functioning.
• On a positive note, once a 60 minute data aggregation window is applied, the data of the Frog sensors seems to correlate to events in the high grade reference sensor quite nicely.

2024.01.07_CO2_Ribbit.pptx