Algorithms to reduce measurement error from analogue sensors

When reading analogue sensors - e.g. load cells, accelerometers - via a microcontroller's Analogue to Digital Converter, there are many sources of potential measurement error including:
a) Warm-up time for electronics
b) Mechanical inertia and resonance
c) Spurious random readings (due to e.g. vibration, electrical interference).

My current software hack for a) and b) is to simply add a suitable time delay before reading the sensors; however there must be a more robust way of detecting and screening for resonance...

My hack for c) is to take the mean of a given number of consecutive readings, but discard any individual readings which exceed a given percentage range from the current mean (pseudocode example below). The trouble with this approach is that it requires an inital baseline' sensor reading, and if this happens to be spuriously high or low, it can cause subsequent correct readings to be rejected. Another approach might incorporate a weighted moving average, but this would still be skewed by spurious random readings.

Are there any robust and elegant approaches / algorithms for the above which are commonly applied in commercial devices (e.g. domestic or laboratory weighing scales) that I could include in my code to improve the accuracy and reliability of measurements?

Many thanks in anticipation

// Take mean of multiple sensor readings excluding outliers:

# define SAMPLES 100 // number of samples to take mean of
# define TOLERANCE 0.1 // % range of allowable deviation
//between current mean and new values, expressed as decimal

n = 1; // loop variable
total = analogRead(analogPin); // Running total for calculation of mean & baseline reading
mean = total; // mean of all current data

while (n < SAMPLES) {
new = analogRead(analogPin);
if (new < (1 + TOLERANCE) * mean) and (new > (1 - TOLERANCE) * mean){
total += new; // if new reading is within range, calculate new mean
n += 1;
mean = total / n;
}
// (else ignore new reading)
}
Serial.print mean;

• You might want to investigate Standard Deviation and similar statistical models: en.wikipedia.org/wiki/Standard_deviation – Majenko Oct 10 '15 at 13:28
• Thanks Majenko - I'm aware of use of SD in statistical analysis of data. Is this used in commercial algorithms for reading sensors with microcontrollers in real time though? – Dave Oct 10 '15 at 13:45

3 Answers

I use something very similar to what you are proposing to measure RSSI in an antenna pattern generator. Say you have an array of 16 samples. You start filling the array with samples but don't actually output a result until the array is full. This will give you a startup period of 16 samples where no decisions are made.

After the array is loaded, set a semaphore that indicates that results may now be used to make decisions. New samples overwrite the oldest sample in a recirculating buffer. You will now get decision outputs at the same rate as the sample rate with a lag, i.e. when the inputs go to zero abruptly, it will take 16 sample periods for the output to catch up. I call this a 16 sample rolling average.

For relatively stable inputs, simply take the mean of the array and output that result.

For inputs that are glitchy such as RSSI take the mean of the array and then find the sample(s) that is (are) the farthest away from that mean. You then recalculate the mean without that sample included. Note: Do not discard that sample, simply ignore it for that cycle. I call this a 15/1 outlyer average. For my RSSI measurements, this gives me the equivalent of a 48 sample rolling average. You can do an 8/1, 16/2 system etc requiring 9 and 18 samples per sample respectively, tuning the counts to your particular application. I am able to get nearly anaechoic chamber results in the middle of a radio noisy office.

Again, note that the outlyers are NEVER discarded as they may represent a real trend.

Thank you John! Your solution is what I was looking for: it resolves my conflict between preserving all the data and excluding the skewing effects of outliers; is simple enough to implement on a microcontroller; and seems to work perfectly. I'm guessing that the rolling mean will also smooth out any potential swings due to mechanical resonance.
I know this is an Arduino forum, but I find it quicker and easier to write interactive code in Python, so I've written a crude test bed' implementation of your algorithm in Python so that I could get a feel for how it works in practice, with user keyboard input simulating real-life sensor readings. I've avoided any libraries or esoteric Python syntax, so this should be easy to adapt to C++/Arduino. I'm posting it here in case it is of interest to the community.
I've used the formula for Sample Std Deviation (n-1) rather than Population SD (n) which I think is correct in this context, but please let me know if not:
in LaTeX: $SD=\sqrt{\frac{\sum_{(i=1)}^{n}(x_{i}-\bar{x})^{2}}{n-1}}$

If there are any other algorithms or solutions to my post, I'd be very interested to know. Thanks again.

#===================================================
def SampleSD(data,bufLen):
datasum = 0.0 # NB declare as float to prevent integer rounding error
diffsum = 0.0 # NB declare as float to prevent integer rounding error
for m in range (0,bufLen):
datasum += data[m]
mean = datasum / bufLen
for m in range (0,bufLen):
diffsum += (data[m] - mean) ** 2
sd = (diffsum /(bufLen-1)) ** 0.5
print "n: %d\tSum: %f\tMean: %f\tSD: %f" %(bufLen,datasum,mean,sd)
return mean,sd

#=====================================
def addData(data,bufLen):
print "\n------------------------------"
new = input ("Sensor reading: ")
for m in range(bufLen-1,0,-1):
data[m]=data[m-1]
data[0]=new
return data

#======================================
bufLen = input ("Buffer Length: ")
print
data = [0] * bufLen # Declare an array of suitable length in Python
n = 0

# Fill buffer with initial data
while n < (bufLen):
new = input ("Initial sensor reading: ")
data[n] = new
n +=1
print "\nBuffer initialised."
print "Data: ",data

while True:
addData(data,bufLen)
print "Data: ", data
mean,sd = SampleSD(data,bufLen)
#===========================================
# Recalculate Mean only using data within range +/- 1 SD of mean
if sd != 0: # Error trap to avoid divide by zero error if sd = 0
print "\nChecking for outliers:"
newSum = 0.0 # NB declare as float to prevent integer rounding error
newLen = 0
n=0
while n < (bufLen):
if ((data[n] < (mean + sd)) and (data[n] > (mean - sd))):
print "Data [%d] = %d Included" % (n, data[n])
newSum += data[n] # sum of data which is within +- 1SD of original mean
newLen += 1 # number of data whic are within +/- 1SD of original mean
else:
print "Data [%d] = %d Excluded: Out of range" % (n, data[n])
n+=1
CorrMean = newSum / newLen
print "New n: %d\tNew Sum: %f\tCorrected Mean: %f" %(newLen,newSum,CorrMean)


... and here's my minimal working example for the Arduino. Feel free to adapt it for your own requirements, and please post any improvements or corrections to my script; or any alternative algorithms for the above.

Just noticed that my approach of using the SD to define and exclude outliers is a little more (? unnecessarily) complex than John's algorithm, which defines outliers in terms of the highest and lowest values from the mean; but both methods seem to work nicely.

Cheers!

#define BUFFER 5 // Buffer size (number of readings to use in rolling Mean)
#define SENSORPIN 3 // Analogue input pin from sensor
int data[BUFFER]; // Make an array of appropriate size for Buffer

void setup() {
pinMode(SENSORPIN, INPUT);

// Fill buffer with initial data
int n = 0;
while (n < BUFFER) {
data[n] = analogRead(SENSORPIN);
n++;
}
}

void loop() {
// Calculate Raw Mean & SD
float datasum = 0;
float diffsum = 0;
for (int n = 0; n < BUFFER; n++) {
datasum += data[n];
}
float mean = float(datasum / BUFFER);
for (int n = 0; n < BUFFER; n++) {
diffsum += ((data[n] - mean) * (data[n] - mean));
}
float sd = sqrt(diffsum / (BUFFER - 1));

// Recalculate Corrected Mean only using data within range +/- 1SD of Raw Mean
if (sd != 0) {      // Avoid divide by zero error if sd = 0
float newSum = 0; // Sum of readings within acceptable range
float newLen = 0; // Number of readings within acceptable range
int n = 0;
while (n < BUFFER) {
if ((data[n] < (mean + sd)) and (data[n] > (mean - sd))) {
newSum += data[n];
newLen ++;
}
n++;
}
mean = newSum / newLen; // Corrected Mean
}

// Add your code here to do stuff with variable float 'mean' = the (Corrected) Mean

// Read new sensor reading, append to buffer and delete oldest reading
for (int n = (BUFFER - 1); n > 0; n--) {
data[n] = data[n - 1];
}
data[0] = analogRead(SENSORPIN);
}