3

I'm trying to use an Arduino for data collection, and hoping to get a consistent sampling rate.

I looked into the serial communication using the simplest program I could, but it produces odd results.

void setup() { Serial.begin(9600); }
void loop() { Serial.println("00000"); }

I use a python script to read the data and create a text file and write the time in seconds before the five zeros.

Using the seconds as a marker, I then count the number of data lines for each second. for example:

44 00000  // recorded at time 44 seconds
44 00000
44 00000
45 00000  // recorded at time 45 seconds, etc

I was expecting that there would be a somewhat stable value for this. Using 10 bits per character, printing the 5 zeros should be able to be done about 190 times per second. When analyzed, it turns out to be lower, which isn't surprising due to overhead, but also unexpectedly variable.

Collecting the zeros for 5000 seconds produced the following results: most often, there were 137 sets of 00000 collected per second, followed by 138 sets of 00000. This is easily explainable by not being equally spaced per second, and having one second with 137 followed by one second with 138.

What is harder to explain is when it jumps to 139, 140, 141 and 142 sets of 00000 per second.

Out of the 5000 seconds of data, there were 25 seconds where 140 sets of 00000 were created. That isn't too bad and could be due to timing.

But, out of the 5000 seconds, there were 203 seconds where 142 sets of 00000 were created.

The percent breakdown is as follows:

137 sets of 00000:  79.15%
138 sets of 00000:  13.8%
139 sets of 00000:  0.39%
140 sets of 00000:  0.51%
141 sets of 00000:  2.05%
142 sets of 00000:  4.11%

the first 4 could be explainable with minor timing errors, the last ones are very hard to explain. with 137 sets of 00000 there are 685 characters sent serially in one second.

with 142 sets of 00000 there are 710 characters sent serially in one second.

What I am trying to figure out is why the variability is so wide and also so consistent - as in the pattern, which can be seen in the graph.

frequency of characters sent per second pattern of characters sent

Is there any explanation for the variability?

Because of the variability in the transfer, it is very hard to sync up data or filters or other uses when a constant sample rate is needed.

I've also added delays in the main loop from 1ms to 7ms, 7ms was the maximum because that would not cause the loop to run slower than the free running speed at 142 loops/transmissions per second (1sec/7ms = 142.8 loops per second discounting all other overhead) and found the same patterns.

Using a delay of 8ms produced another interesting result: with a delay of 8ms, most of the data was transmitted at 123 or 124 sets of 00000 per second, but every 12 seconds there would be a burst of 128 sets of characters. this makes sense because the delays add up to less than one second for 11 loops, then get an extra loop for that second, but that adds 15 characters, not just 5.

After re-reading this, it dawned on me that there might be some loop timing/buffering errors, so instead of a delay, I used Serial.flush() to pause the loop until the data was sent.

This resulted in a free-running result of 137 sets transmitted per second for the most common, followed by 138 sets. The new issue is that every 25 seconds there is a drop down to 133 sets of characters transmitted per second, which seems to be the same issue only causing slowing instead of speeding up.

I thought I found the answer with Serial.flush() but there is still something causing huge changes in the sent characters per second. What could be causing this? and what can be done to make the transfer rates more constant?

1

While a serial port "kept busy" can potentially produce data with jitter comparable to its clock's jitter reduced by the number of clocks per word, a USB serial converter introduces a bunching of data into packets that is effectively entirely asynchronous.

As a result, once the data has gone through the USB, you really can't determine much of anything about the timing of its origin just by looking at the timing of its arrival. That remains true even if you are claiming it via a serial-type API on the host, because the USB bus has already irreversibly de-synchronized it in transit.

A more meaningful experiment would be to timestamp the data at the source, ie, on the Arduino. That could potentially be more regular, but if you use a high enough resolution timestamp, you will still find that it is irregular, as by default the Arduino's foreground thread of execution is periodically interrupted by the timer, which produces a brief delay.

If you are sending serial data into the Arduino or using other hardware peripherals, those may cause additional sporadic delay-inducing interrupts.

-1

This could be largely to do with how USB serial works.

USB has a number of different methods of transferring data between the device and the host. Both CDC/ACM use FT232 use a method called bulk transfer. This is a very low priority transfer method and data is moved across the bus as and when it can - that is, when the bus isn't doing anything else. If there is anything else happening on the bus (and there will be) the transfers get delayed until the bus is doing nothing else.

If timing is critical for a protocol, such as USB audio or video where you have to have the data transferred to the host absolutely right now a different method is used. This is called isochronous. It is the highest priority of data transfer, even higher than interrupt data.

However there is no way on the Arduino to use any form of isochronous transfers so you will have to come up with some other method of managing your timing.

As Chris has mentioned the most common method is to include a timestamp with each packet of data. That packet may contain just one sample, or it may contain a block of samples. The important thing is to know when the block starts and from that infer the time of each sample within the block.

It is also important to ensure that your communication with the host isn't running at 100% saturation. You should be running at the highest baud rate that is reliable and only send data at regular intervals giving you time to get the data in between each transfer. Keeping your data packets as small and efficient as possible is also good for keeping you transfer rate sustained properly.

  • The problem is not USB priority, it is that USB is asynchronous from the serial source, so packet grouping should be considered meaningless. Time stamping packets is not the solution and not supported without changing the USB converter anyway. The serial data can encode an almost regular timing, but the timing of the data through USB is meaningless and should be ignored by the application in favor of encoded timing, either literal or implied. Using that, you can do audio over bulk transfers in most modern lightly loaded cases. – Chris Stratton May 21 '16 at 15:01
  • I think you misunderstood most of my post there... CDC/ACM Bulk Transfers are asynchronous to the UART source, yes. Isochronous aren't - they are synchronous (or close enough). That is why they are used for audio and bulk transfers aren't. Yes, it is possible to fudge it enough with bulk transfers using a custom higher level protocol, and it is that higher level custom protocol to which I refer when I talk about timestamping packets - not timestamping USB packets but your serial data. If you have a packet that contains 10 samples which are from 1ms intervals and you know the first sample ... – Majenko May 21 '16 at 15:04
  • ... is at 829736ms then you also know the time of each subsequent sample in that packet. That batching of packets isn't real-time but it is accurate on a per-sample time basis. Bulk transfers can, and will, be delayed by higher priority USB traffic - especially on a bus that is shared with other devices - especially devices that use isochronous transfers. To get the best results directly connect the Arduino to a USB port on the computer and, if possible, check that other devices don't use the same root hub. – Majenko May 21 '16 at 15:06
  • No. Even isochronous transfers, while more regular in rate, are still asynchronous to the source - and they are allowed to be dropped entirely. In neither case can one ascribe meaning to the USB timing, or even the timing of the USB packetizing process. Rather, the timing has to come from the original sample clock, or if one is using a serial UART kept busy to sample something that does not have its own clock, then the baud rate. The receive then either operates using only the source's timing implicit in the data (ie, if is is only doing signal processing). – Chris Stratton May 21 '16 at 15:27
  • Or if it is producing audio output, it must buffer the data and then play back using another sample clock, that is either synchronized by a low bandwidth PLL to the buffer level, or more commonly fixed frequency with a variable sample rate converter in between. – Chris Stratton May 21 '16 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.