1

I'm trying to send info of about 30-40kB over the Serial ports on an Arduino Due. But I'm running into a few problems. First, before we get into the actual transfer over the Serial, I want to try storing 30000 Bytes into a char * array.

Also as a disclaimer, I've modified the SAM library's RingBuffer.h SERIAL_BUFFER_SIZE to 256 from 128.

void setup() {
  Serial.begin(115200);
int i,j=0, cksum=0;
int *test;
test = (int*)calloc(1,30000);
Serial.println("Starting...\n");

for(i=0;i<30000;i++){
  test[i]=i;
  cksum +=j;
  j++;
  if(j>255){j=0;}
}
Serial.print("cksum is: ");
Serial.println(cksum);
cksum = 0;
for(i=0;i<30000;i++){
  if(i<300){ //only print out the first 300 characters, otherwise it's hard to see anything
    Serial.print("[");
    Serial.print(test[i],HEX);
    Serial.print("]");
  }
  cksum+=(int)test[i];
}
Serial.print("\nNow cksum is: ");
Serial.println(cksum);
}

Obviously, since *test is a byte array, I'm only storing int8 into there - so from 0-255. That means that the data received in *test increments from test[255]= 255 to test[256] = 0. The way around this would be to use some kind of itoa function to write out multiple bytes that represent the numbers being written (causing an overflow because it would no longer be an array for 30000 Bytes). But for these purposes is it ok to simply realize that this is happening and just write the bytes to the array.

Ok, so for part two, I want to do the same thing, but by writing to Serial2 from Serial3 and receiving all of the bytes:

void setup() {
  Serial.begin(115200);
  Serial2.begin(115200);
  Serial3.begin(115200);
int i,j=0;
double cksum=0;
char *buf;
Serial.println("Starting...\n");
buf = (char*)calloc(1,30000);

Serial.println("Writing to Serial3");
  for(i=0;i<30000;i++){
    Serial3.write((byte)i);
    cksum+=j;
    j++;
    if(j>255){j=0;}
  }
 Serial.println(cksum);
 Serial.println("Reading from Serial2");
 Serial2.readBytes(buf,30000);
 cksum = 0;
  Serial.print("Buf is: ");
  for(i=0;i<30000;i++){
    if(i<300){ //only print out the first 300 characters
      Serial.print("[");
      Serial.print(buf[i],HEX);
      Serial.print("]");
    }
    if(buf[i] != j){
      Serial.print("\nMismatch at i = ");
      Serial.print(i);
      Serial.print("\nBuf is: ");
      Serial.print(buf[i-1],HEX);
      Serial.print(",");
      Serial.println(buf[i],HEX);
    }
    cksum+=buf[i];
    j++;
    if(j>255){j=0;}
  }
  Serial.println("\n\nEnd");
  Serial.println(cksum);

}

What happens here is that I get a mismatch at byte 255. The increment (in hex) goes:

[254] = FE
[255] = 30
[256] = 31

But it should read:

[254] = FE
[255] = FF
[256] = 0

After reading some other forums, and the fine AVR Arduino post here, I feel like something is happening on the Serial interrupt level that I am not aware of. All of the data is properly transferred to the Serial and I am able to see all of the bytes in the correct order coming out of TX and into RX on my logic analyzer. I've also experimented with different baud rates, but see no reason why a 84Mhz chip would not be able to handle a baud rate of 115200.

Has anyone else run into this? How do you handle these transfers of continuous data that is larger than the buffer? Is the only option here to run readBytes in SERIAL_BUFFER_SIZE increments until Serial is no longer available? Or can I still salvage using readBytes all in one go as I am trying to do?

I know this question has been asked in certain forms in other threads like here and here but they do not solve this particular problem I am experiencing, most likely because they are not Due specific.

  • I think you have a fundamental lack of understanding about how serial communications works. – Majenko Sep 29 '16 at 14:06
  • You write 30,000 bytes of data out of a serial port, then read that data as it appears on the UART pins into a 256 byte buffer. Once you have sent all 30,000 bytes into that 256 byte buffer you then try and read 30,000 bytes out of that 256 byte buffer and wonder why it doesn't work right...? – Majenko Sep 29 '16 at 14:07
  • Imagine you have a bucket with 30,000 marbles in it, and two cups that can hold 256 marbles each. You want to empty the bucket of marbles and then put all the marbles back in again. How do you do it without getting marbles all over the floor? – Majenko Sep 29 '16 at 14:20
  • Yes, clearly the Arduino code does not handle this contingency properly as I would have hoped. It requires a rewrite of the stream function read(). – Andrew Sep 29 '16 at 20:57
  • 2
    No, it requires a rewriting of your understanding of serial communications. – Majenko Sep 29 '16 at 22:58
2

Let me dissect your program and show you precisely what is going on and why your approach is completely wrong:

Your first step is to throw 30,000 bytes at the serial port. Those go via the TX ringbuffer, which is (as you have changed it for some reason) 256 bytes:

buf = (char*)calloc(1,30000);

Serial.println("Writing to Serial3");
  for(i=0;i<30000;i++){
    Serial3.write((byte)i);
    cksum+=j;
    j++;
    if(j>255){j=0;}
  }

At this point 29744 bytes have left the hardware UART. 256 are still sat in the TX ringbuffer being sent out by the TX interrupt routine. The first 256 bytes that were sent out were picked up by the RX interrupt routine and stored in the RX ringbuffer. The next 29488 bytes were discarded (they are now all over the carpet with your marbles).

 Serial.println(cksum);
 Serial.println("Reading from Serial2");

Now you try and read 30,000 bytes from the 256 byte RX ringbuffer. While you are doing that the remaining 256 bytes in the TX ringbuffer are being slowly sent out. You are now making some room in that RX ringbuffer since you are removing bytes from it, so some of the remaining 256 received bytes can now be stored in it. The first one of those happens to be 0x30.

 Serial2.readBytes(buf,30000);

Now you go and print out your array, which has only a tiny fraction of the 30,000 bytes you sent in it.

 cksum = 0;
  Serial.print("Buf is: ");
  for(i=0;i<30000;i++){
    if(i<300){ //only print out the first 300 characters
      Serial.print("[");
      Serial.print(buf[i],HEX);
      Serial.print("]");
    }
    if(buf[i] != j){
      Serial.print("\nMismatch at i = ");
      Serial.print(i);
      Serial.print("\nBuf is: ");
      Serial.print(buf[i-1],HEX);
      Serial.print(",");
      Serial.println(buf[i],HEX);
    }
    cksum+=buf[i];
    j++;
    if(j>255){j=0;}
  }
  Serial.println("\n\nEnd");
  Serial.println(cksum);

}

The simple fact is, you are transmitting, en-masse, far more bytes than the Arduino's RX ringbuffer can hold. In order to receive more than the 256 bytes of the ringbuffer you have to read the bytes from that ringbuffer to make room for more to arrive. And since you are the one that is transmitting that means you have to do your receiving at the same time as the transmitting.

The only possible way that your arrangement of program could work would be if you increased the RX ringbuffer to greater than 30,000 bytes so it can store all the received bytes as you transmit them for you to then, afterwards, copy the contents of that ringbuffer into your own buffer.

  • After a lot more research, this is actually the way that the buffer should work (the way I had originally written it) according to the Atmel datasheet and Atmel ASF. There is a bug in the Arduino kernel, I have fixed Streamio it and it works now (as the original code) that you where you confuse the functioning of RingBuffer and how StreamIO works on an ARM chip. I will recommend this fix to Arduino. – Andrew Oct 3 '16 at 0:32
2
int *test;
test = (int*)calloc(1,30000);

First, I would be testing if that calloc worked. 30000 bytes is a lot on any microprocessor. For example:

test = (int*)calloc(1,30000);
if (test == NULL)
  {
  Serial.println ("Insufficient memory.");
  exit (1);
  }

Second, can't you calculate the sumcheck on-the-fly?

for(i=0;i<30000;i++){
  test[i]=i;
  cksum +=j;
  j++;
  if(j>255){j=0;}
}

If you want to send a sumcheck at the end, calculate as you send each byte and then send it afterwards. That way you don't need the 30000 byte array.


Your technique of sending 30000 bytes and then trying to read them back is hoping that the 30000 bytes will be saved somewhere while you are writing and before you read. Where?

I've modified the SAM library's RingBuffer.h SERIAL_BUFFER_SIZE to 256 from 128.

Certainly not there. That now holds 128 bytes.

So you are expecting a buffer of 30000 bytes which you set up, plus another 30000 bytes somewhere to hold the data while you send it. Even on the Due you are using most of your available RAM for this exercise.


What are you really trying to do here?

http://xyproblem.info/

  • Well I've got two FIFOs here. When the read comes in, I get an interrupt and service it. We're talking milliseconds. I should be able to service the interrupt, grab the byte, service it and go into the next one. – Andrew Oct 1 '16 at 5:14
  • Is it a problem in the streamio? Or is it the Arduino library over it? All of this stuff is an extension of Streamio. Arduino is extending this class which is where HardwareSerial comes from. HardwareSerial uses a class inheritance to make its own reads and writes. So you are saying that the Due framework does not handle the serial buffer properly? – Andrew Oct 1 '16 at 5:25
  • 1
    Yes, this is just an exercise I'm using to get a handle on how the Arduino kernel is handling serial data. developer.mbed.org/cookbook/Serial-Interrupts – Andrew Oct 1 '16 at 6:43
  • 1
    Oddly enough, an exercise to check how things might work in a larger situation, can fail in a simulation. For example, sending from HardwareSerial to SoftwareSerial on a single board. A better test would be to split the test between two Arduinos, where one sends, and the other receives. That is, after all, closer to what you will eventually do. As an analogy, try making up 1000 random numbers, and telling them to yourself (as a batch). You will probably have trouble doing that, but if you told them to another person they could repeat each number as they got it. – Nick Gammon Oct 1 '16 at 7:20
  • 1
    I have made this example work by modifying the Arduino kernel and Streamio. No need to send the data as a batch, but if you read chapter 34 of the Atmel SAM chip datasheet it explains what should be done to the UART, and the Arduino files did not handle it correctly. – Andrew Oct 3 '16 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.