Sending Large Packets and Reassembling over Serial

Question

I'm trying to send info of about 30-40kB over the Serial ports on an Arduino Due. But I'm running into a few problems. First, before we get into the actual transfer over the Serial, I want to try storing 30000 Bytes into a char * array.

Also as a disclaimer, I've modified the SAM library's RingBuffer.h SERIAL_BUFFER_SIZE to 256 from 128.

void setup() {
  Serial.begin(115200);
int i,j=0, cksum=0;
int *test;
test = (int*)calloc(1,30000);
Serial.println("Starting...\n");

for(i=0;i<30000;i++){
  test[i]=i;
  cksum +=j;
  j++;
  if(j>255){j=0;}
}
Serial.print("cksum is: ");
Serial.println(cksum);
cksum = 0;
for(i=0;i<30000;i++){
  if(i<300){ //only print out the first 300 characters, otherwise it's hard to see anything
    Serial.print("[");
    Serial.print(test[i],HEX);
    Serial.print("]");
  }
  cksum+=(int)test[i];
}
Serial.print("\nNow cksum is: ");
Serial.println(cksum);
}

Obviously, since *test is a byte array, I'm only storing int8 into there - so from 0-255. That means that the data received in *test increments from test[255]= 255 to test[256] = 0. The way around this would be to use some kind of itoa function to write out multiple bytes that represent the numbers being written (causing an overflow because it would no longer be an array for 30000 Bytes). But for these purposes is it ok to simply realize that this is happening and just write the bytes to the array.

Ok, so for part two, I want to do the same thing, but by writing to Serial2 from Serial3 and receiving all of the bytes:

void setup() {
  Serial.begin(115200);
  Serial2.begin(115200);
  Serial3.begin(115200);
int i,j=0;
double cksum=0;
char *buf;
Serial.println("Starting...\n");
buf = (char*)calloc(1,30000);

Serial.println("Writing to Serial3");
  for(i=0;i<30000;i++){
    Serial3.write((byte)i);
    cksum+=j;
    j++;
    if(j>255){j=0;}
  }
 Serial.println(cksum);
 Serial.println("Reading from Serial2");
 Serial2.readBytes(buf,30000);
 cksum = 0;
  Serial.print("Buf is: ");
  for(i=0;i<30000;i++){
    if(i<300){ //only print out the first 300 characters
      Serial.print("[");
      Serial.print(buf[i],HEX);
      Serial.print("]");
    }
    if(buf[i] != j){
      Serial.print("\nMismatch at i = ");
      Serial.print(i);
      Serial.print("\nBuf is: ");
      Serial.print(buf[i-1],HEX);
      Serial.print(",");
      Serial.println(buf[i],HEX);
    }
    cksum+=buf[i];
    j++;
    if(j>255){j=0;}
  }
  Serial.println("\n\nEnd");
  Serial.println(cksum);

}

What happens here is that I get a mismatch at byte 255. The increment (in hex) goes:

[254] = FE
[255] = 30
[256] = 31

But it should read:

[254] = FE
[255] = FF
[256] = 0

After reading some other forums, and the fine AVR Arduino post here, I feel like something is happening on the Serial interrupt level that I am not aware of. All of the data is properly transferred to the Serial and I am able to see all of the bytes in the correct order coming out of TX and into RX on my logic analyzer. I've also experimented with different baud rates, but see no reason why a 84Mhz chip would not be able to handle a baud rate of 115200.

Has anyone else run into this? How do you handle these transfers of continuous data that is larger than the buffer? Is the only option here to run readBytes in SERIAL_BUFFER_SIZE increments until Serial is no longer available? Or can I still salvage using readBytes all in one go as I am trying to do?

I know this question has been asked in certain forms in other threads like here and here but they do not solve this particular problem I am experiencing, most likely because they are not Due specific.

Answer 1

Let me dissect your program and show you precisely what is going on and why your approach is completely wrong:

Your first step is to throw 30,000 bytes at the serial port. Those go via the TX ringbuffer, which is (as you have changed it for some reason) 256 bytes:

buf = (char*)calloc(1,30000);

Serial.println("Writing to Serial3");
  for(i=0;i<30000;i++){
    Serial3.write((byte)i);
    cksum+=j;
    j++;
    if(j>255){j=0;}
  }

At this point 29744 bytes have left the hardware UART. 256 are still sat in the TX ringbuffer being sent out by the TX interrupt routine. The first 256 bytes that were sent out were picked up by the RX interrupt routine and stored in the RX ringbuffer. The next 29488 bytes were discarded (they are now all over the carpet with your marbles).

 Serial.println(cksum);
 Serial.println("Reading from Serial2");

Now you try and read 30,000 bytes from the 256 byte RX ringbuffer. While you are doing that the remaining 256 bytes in the TX ringbuffer are being slowly sent out. You are now making some room in that RX ringbuffer since you are removing bytes from it, so some of the remaining 256 received bytes can now be stored in it. The first one of those happens to be 0x30.

 Serial2.readBytes(buf,30000);

Now you go and print out your array, which has only a tiny fraction of the 30,000 bytes you sent in it.

 cksum = 0;
  Serial.print("Buf is: ");
  for(i=0;i<30000;i++){
    if(i<300){ //only print out the first 300 characters
      Serial.print("[");
      Serial.print(buf[i],HEX);
      Serial.print("]");
    }
    if(buf[i] != j){
      Serial.print("\nMismatch at i = ");
      Serial.print(i);
      Serial.print("\nBuf is: ");
      Serial.print(buf[i-1],HEX);
      Serial.print(",");
      Serial.println(buf[i],HEX);
    }
    cksum+=buf[i];
    j++;
    if(j>255){j=0;}
  }
  Serial.println("\n\nEnd");
  Serial.println(cksum);

}

The simple fact is, you are transmitting, en-masse, far more bytes than the Arduino's RX ringbuffer can hold. In order to receive more than the 256 bytes of the ringbuffer you have to read the bytes from that ringbuffer to make room for more to arrive. And since you are the one that is transmitting that means you have to do your receiving at the same time as the transmitting.

The only possible way that your arrangement of program could work would be if you increased the RX ringbuffer to greater than 30,000 bytes so it can store all the received bytes as you transmit them for you to then, afterwards, copy the contents of that ringbuffer into your own buffer.

Answer 2

int *test;
test = (int*)calloc(1,30000);

First, I would be testing if that calloc worked. 30000 bytes is a lot on any microprocessor. For example:

test = (int*)calloc(1,30000);
if (test == NULL)
  {
  Serial.println ("Insufficient memory.");
  exit (1);
  }

---

Second, can't you calculate the sumcheck on-the-fly?

for(i=0;i<30000;i++){
  test[i]=i;
  cksum +=j;
  j++;
  if(j>255){j=0;}
}

If you want to send a sumcheck at the end, calculate as you send each byte and then send it afterwards. That way you don't need the 30000 byte array.

---

Your technique of sending 30000 bytes and then trying to read them back is hoping that the 30000 bytes will be saved somewhere while you are writing and before you read. Where?

I've modified the SAM library's RingBuffer.h SERIAL_BUFFER_SIZE to 256 from 128.

Certainly not there. That now holds 128 bytes.

So you are expecting a buffer of 30000 bytes which you set up, plus another 30000 bytes somewhere to hold the data while you send it. Even on the Due you are using most of your available RAM for this exercise.

---

What are you really trying to do here?

http://xyproblem.info/