The only alternative structure I can think of is, since you have a number of data fields to send, you could create a struct to contain all the fields. Write the gathered GPS data to the file in its raw format, with file.write(&your_struct, sizeof(your_struct)) and read in a similar manner into a struct.
- This will reduce the number of bytes written or read for any data point, and
- Reduce the overall size of the file, compared to your original method, for the same number of GPS points.
The former improvement means there will be about 3 times as many GPS data points in a single block of data read from or written to the SD card, when compared to your current structure. Another more subtle advantage is that each data point occupies a known number of bytes in the file, making file navigation potentially easier. But you can only use this structure if you don't intend to view the CSV file with some spreadsheet program like MS Excel. You could, of course, very easily write your own PC program to parse the file and write to proper CSV. The Python struct module is very capable.
However, if you wish to continue with your current format, you can keep track of your current position in the file with file.position(); no need to use # as an indicator. You create 2 position variables, one to hold the position of the next line to be uploaded, curr_pos, and the other to hold the position of the end of the file for writing new data, end_pos. Initialize the former to 0.
- You gather data as usual, and after 5 mins, you save the current file position to
end_pos and seek to curr_pos with file.seek().
- You read a line from the file and send it to the server.
- If it's sent successfully (use the return code), you update
curr_pos with file.position() so that it's now pointing to the next line in the file to be uploaded and go to step 2.
- Else if the upload fails perhaps due to service, you stop uploading and don't update
curr_pos. You seek to end_pos with file.seek() and go to step 1.
You will, of course, use available() to know when you've hit the end of the file to stop uploading, or you can compare curr_pos with end_pos for equality, whichever's more direct for you.
This involves a lot of operations on the file and I/O operations don't come cheap with SD cards, both in terms of current drawn and latency. Depending on how often you collect data and if you don't need to view it on a PC, you might consider some other means of storage (SPI flash, perhaps) or use a more capable Arduino like the Mega with more RAM and store the data temporarily in an array before upload, granting you faster access by orders of magnitude and much reduced power drain in the long term, since each block of file data will be accessed only once.
