0

I have the Sparkfun CAN Bus shield with integrated SD card writer attached to an Arduino Mega 2560. I can delete a logged file from the SD card using Windows, log more data using the SD library, then look at the logged data in, say, Excel. The new log file will include the logged data from the old, deleted file.

The card is formatted in Windows in FAT32 and the logged file has the same name as the original. It seems like the SD card reader/SD library is able to 'recover' the deleted file by name, open it for writing having read its uncorrupted contents, and continue logging to it, despite it having been 'deleted' in Windows. I know a deleted file doesn't get rid of the data unless it is overwritten, but there seems to be some bug/oversight in how the SD library in card reader or how it interrogates the SD card or FAT32 volume for the existence of the file.

Has anyone come across this before? Is it a bug or am I doing something wrong?

Improve this question
6
  • Do you remove and reinsert the card from/into the arduino with the power still on?
    – Majenko
    Oct 29, 2016 at 12:54
  • I have implemented a switch to initialise writing (file.open();) and stop writing (file.flush(); file.close();). This was done with the intent of ensuring the system need not be powered off to remove the card. You make an interesting point if the library maintains some kind of awareness of past files.
    – J Collins
    Oct 29, 2016 at 13:11
  • It could be caching at the block level, not just the file level. Not sure if there is a way to flush that cache...
    – Majenko
    Oct 29, 2016 at 13:21
  • The SdVolume class seems to have a cacheClear() function which flushes the block-level cache and invalidates all the cache entries. You don't normally have access to the volume entry of the SDClass, so you would have to break it down into card/volume/root in your sketch instead so you can access all the individual parts (or modify the SDClass to allow access).
    – Majenko
    Oct 29, 2016 at 13:49
  • Personally though I never use SD.h because it's horrendously slow and badly written. I use (various implementations of) FatFS.
    – Majenko
    Oct 29, 2016 at 13:50

1 Answer 1

Reset to default
1
  • How large is the file when it is deleted in Windows? I can imagine what you describe happening if there's a bug in the Arduino File System (see below) - but that requires relatively small files (less than 4K or so).
  • I assume that the SD card has zero or more "static" files, that don't change; and then the log file, that grows. That is, when the log file is deleted, the newly-freed blocks get re-used again for the new log file?
  • I also assume that you're using Windows' "Safe Removal" mechanism - giving it a chance to tidy up the SD Card before you yank it out.

To delete a file in FAT, the file system needs to do two steps:

  1. Unlink the used blocks from the File Allocation Table (the eponymous F.A.T.).
    These should be set back to zero - making it hard to accidentally re-use them again unless they're the same blocks again (which, for SD Cards, is unlikely due to wear levelling!)
  2. Mark the directory entry as available again.
    This is done by merely overwriting the first character of the directory entry with 0xE5. When FAT needs a new directory entry, it will use entries with either 0x00 or 0xE5. If the deleted file is being "resurrected", then a bug could be that it re-uses the directory entry but doesn't zero the existing contents, allowing the existing block number and length to be appended to.

Rather than deleting the file in Windows, try renaming it to something else entirely. That way the Arduino will create a new file, and you shouldn't have the problem. If you still do (worse: if the renamed file gets corrupted) then it's a caching problem. If you don't, then it's a bug in the file system.

Improve this answer
2
  • The files have been of varying sizes from 1 to 100 kB. At present this is the only file on the card, though hopefully that won't be a limitation going forward. Yep, using 'Safe Removal' in Windows, there seem to be plenty enough corruption issues with SD cards to make that procedure mandatory. Have to admit, I don't know how file allocation tables work, especially on SD cards. When you say unlinking the blocks, is a link to a set of blocks literally just the start and end block? What does the directory entry look like and are you referring to the character array of its name to overwrite?
    – J Collins
    Oct 29, 2016 at 15:21
  • Without going into the whole FAT mechanism: 1) In the FAT a file is defined as a series of blocks that are indexed as a "chain" of clusters: the directory entry defines the first cluster, and each entry in the F.A. Table defines the next cluster in the chain - with -1 to end it. 2) The directory entry contains six entries: the name; the attributes (read only, directory, etc); the create and modified times; the start cluster; and the size of the file. And yes, when deleting the file the first character of the name array is modified.
    – John Burger
    Oct 30, 2016 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.