Why can't an Arduino/ATmega328 act as an SPI slave at anywhere approaching the maximum speed claimed in the datasheet?

Question

The ATmega328 datasheet claims a maximum SPI slave speed of Fosc/4. For a chip clocked at 20MHz, that would be 5Mbps.

However, my tests show that even with no processing whatsoever in the interrupt (other than setting the response), any speed over 128Kbps results in corrupted data. I have no idea why this should be the case.

In my setup, the master runs at 3.3V and 12MHz; and the slave at 5V and 20MHz.

---

Master code:

#include <SPI.h>
#include "pins_arduino.h"

unsigned int successcount = 0;

void setup(void) {
  Serial.begin(9600);
  SPI.begin();
  SPI.setClockDivider(SPI_CLOCK_DIV4);   // 3Mbps
}

void print_successful_transfers() {
  if (successcount > 0) {
    Serial.print("(");
    Serial.print(successcount);
    Serial.println(" successful transfers.)");
    successcount = 0;
  }
}

void loop(void) {
  unsigned char n = 0;
  unsigned char received;
  unsigned int failurecount = 0;

  delay(500);       // wait for the slave to be ready

  digitalWrite(SS, LOW);    // SS is pin 10

  while (failurecount < 10) {
    for (unsigned int count = 0; count < 10000; count++) {
      received = SPI.transfer(n+1);
      if (received != n) {
        print_successful_transfers();
        Serial.print("MISMATCH: sent ");
        Serial.print(n+1);
        Serial.print(", received ");
        Serial.print(received);
        Serial.println(".");
        failurecount++;
        if (failurecount == 10) break;
      } else
        successcount++;
      ++n;
    }
    print_successful_transfers();
  }
  while(true);
}

Slave code:

#include "pins_arduino.h"

volatile char n = 0;

void setup (void) {
  pinMode(MISO, OUTPUT);
  SPCR |= _BV(SPE);
  SPCR |= _BV(SPIE);
}

ISR (SPI_STC_vect) {
  n = SPDR;
  SPDR = n;
}

void loop (void) { }

---

Sample output:

MISMATCH: sent 1, received 245.
(1290 successful transfers.)
MISMATCH: sent 12, received 5.
MISMATCH: sent 13, received 140.
(1350 successful transfers.)
MISMATCH: sent 84, received 41.
MISMATCH: sent 85, received 212.
(996 successful transfers.)
MISMATCH: sent 58, received 28.
MISMATCH: sent 59, received 186.
(5455 successful transfers.)
MISMATCH: sent 139, received 197.
MISMATCH: sent 140, received 11.
(900 successful transfers.)
(114 successful transfers.)
MISMATCH: sent 131, received 193.

---

Here is my schematic. I am using bare ATmega328 chips rather than Arduinos, but the situation would be the same if I were using a 5V Arduino Nano and a 3.3V Arduino Nano.

(Except the slave would run at 16MHz instead of 20MHz.)

I also tried with the following level shifter on the MOSI, SCK and SS lines:

---

EDIT:

To rule out voltage as the possible cause, I ran both ATmega328s at 3.3V and 12MHz. Same problem.

---

EDIT: I connected two Arduino Nanos together and got different - but still erroneous - behaviour.

The SS pin, despite being set to LOW at the start of the program and never touched again - goes briefly high every so often, corrupting the communication:

(If I use a different pin for slave select, I get the same behavior.)

Note that this is NOT the issue I experienced with my bare ATmega328 circuit.