Trouble programming custom ATMEGA2560 board

Question

I need some help. I copied the heart of MEGA2560 design into a new project. I have the new PCBs with blank MCUs and now I need to get things programmed. (I have an existing project that runs on a MEGA2560)

I was able to get the MEGA16U2 programmed using the procedure here: https://www.arduino.cc/en/Hacking/DFUProgramming8U2, (I loaded "Arduino-usbserial-atmega16u2-Mega2560-Rev3.hex") and it shows up in the Arduino IDE.

I got the ATMEGA2560 bootloader loaded using the Arduino-as-ISP instructions here: https://www.arduino.cc/en/Tutorial/ArduinoISP

But, now when I try to program my board with the with Arduino IDE (through the USB connection of the MEGA16U2), I just get:

avrdude: stk500v2_ReceiveMessage(): timeout

The full output of the IDE is:

Sketch uses 30056 bytes (11%) of program storage space. Maximum is 253952 bytes.
Global variables use 1280 bytes (15%) of dynamic memory, leaving 6912 bytes for local variables. Maximum is 8192 bytes.
/home/xxxxx/.arduino15/packages/arduino/tools/avrdude/6.3.0-arduino17/bin/avrdude -C/home/xxxxx/.arduino15/packages/arduino/tools/avrdude/6.3.0-arduino17/etc/avrdude.conf -v -patmega2560 -cwiring -P/dev/ttyACM0 -b115200 -D -Uflash:w:/tmp/arduino_build_464802/yyyyy.ino.hex:i 

avrdude: Version 6.3-20190619
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2014 Joerg Wunsch

         System wide configuration file is "/home/xxxxx/.arduino15/packages/arduino/tools/avrdude/6.3.0-arduino17/etc/avrdude.conf"
         User configuration file is "/home/xxxxx/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : /dev/ttyACM0
         Using Programmer              : wiring
         Overriding Baud Rate          : 115200
avrdude: stk500v2_ReceiveMessage(): timeout
avrdude: stk500v2_ReceiveMessage(): timeout
avrdude: stk500v2_ReceiveMessage(): timeout
avrdude: stk500v2_ReceiveMessage(): timeout
avrdude: stk500v2_ReceiveMessage(): timeout
avrdude: stk500v2_ReceiveMessage(): timeout
avrdude: stk500v2_getsync(): timeout communicating with programmer

avrdude done.  Thank you.

An error occurred while uploading the sketch

I can see the RX led of the MEGA16U2 flashing, and the data at the RX pin of the ATMEGA2560 IC, indicating that data is getting to the MCU from the IDE, but there is no corresponding TX activity.

The IDE and USB cable can program an Elegoo MEGA2560 clone with no problem.

Is this a baud rate problem? A protocol problem? What should be my next debugging step?

Thanks!

---

EDIT: Per Nick Gammon's comment, I have run his chip detector script. Here is the output:

Atmega chip detector.
Written by Nick Gammon.
Version 1.20
Compiled on Sep  5 2020 at 07:17:55 with Arduino IDE 10812.
Attempting to enter ICSP programming mode ...
Entered programming mode OK.
Signature = 0x1E 0x98 0x01 
Processor = ATmega2560
Flash memory size = 262144 bytes.
LFuse = 0xFF 
HFuse = 0xD8 
EFuse = 0xFD 
Lock byte = 0xCF 
Clock calibration = 0x88 
Bootloader in use: Yes
EEPROM preserved through erase: No
Watchdog timer always on: No
Bootloader is 8192 bytes starting at 3E000

Bootloader:

3E000: 0x0D 0x94 0x89 0xF1 0x0D 0x94 0xB2 0xF1 0x0D 0x94 0xB2 0xF1 0x0D 0x94 0xB2 0xF1 

<....snipped for length....>

3FFF0: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 

MD5 sum of bootloader = 0xD9 0xE6 0x6B 0x4E 0xD1 0xA6 0x11 0x2C 0x61 0x8F 0x9B 0xD5 0x5D 0x24 0xE2 0x13 
Bootloader MD5 sum not known.

First 256 bytes of program memory:

00: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
10: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
20: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
30: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
40: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
50: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
60: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
70: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
80: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
90: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
A0: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
B0: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
C0: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
D0: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
E0: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 
F0: 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 

Programming mode off.

Answer 1

As per the comments your output shows that the chip is working "normally" in that it reacts to the chip detector sketch. Thus the bulk of the design/implementation is OK.

Referring to Engbedded Atmel AVR® Fuse Calculator your fuse settings means the processor is expecting an external crystal oscillator. The bootloader expects (I think) a 16 MHz crystal (with appropriate capacitors). If you have not provided this then the bootloader will be running at the wrong baud rate (so to speak) — or not at all — and not react to attempts to download it by serial programming.

If you have a different frequency crystal then you may be able to work around this by adjusting the baud rate in the boards.txt file by an appropriate scaling factor.

The chip detector sketch worked because that uses ICSP programming, which provides its own clock.