How to create custom bootloader with accompanying program loader for ATMega2560

Question

Just for some context, I have created a custom board based on the ATMega2560. I currently have the original Arduino Bootloader for the ATMega2560 and have no problems uploading code via the USB Port on TX0 and RX0.

What I am hoping to achieve is creating a custom Bootloader that allows me to upload my .hex binaries to flash but in a proprietary way, so that any program cannot be uploaded to it with the standard Wiring Protocol used by Arduino IDE. In addition, I will need a custom PC based program that can upload the code via the USB port.

The reason why I need this is that I will have this device out in areas where I would not want someone taking the device and uploading their own program to it and screw it up. I know it sounds paranoid, however, it would be preferred to have this feature so I can update the firmware via the USB Port without using the Wiring protocol.

Has anyone accomplished this before?

TL;DR

Want to upload custom Bootloader to Arduino Mega so that I can upload my firmware with a proprietary programmer via the USB Port on TX0 and RX0 because I am paranoid.

Answer 1

The source of the bootloaders is publicly available, so a simple thing would be to make a minor modification, for example XOR each incoming byte with 0xEE. That would mean the bootloader wouldn't work with normal uploading programs (the IDE) but with a modified uploader.

However even simpler would be to upload your code using ICSP and not include a bootloader at all. Or, leave the bootloader there but turn off the BOOTRST fuse so that the processor ignores it.

That way, it takes some effort for people to replace your code.