I'm trying this project so far I've got the schematic of a Pro Micro, which is powered through GPIO from a Raspberry Pi Zero, and want to remove the micro usb connection and able to flash it using an avr programer and a few pins.
Welcome to arduino.stackexchange.
You can program your ATmega32U4 on at least two ways:
1. Use Arduino's bootloader and D+/D- pins. This way you won't need an external programmer. Try search for phase "programming ATmega32U4 with bootloader".
2. Use ISP (In-System-Programming) method. In this case you can use your avr programmer. To do so you also need access to PINs named Reset, MISO, MOSI, SCLK. See this link or this one
As always, the important read is ATMega32U4 datasheet.
An avr programmer uses the ISP interface to program a chip. For this you need the pins for MOSI, MISO, SCK and Reset. Of course you have also to provide power to the board through Vcc and ground.
I'm not sure, if it's OK to keep the USB connection while programming via ISP, but since you want to remove that connection either way, this isn't a problem for you. You can remove the USB connection completely (no D+ or D- connection).
Note: when you are programming the Arduino Micro via ISP without also burning the bootloader, you won't be able to use USB for programming anymore, until you burn the bootloader again.
Why don't you buy a pi zero to USB add-on board? You can stack the pi on top and plug it into a normal female USB port on your computer.
The USB board has some pins that connect to the solder points of the mini USB, so you can delete all the connectors you won't need.
Other option is to solder a wire to the pi and extend the connector with a cable.
I'm not shire, if you can program the pi true its serial pins. USB is a serial connection after all. So I think this should be possible