Im just trying to learn AVR programming without the I guess "handholding" of arduino functions/etc.
You don't actually need to do anything. You don't need any breadboard, any programmer, nothing.
You can use the Arduino environment, upload facility, and board to learn low-level AVR programming.
Just because the Arduino helper functions exist it doesn't mean you actually have to use them.
The first step on your progression to professional embedded developer is to get to grips with low-level registers. You can do all that from within the Arduino environment. You have access to everything in the Arduino IDE that you have in any other IDE in the way of low-level chip functions. They are functions of the chip, not the IDE.
If you want to progress to a professional IDE and use hardware debugging you can still use an Arduino board. After all, it's just an MCU on a PCB. Just plug your hardware programmer into the hardware programming connector on the board. Why mess around with breadboards and all the trouble that they give with dodgy connections? Just because an Arduino has an Arduino logo on it you aren't forced to use it with the Arduino IDE.