I want to control a servo with the least amount of components. But generally, what is the simplest way to do this without actually using a fullblown arduino board?
If you what you're looking for is the cheapest option with the fewer components possible, then you can use a regular ATMega328p chip, with a few passive components attached to it.
You just need to program it with an Arduino Bootloader once, in order to be able to program it easily, but you can use any existing Arduino to do that. After doing this once, you can program it with any FTDI-type USB to Serial converter.
Here's an Instructable that explains the whole process.
Please, take a look at this guide by Nick Gammon, which explains in detail how to build a breadboard Arduino.
You'll need also an Arduino board without the chip (a good UNO will do the job) because you need to program it some way. It is needed absolutely in the case that you buy an ATMEGA328 that hasn't the bootloader, as you'll have to write it on the blank chip.
To speak directly via serial with the standalone Arduino (and to program it), I suggest you to buy a FT232RL board (just be careful to buy a real one, as there are fakes that FTDI blocked and bricked with drivers)
Then, for the servo you just have to plug it correctly: two cables for power, one for the control.
This is useful if you don't need all the I/O of the Atmega.
If you want a simple board just to run a servo or two and still have some I/O you could use a ATTiny85 chip and use this software servo library that works with the attiny. It also works best with the High-Low tech cores.
I used this with a project and it works flawlessly without stalling.
This would give you small setup as you can use the internal clock 8MHz and minimal external components, i.e. regulator and Capacitors.