Step one: read the data sheet for the microcontroller. In there is a nice diagram that shows you exactly how the IO pins work.
The pins can work in 2 modes: input, and output. You're interested in output mode.
The drive state of a CMOS output pin consists of two MOSFETs, one connecting the pin to Vcc and the other connecting it to GND. Only one of the MOSFETs is ever turned on at a time, so the output pin is either connected to Vcc through one MOSFET, or connected to GND through the other.
When it's connected to Vcc it's said to be sourcing current, since it's at a positive potential compared to ground and current can flow out of the pin to light an LED (say), and when it's connected to GND it's said to be sinking current because current can flow into the pin to get to ground.
As an example of how it all works, consider how you would connect up and power a capacitative humidity sensor.
These sensors require a square wave at around 1KHz to operate. The humidity defines the impedance, and as part of an impedance divider (like a resistor divider) the output voltage is relative to the humidity.
Now, they don't just want any square wave, but require a square wave that reverses polarity around a virtual ground point, and that means reversing the polarity of the power across the sensor at 1KHz. How can you do that with an Arduino? The answer is simple:
Connect the sensor to two IO pins rather than 1 IO pin and GND.
Both pins are set as output, and one is set HIGH with the other set LOW. So the voltage across it is 5V.
Now you switch the outputs over, so the one that was HIGH is now LOW, and the one that was LOW is now HIGH. The power across the sensor has been reversed! It's now (effectively) -5V (compared to what it was before).