In a lot of the sample code online people add the line Serial.begin(9600) in the setup block.
When I look up what Serial.begin() is on the official documentation, it says that it controls the bit per second data transfer.
So the obvious question is, why not use 28800, the highest transfer rate? Why do people settle for 9600? What is the limitation here?
Why do people settle?
People settle because it is more than fast enough. The most common use is just to print some stuff on a terminal for debuggin. 9600 baud is 960 characters per second, or 12 x 80 character lines per second. How fast can you read? :)
If your program is using the serial port for bulk data transfer, you would choose not to settle.
What is the limitation...
The limits on serial are high. Directly you can use 115200 baud in your programs and it will just work. The Arduino terminal will allow a max of 115200, but other programs such as RealTerm would let you run higher.
Hardware serial will run to 1 M baud. If you read around you will see people have used up to 1 M by directly controlling the UART. You might get benefit of high baud rates for uses such as transmitting via a bluetooth chip. If you are using the hardware serial interface to exchange from chip to chip with just a short distance, then 1 M baud is completely feasible. Think of all the SPI and I2C devices that operate just fine at 1 MHz clock rate.
Over larger distances, you will start to have problems with noise when using logic level (plain 0 to 5V) signalling. To use larger distances, you would add a transceiver to provide robust signalling, commonly RS-232 and less commonly RS-485. With RS-232 you could run a mega bit at distances of 10's of feet.
The microprocessor clock speed will be the real limit. With a hardware UART, the processor must load one byte to the UART every 10 bits (for N81). So when you get to 1 M baud it will be a challenge for the 16 MHz processor to keep the UART supplied with data. A new byte will be sent every 160 clock ticks, which is very few lines of code. For a short burst of data, you might achieve that rate. The message is, the processor will run out of speed before the UART is the limit.
Note, this all applies to HardwareSerial, software serial is very different.
In addition to all the interesting answers, it worth mentioning that setting the serial speed to XXX bits/s does not necessary imply XXX bits/s on the hardware.
Clocks -- even quartz based -- are imperfect and subject to drift. In addition, as the serial clock is usually generated through a power-of-two pre-divisor and (integer) counter, all value cannot be accurately obtained given a base clock frequency. With the help of the start/stop bits, asynchronous serial communication may be tolerant to some clock drift. But this has limits.
For example, if your ATmega328PA is running at 1MHz, you can achieve 9600b/s at 0.2% of error. But at 14400b/s the error is -3.5% (actually communicating at 13900b/s). And at 28800b/s, the error is +8.5% (actually communicating at 31200b/s). All those figures are from ATmega48PA-88PA-168PA-328PA datasheet, p200.
This is not an issue when two identical devices communicate together (as there are in fact communicating at the same speed). It might be an issue when communicating between different devices.
Increasing the base frequency does not necessary improve significantly the accuracy. For example, running the same ATmega328PA as above at 2MHz does not really give better results as those are mostly due to rounding errors. But running it 1.8432MHz give very accurate bps from 2400b/s up to 57.6kHz.
I think it is a kind of tradition to use a transfer rate that is not the slowest one (300) but also not one that could eventually cause problems in some setups (28800 or even 115200). The PC serial port (most often a FTDI232 USB adapter) can cope with higher rates but your DIY hardware may not. So 9600 bps has established itself as some kind of standard transfer rate for code examples.
Back in the mists of time, the "gold standard" for remote keyboards (using a a telephone modem, and teletypes, if you remember those) was 9600 baud, initially achievable only over a dedicated phone line. Time moves on, slowly; technology advances, rapidly; and memory moves even more slowly than time (it seems). We can routinely communicate, at least over a several meters, at a couple of orders of magnitude faster than 9600 baud. What was once considered a gold standard is no longer gold, but still thought of as standard.
tl;dr: It's history, not technology.
I think the main reason people use 9600 most of the time is that it's the default baud rate in the Arduino IDE. Also, faster data rates could also be unreliable if the serial signal has to travel a long way - although I have no idea why this was selected as an optimum speed.
