Sometimes Tx and Rx are connected directly despite the fact, that it works on different voltages, sometimes resistor dividor is used, sometimes it goes through resistor. Why no consensus? What are differences?
There is some uncertainty as to whether the ESP8266 has 5V tolerant inputs or not. This uncertainty stems from differences in an early Chinese datasheet translated by the community, and the current English datasheet. It's better to err on the side of caution though and translate your 5V signal from the Arduino to a 3.3V one with either a proper logic level translator or a resistive divider (note that the latter can limit the maximum baud rate you can use).
You can read more here:
Why contributors are so stubborn to use software UART? Is there any reason I can't understand, that should stop me from using built in UART?
Bad information begets bad information. People copy poor quality examples and tutorials and pass that bad knowledge on to others1.
Many Arduino boards only have one hardware UART port, and that is shared with the USB connection to the PC. It is hard to reliably use that UART for other serial communication, since doing so will break your ability to upload code while your hardware is connected. So it's simpler in that case to use SoftwareSerial (or one of the better implementations).
However, I would never advocate using SoftwareSerial unless you absolutely have to. It barely functions at 115200 baud (one of the common default ESP8266 baud rates). If you have a convenient hardware UART then really you must use it, for the sake of your sanity if nothing else.
1: Even worse though are the examples where people use SoftwareSerial to communicate over the pins of a hardware UART... that's just plain stupid...