do I have to connect RX with RX and TX with TX or RX to TX and vice versa?
If you want to do serial communication between a sketch running on the Arduino and the ESP8266 you must connect the RX pin on the Arduino to TX on the ESP8266 and the TX pin on the Arduino to TX on the ESP8266.
If you want to do serial communication between the Serial Monitor on your computer through the USB port on the Arduino to the ESP8266 you must connect the RX pin on the Arduino to TX on the ESP8266 and the TX pin on the Arduino to RX on the ESP8266.
TX stands for Transmit and RX stands for Receive. So it makes sense that you should connect the transmit pin to the receive pin in the first case. But why connect transmit to transmit and receive to receive in the second case? The reason is that you are not communicating between the main microcontroller (ATmega328P on the Uno) and the ESP8266. You are communicating between the USB-serial chip on the Uno (usually ATmega16U4 on the official Arduinos or CH340 on the clones) and the ESP8266. The main microcontroller is not being used at all, you're just using the Arduino as a bulky and inconvenient USB-serial adapter.
In fact some people will go so far as to remove the ATmega328P from their DIP Unos for this purpose. The RX pin on the USB-serial chip is connected to the TX pin on the main microcontroller as well as the TX header pin and vice versa for TX pin on the USB-serial chip. This means by connecting the ESP8266's TX pin to the TX pin on the Uno's header you are actually connecting it to the RX pin on the USB-serial chip. I would highly recommend getting a USB-serial adapter made for this purpose rather than trying to use an Arduino. You can buy FTDI FT232 or CH340 breakout boards on eBay or Aliexpress for < $2 USD w/ free shipping.
do I need any converters? Because now I directly connect the TX and RX to the Arduino's TX and RX
This is actually two questions. Let's address the easy one first. The TX pin of the ESP8266 is going to put out around 3 V. Even though the ATmega328P is running at 5 V this will still be seen as a high so no level conversion on that pin is necessary.
Now for the question of whether it's needed on the ~5 V level from the Arduino to the ESP8266 RX pin. There's a lot of disagreement on the question of whether the IO pins on the ESP8266 are 5 V tolerant. We certainly know that it's possible to connect the ESP8266 to 5 V levels as there are many projects published online that do this but the question is will this will shorten the life of the ESP8266?
The datasheet for the ESP8266 does not say the inputs are 5 V tolerant. There is a statement on the ESP8266 forum from a member of the Espressif (creator of the ESP8266) staff:
We donot recommend you to apply 5V to IO.
We have a comment from someone on Facebook named "Teo Swee Ann" who appears to be the CEO of Espressif:
i can reply officially here: it is 5V tolerant at the IO. while the supply voltage is at 3.3V.
So there is conflicting information and none of it has been presented in the most official manner, such as actually putting it in the datasheet. One thing we do know for sure, you will not harm the ESP8266 with 3.3 V levels. So the safest option is to avoid giving it 5 V inputs. However, as you are currently struggling just to get the most basic communication working the extra complication introduced by adding level shifting circuitry may not outweigh the possibility of reducing the life expectancy of your ESP8266.