Does anyone know where on the board are the RX and TX pins
The serial Rx and Tx signals from the ATmega2560, go to the nearby FT232RQ USB-to-serial adapter. You can look at the part numbers on the ICs and find those two devices on the PCB.
no more info on the board in the internet and forums
It's good news that there does seem to be more information available :-) After a few minutes with Google (searching for GT2560 schematics) I found this download link for the schematics on this forum page.
Although it would be possible to reverse-engineer the PCB (or at least the relevant part of it) the schematics are still helpful, since the white coating makes the actual copper traces harder to follow visually.
The schematic shows that two 1k resistors, R3 and R13, are in series with serial signals:
- R3 between ATmega Tx -> FT232 Rx
- R13 between ATmega Rx <- FT232 Tx
You may want to confirm that the linked schematic still applies to your specific PCB (since PCB and schematic versions could have changed). With the power disconnected, you could check with a multimeter that there is continuity between the appropriate ATmega pin and one end of each resistor R3 & R13, and between the other end of each resistor and the appropriate FT232 pin, as shown on the schematic.
I have put a red ring around those 2 resistors on the GT2560 top view below, to help you to find them:
(original image source)
best place to connect to them
If you are going to drive those serial signals from an ESP8266, you must first disconnect them from the on-board FT232, to avoid contention and so the ESP8266 replaces the FT232. Removing those two resistors would make each resistor's (now vacant) pads leading to the ATmega serial port, an ideal place to connect the ESP8266. (Of course you could attach wires directly to the pins of the ATmega or FT232, but I don't see any advantage to doing that.)
One issue which you need to consider, is the voltage compatibility of the serial port signals. The ATmega2560 uses 5V signal levels, whereas the ESP8266 is a 3.3V device. There have been discussions on various websites (example) that suggest the ESP8266 is 5V-tolerant on I/O pins. You will need to satisfy yourself whether to use voltage translation, or to rely on information discussed on web pages like that one.