Are you sure one pins is configured as input? If one would be output high and the other output low you basically get a short circuit: from Vcc through the top FET of the high output, through the wire to the other Arduino, and into the lower FET of the low output, and back to ground.
If the short circuit were perfect the voltage would drop to zero, but both FETs have a non-zero resistance, which makes them a voltage divider.
One way to protect your Arduino against this (such a short circuit does damage the I/O circuits!) is to place a 1 kΩ series resistor between the pins. Since an input pin won't draw any current there won't be a voltage drop across the resistor either.
I wouldn't use Harry's I2C, because it's probably too complicated for your needs. (I don't know what data you want to exchange). A simple connection is the UART: one transmit line and one receive line. It's true that the UART is used for the USB communication, but that connection goes via a set of resistors, so that another controller using the bus should be able to override the USB's signals.
If you have a long distance between the Arduino's you'll want RS232 level shifters.