12V is the upper limit for input voltage on an Arduino. With a 12V input the regulator has to convert 12-5, or 7 volts, to heat. (That's how linear voltage regulators work.) As the other poster said, Arduinos don't have much heat dissipation on their voltage regulators.
Like jksemple, I would suggest powering both the ESP and the Nano from the external regulator (You might want to cut a USB cable and wire it to your external voltage regulator, then plug it into your Nano. That way you're not bypassing the power control circuits on the Nano.
By the way, did you build a full voltage regulator with the LM1117 chip? You need some extra components like capacitors and resistors to complete the voltage regulator.
Finally, you might think about using a DC-DC buck converter rather than a linear regulator. That will run a lot cooler and waste a lot less power.