You should have a resistor of a few hundred ohms on the base connection of the transistor - that is, between base and pin 13.
A PNP transistor is somewhat harder to get to grips with than an NPN transistor. Much of it does things backwards.
The operation you see seems to be completely correct when you understand how a PNP works.
With and NPN transistor a high voltage on the base turns it on and a low voltage turns it off. The opposite is true with a PNP transistor.
When you have a voltage close to or above the voltage at the emitter the transistor will be off. That means, providing a HIGH to the base will turn it off. A voltage on the base that is below the emitter voltage minus the threshold voltage of around 0.7V will turn the transistor on. That means providing a LOW to the base will turn it on.
So that means that when the on-board LED is on the transistor is off, and so the LED on the transistor will be off. When the LED on the board is off the transistor will be on, so the LED on the transistor will be on.
"But", you say, "the on-board LED doesn't turn off". That is correct. It just gets dimmer. That is, again, because of how the PNP transistor works. Current (seen from the conventional sense - i.e., flowing from + to -) enters the emitter and is then split between the collector and base. The amount of current allowed to flow out of the base defines how much is allowed to flow out of the collector (in an NPN it's current into the base and collector that combine to flow out of the emitter, and the amount that is allowed to flow into the base defines how much is allowed to flow into the collector). So there will be some current flowing out of the base - which makes its way to ground both through the microcontroller's IO pin and also through the LED on board, making it glow slightly*.
*That is a little crudely put, but kind of illustrates what is going on.