Multiplexing is the answer.

If you want to, as an example, light up only LEDS 1, 3 and 4, you will:
- activate line A
- activate line X
- de-activate line X
- de-activate lines A and B (yes, B was not active)
- activate line A and B
- activate line Y
- de-activate line Y
- de-activate lines A and B
- start again from top
Do this so that the loop takes less than 10 milliseconds and you are actually refreshing the entire matrix at 100Hz, fast enough to trick the eyes of any human being as if the LEDs 1,3 and 4 are lighted up at the same time: persistence of vision is your friend.