Detecting colour with RGB illumination and an LDR is not particularly accurate in my experience. The idea is you illuminate the card (or whatever) with red light, take an LDR reading, then with green then blue.
This gives you a set of 3 light levels for the different colour components of the object. Combining them together gives you an RGB signature for the object. Ideally, on say a red card, the value from the LDR under red illumination is much stronger, and blue & green much weaker.
RGB LED + LDR colour detection does work, but it's not accurate, and it helps to block out ambient lighting. Maybe if you put a lot of time into calibration, it would for you. I had a classroom of 14 kids, each made-up one of these circuits, yet only a couple had any moderate success, if at all. That said, we were using very cheap RGB LEDs and similar quality LDRs.
There's a bunch of stuff you can do to help the accuracy, like subtract readings with only ambient illumination, take multiple readings, etc.
I think you would save yourself a lot of time and frustration buy using a purpose-made colour sensor.
Something like the ADAFruit Colour Sensor - https://www.adafruit.com/products/1334