This is because your LDR catches reflected sunlight from the ground and thinks that it is actually a sun...
Maybe you could install few LDRs and compare their values. The LDR with highest value is looking at sun others are just receiving reflected light.
Why not just adjust sensitivity of single LDR to filter reflected light and react only to direct sunlight?
Because intensity of the sunlight will vary during the day, and if you adjust it in the midday it may not react to afternoon's sunlight at all. If you adjust it in the evening it may react even to reflected light in the midday.