I suppose "at desktop readings" means you have your sensor resting on the desktop so you where looking for a constant output value over the time. Well, there are some mechanical limitations about gyroscope's accuracy, resolution and external factor like temperature that make noisy or non-sense readings.
What you are obtaining from this library is a calculated orientation that carries or accumulate an error over time. Basically there is not a perfect algorithm to calculate orientation as at MEM's scale electrons still moving due to temperature and other radiations.
Solution depends on your application scenario: if your sensor will be most the time quite, maybe just mathematical compensate with inverted slope (about 4-5 degrees per minute) and look for more rapid changes to determine movement. If your application will be on constant move maybe it is not required any compensation. Trade off must be considered if both scenarios will take place on your application. Anyway, reading measures on your actual application will give you more clue on how to handle this.
Below the L3GD20's mechanical characteristics