I am not entirely sure how a magnetometer works in IMU units, so I was wondering how a magnet near the sensor affects readings?
Will a magnetometer give distorted values if there is a magnet near?
In fact no, as the reading it gives WILL BE the field at that point BUT the field will be distorted compared to what you would see when the magnet is absent.
The effect of magnets at a distance is a somewhat complex subject - especially when there are several presnt and/or where ther are "magnetic materials" present but, as a guide.
At a distance large compared with magnet dimensions the field falls as the cube (not square) of the distance.
Very rough but useful rule of thumb: A top-end rare earth magnet will produce a field of about 1 Tesla at a distance of 1/2 it's thickness from the centre of its pole face. So eg a 20mm thick circular magnet will produce ABOUT 1 T at 10mm from the centre of its pole face. This is more useful in motor or alternator applications than when using magnetometers.