Your problem is that you are trying to measure motes of dust with a yard stick. The yard stick is great at measuring things in inches and feet, but when you want to find the size of something that's around 1mm in size all you can say is "it's less than an inch" or "smaller than an eighth of an inch".
You have to remember that a 20A sensor has to fit that whole range of 0-20A - that's 0-20,000mA into half the range of the ADC. That's at most 512 graduations on an Arduino, and since you can usually count on "±1 LSB quantisation noise" you really only get half that as a usable value. So you're looking at a resolution of about 40mA at best with an accuracy of ±40mA. When you want to measure small currents they're just going to get lost in the lack of resolution. At 100mA you're still only in the range of just "There might be a current there, somewhere".
For measuring current you need to size your sensor / shunt appropriately to the range of the currents you're sensing. If you're working with small variations in a large DC current then pre-conditioning the sensing to change the offset and amplify the interesting part of the signal is required, so that the signal isn't swamped by the underlying offset. If you're working with small currents then you require a sensor / shunt that gives you an appropriate signal for that current range.
Additionally, different variants of the sensor have different sensitivity: 185 mV/A for the 5A model, and 100 mV/A for the 20A model. The code you are using doesn't take this in account. Using the correct sensitivity, the last value in your table, 683 mA, becomes 1,264 mA (683×185/100=1264). This is close enough, if you consider the ±40mA accuracy.
