Your confusion here is mixing "digit" and "byte". The two are not the same.
HD44780's can never have more than two rows. If you look at the block diagram you can see there are 16 "COM" connections. Those are the rows of the characters - 8 per line. That gives at most 2 lines of text.
20x4 displays are actually 40x2 displays but sliced in half and stacked. The 21st character of the first line is actually the first character of the 3rd line.
These two lines are split into two separate blocks of memory. One starts at 0x00 and the other at 0x40. The two blocks are pretty much completely separate. There is nothing of any interest to you between 0x28 and 0x3F. In fact that memory may not even exist - however those addresses may map to other addresses within memory. This will purely be an artifact of how the memory is addressed internally.
For simplicity of decoding the addresses the two blocks of memory are distinguished purely by the most significant bits of the address. The lower bits, then, will always be the same for the same location in both lines of text. That is, for the same horizontal character position on the display the first row (COL0-COL7) will have the same address in memory as the second row (COL8-COL15) except the most significant bits.
Since the HD44780 operates in blocks of 40 bits (8 characters, 5 bits per character) for its displays and uses "extension modules" to give each extra block of 8 bits (which is why you get 8, 16, 20, etc widths) having the two lines of text with truly consecutive memory addresses would make decoding the addresses needlessly difficult. So they don't.
If you take an example of the 3rd character horizontally on the display the pixel address in memory will be (note: this is the pixel address, which is 5x what the character address would be after the data has been generated from the internal character set):
Line 1: 0x02 = 0b00000010
Line 2: 0x42 = 0b01000010
Since the memory itself is 80 bytes and designed to be no more than 40 characters wide with 1 or two rows, there necessarily has to be a dead zone between the blocks of addresses. If the addresses were completely contiguous, so that the first character of the second row (character 40) was the 40th address in memory, the memory addresses would look like this:
Line 1: 0x02 = 0b00000010
Line 2: 0x2a = 0b00101010
Mapping those two addresses to the signals COM0-COM7 and COM8-COM15 would be considerably harder and take far more complexity in the design of the chip - which the manufacturer certainly doesn't want.
It is far simpler to add a little bit of logic into the cursor control which says "If the cursor moves from position 39 to 40 then set the memory address to be 0x40".
So when writing successive characters to the display the cursor logic hides the segmented memory from you. But if you are directly manipulating that segmented memory then you have to take that segmentation into account.