You're trying to port the library from the Due to the Mega? If the library is large a "port" just isn't going to happen because relatively minor modifications to the code are very unlikely to shrink the size significantly; you'll need a complete rewrite.
My suggestion is to start by studying the data sheet for the device and the code for the existing Due driver to get an idea of how it works. Then sit down and write some sketches for your Mega that make something happen on the display (perhaps just clearing it, turning on and off a few specific pixels, the like) to verify your understanding. Once you've done that, you can start to consider the design of your driver.
Because of the vastly different memory limitations it's likely that your driver will look and work very differently from the Due driver. For example, a 264 x 176 x 1 bit frame buffer would use almost 6K of RAM (most of the RAM in a Mega 2560 and far more than in an Uno), so you're not going to want to have one. I don't know if the Due driver has a frame buffer (I'm just using this as an example of a typical difference you might encounter), but if it does, and that's used for doing certain kinds of things, you'll need to do them in a different way or not do them at all.
This is not a simple project, but it's a great one and will teach you a lot about microcontroller programming!