A great deal of what you learn with any comprehensive IDE will transfer to another one, if you need to. The IDEs that come to mind are Eclipse (listed first, only because it is my personal choice and has a broad range of application), Visual Studio, and others I have no experience with.
The Arduino IDE is as limited as you found it, primarily because it was written as (1) a learning tool for (2) AVR-based Arduinos, and goes to some lengths to offer an easier entry into embedded systems programming for non-programmer hobbyists. To that end, it achieves its goal very well. But as you have observed, you quickly run into its limitations, especially when your goal is to branch out onto other processors and other tool-sets.
Once someone is over the hump of basic programming and debugging, more advanced tools like a full-on IDE take most of those limitations out of your way, at the cost of less hand-holding, but by that time, you presumably need the freedom much more than the help!
I don't want to outright tell you "You need to use FiddleFoo 18.4 or you'll never get anywhere" - one (or a couple) of the industrial strength IDEs will appeal to your view of programming and your personal work-style. Try a few of them, paying attention to what toolsets and processors each one can target with OEM or third-party support, for the processors you think you might become interested in. It's quite likely T-H-E right IDE for you will stand out from the rest.
Eclipse will need a plug-in to target Arduino and other embedded system boards. The one I use - and the only one being maintained AFAIK - is Sloeber by Jan Baeyens, who occasionally posts here as jantje. It is pretty comprehensive, including the ability to write and import new board descriptors.
Memory limitations come with the hardware, and while compiler advances can help this out, it's pretty much governed by the numbers. An Arduino Uno still has only 2K of RAM and 32K of Flash minus whatever the bootloader uses (if you choose to use a bootloader). Heap is still problematic for all the usual reasons - fragmentation of free memory (positioned, as it is, between static area and the bottom of the stack), and reduction of available stack and/or data space. There are, of course, more embedded boards available, with larger memories, but that just helps you put off the inevitable. I view the use of heap in an embedded system as a reliability issue even more than as a memory limitation; it is all too easy for fragmentation and memory leaks to eventually take down your system, and a Blue Screen Of Death is a big no-no in embedded systems.