Arduino sketch works on ATmega1280, but doesn't work on ATmega328 (Arduino Nano)

Question

I'm working on arduino based RGB LED strip controller.
This simple schematic was used to control 12V channels via 5V PWM signal from arduino:

I have also connected an HC-05 bluetooth module to be able to control it using android phone.
Android application, created for this project, is sending 4 bytes to set mode, red, green and blue colors. It looks like {0,255,0,0} to set static red color, {1,0,0,0} to start fade mode and {2,0,0,0} to start blink mode.

Here is my first demo video of this project: click here.

The problem is that my code works perfectly on Arduino Mega 1280, but when I use Arduino Nano it works only with manual mode and blink mode. If I send {1,0,0,0} to turn on fade mode, LED strip starts to increase random color brightness and stops, before it gets to max value. In a moment LED strip turns off and doesn't react to bluetooth commands. Reset button helps in this situation.

I'm not sure, but I think there is not enough SRAM (only 2KB) on Arduino Nano to process fade mode data.

Source code:

#include <SoftwareSerial.h>

SoftwareSerial mySerial(12, 13);

#define REDPIN 6
#define GREENPIN 3
#define BLUEPIN 9

#define FADESPEED 10
#define PANICSPEED 425

//0 - red
//1 - green
//2 - blue
int fadeColor = 0;
int panicColor = 0;

//values (brightness) for red, green and blue colors [0; 255]
int colors[] = {0, 0, 0};

long previousTime = 0;

//0 - manual, 1 - fade, 2 - blink
int ledMode = 0;

int fadeAmount = 1;

//if color is selected, it'll change brightness;
//when we got 0 or 255 brightness we'll change
//this flag and select next color in loop()
boolean fadeColorSelected = false;

void setup() {
  pinMode(REDPIN, OUTPUT);
  pinMode(GREENPIN, OUTPUT);
  pinMode(BLUEPIN, OUTPUT);
  mySerial.begin(9600);

  randomSeed(0);
}

void loop() {
  long currentTime = millis();

  //data format - 4 bytes {mode, red, green, blue}
  if (mySerial.available() > 3) {
    ledMode = mySerial.read();
    int red = mySerial.read();
    int green = mySerial.read();
    int blue = mySerial.read();
    colors[0] = constrain(red, 0, 255);
    colors[1] = constrain(green, 0, 255);
    colors[2] = constrain(blue, 0, 255);
  }

  switch (ledMode) {
    //manual mode - just set received color on the led strip
    case 0:
      setColor();
      break;
    //fade mode - if color is selected, we'll 
    //increase/decrease it's brightness by fadeAmount
    case 1:
      if (currentTime - previousTime > FADESPEED) {
        //change color (we can't choose the same color as previous 
        //and we can't choose the color, if it's the last one,
        //because at least 1 color should be always ON
        if (!fadeColorSelected) {
          int randomColor = fadeColor;
          while (randomColor == fadeColor || isColorAlone(randomColor)) {
            randomColor = random(3);
          }
          fadeColor = randomColor;
          fadeColorSelected = true;
        }
        //depending on current brightness of selected color
        //we'll either increase, or decrease it's brightness
        if (colors[fadeColor] == 0) {
          fadeAmount = abs(fadeAmount);
        } else if (colors[fadeColor] == 255) {
          fadeAmount = -abs(fadeAmount);
        }
        colors[fadeColor] += fadeAmount;
        setColor();
        //when we got to max/min brightness, we should change current fadeColor
        if (colors[fadeColor] == 0 || colors[fadeColor] == 255) {
          fadeColorSelected = false;
        }
        previousTime = currentTime;
      }
      break;
    case 2:
      if (currentTime - previousTime > PANICSPEED) {
        int randomColor = panicColor;
        while (randomColor == panicColor || isColorAlone(randomColor)) {
          randomColor = random(3);
        }
        panicColor = randomColor;
        if (colors[panicColor] == 0) {
          colors[panicColor] = 255;
        } else {
          colors[panicColor] = 0;
        }
        setColor();
        previousTime = currentTime;
      }
      break;
  }
}

void setColor(int r, int g, int b) {
  analogWrite(REDPIN, r);
  analogWrite(GREENPIN, g);
  analogWrite(BLUEPIN, b);
}

void setColor() {
  analogWrite(REDPIN, colors[0]);
  analogWrite(GREENPIN, colors[1]);
  analogWrite(BLUEPIN, colors[2]);
}

boolean isColorAlone(int colorNumber) {
  if (colors[colorNumber] == 0)
    return false;
  for (int i = 0; i < 3; i++) {
    if (i != colorNumber && colors[i] > 0) {
      return false;
    }
  }
  return true;
}

Do you have any ideas why it happens?

If there is not enough SRAM, how can I optimize this code to solve the problem?