Problem sending colors from an audio visualizer to Arduino

Question

Sorry in advance for the length of my question; I want to provide proper context. I wrote an audio visualization program in Java that (in addition to making "pretty pictures" based off of the amplitudes of different frequencies being sent from the computer) sends a color to an Arduino Uno based off the sound it is "hearing."

I know the algorithm for generating this color is functional (in addition to sending that color to the Arduino, it also changes the color of an item on the DrawingCanvas). When the color is sent to the Arduino (as 3 bytes representing red, green, and blue), though, the physical LEDs don't respond correctly (they become a random color for a few seconds then switch to another random color for a bit – the behavior is hard to describe, but certainly not desired!).

I have been able to send colors to the Arduino using the same client program, but when the audio visualizer sends colors by the same method, this odd behavior occurs. My thought is that, because the audio visualizer sends color profiles dozens of times per second, the Arduino might be getting overloaded with data. Is that possible?

Here are the relevant pieces of code from the client, the audio visualizer, and the Arduino:

Client:

public class Client extends WindowController{
  //USB is a class I wrote that extends PApplet. Unless baud rates are the issue, I don't think we need to worry about this.
  static USB port = new USB("COM3", 9600);
  byte[] code = new byte[LENGTH];
  static byte killTerm = -100;
  private AudioVisualizer vis;

public void begin() {
  resize(1500,900);
  for (int k=0;k<LENGTH;k++){
    code[k]=killTerm;
  }
  vis = new AudioVisualizer(10,30,1200,300,canvas);
}

public void onMouseClick(Location point) {
  runVis = !runVis;
  if(runVis) {
    vis.start();
    code[0]=-22;
    sendCodeUSB(code);
  } else
    vis.stop();
}

public static void sendCodeUSB(byte[] code) {
  for (int k=0; k<code.length; k++) {
    port.writeProfile(code[k]);
  }
}

AudioVisualizer

public class AudioVisualizer implements Runnable {
  Thread thread = null;
  DrawingCanvas canvas;
  byte[] code = new byte[3];
  private int xPos;
  private int yPos;
  private int width;
  private int height;
  private int scale;
  private Location topLeft;
  private Location bottomRight;
  private FilledRect background;
  private int brightness=0;
  private boolean needOff = false;

  public AudioVisualizer(int x, int y, int w, int h, DrawingCanvas myCanvas) {
    canvas=myCanvas;
    xPos=x;
    yPos=y;
    topLeft = new Location (xPos,yPos);
    bottomRight = new Location (xPos+width, yPos+height);
    width=w;
    height=h;
    background = new FilledRect(topLeft,width,height,canvas);
    background.setColor(Color.BLACK);
    initiateMonitor();
    scale = height/30;
  }

  public void start() {
    if (thread == null) {
      thread = new Thread(this);
      thread.start();
    }
  }

  public void stop() {
    thread=null;
  }

  public void run() {
    while (thread != null) {
      try {
        Thread.sleep(40);
      } catch (InterruptedException e) {
      }
      graphAmps();
    }
    thread = null;
  }

  public void graphAmps() {
    //there is a whole bunch of stuff here that is unrelated that I removed.
    //basically, just know that graphAmps() calls avgColors()
    avgColors();
  }

  public void avgColors(){
    //there is stuff here that generates avgRed,avgGreen,and avgBlue (known to be working)
    Color c = new Color (avgRed,avgGreen,avgBlue);
    background.setColor(c);
    //so we don't spam the Arduino more than we need to
    if(avgRed>10 || avgGreen>10 || avgBlue>10) {
      sendArduinoColor(c);
      needOff = true;
    } else if (!(avgRed>10 || avgGreen>10 || avgBlue>10) && needOff) {
      sendArduinoColor(c);
    }
  }

  public void sendArduinoColor(Color c) {
    //takes the color and sets code = an array of 3 bytes with values from 0-100 (100 is the highest brightness I ever want the LEDs to be)
    //I am sure this is done correctly so I've left it out.
    constrainArduinoColor(c);
    //calls the static method from the client and passes in the byte array with the rgb values
    Client.sendCodeUSB(code);
  }
}

Arduino

#include <EEPROM.h>

int redPin = 11;
int bluePin = 10;
int greenPin = 9;
//declared here for debugging ease of access
int r;
int g;
int b;
//are we going through the Spectrum Cycle loop for the first time?
boolean sclFirstTime = true;
//temporary initialization
boolean waiting = true;
int index = 0;
const int LENGTH = 20;
byte profile[LENGTH];
byte avProfile[3];
byte killTerm = -100;
byte audioVisualizationCode = -22;

void setup() {
  //start the serial communication
  Serial.begin(9600);
  //initialize the profile to all -1 values
  clearProfile();
}

void loop() {
  //if there is nothing over serial and nothing is in the EPROM, we are waiting
  if (!eepromProfileAvailable() && Serial.available()==0){
    waiting = true;
  } else if (eepromProfileAvailable() && Serial.available()==0) {
    //if there is something in EEPROM and nothing over Serial, we aren't waiting and load the EEPROM profile
    waiting = false;
    loadEEPROMProfile();
    //load the EPROM color profile into the array
  } else if (Serial.available()>0) {
    //if something enters over Serial, we aren't waiting, clear the profile, and load the new profile with the serial info
    waiting = false;
    clearProfile();
    while (Serial.available()<LENGTH) {}
    // Wait until there are LENGTH number of Bytes waiting
    for(int n=0; n<LENGTH; n++) {
      profile[n] = Serial.read(); // place them in the profile.
    }
    Serial.flush();
    //clear out the serial to prep for next time
  }
  //if we have a profile (either from EEPROM or Serial), save it, check for special codes (if found, apply the effect),
  //and if there are no special codes, the profile is a static color profile, and set LEDS to that.
  if(!waiting) {
    saveEEPROMProfile(); //omitted but working
    if (profile[0]==101) {
      while (Serial.available()==0)
        spectrumCycle(); //omitted but working
    } else if (profile[0]==-22) {
      audioVis();
    } else {
      anodeSafteyOut(redPin,profile[0]);
      anodeSafteyOut(greenPin,profile[1]);
      anodeSafteyOut(bluePin,profile[2]);
    }
  }
}

//a bunch of preset color effects omitted
//we are using a common Anode LED, so low value is 255 instead of 0
void anodeSafteyOut(int pin, int c) {
  if(c>=0) {
    c = 255 - c;
    //as a safety measure because there are no resistors (yes, I know im breaking an intergalactic law by not using any), make sure we aren't writing something too bright
    if (c>=155) {
      analogWrite(pin,c);
    }
  }
}

//wait for 3 bytes to come in, assign them to the avProfile, set the colors to that, do it again
//THIS IS PROBABLY WHERE THE PROBLEM IS
void audioVis() {
  while (Serial.available()<3);
  for (int k=0;k<3;k++)
    avProfile[k]=Serial.read();
  anodeSafteyOut(redPin,avProfile[0]);
  anodeSafteyOut(greenPin,avProfile[1]);
  anodeSafteyOut(bluePin,avProfile[2]);
  audioVis();
}

So that's a lot, and I've left a lot out that I didn't think was necessary, though I'm happy to provide the full programs if needed. Thanks for your help!

Here is constrainColorArduino:

public void constrainArduinoColor(Color c) {
  int r = c.getRed();
  int b = c.getBlue();
  int g = c.getGreen();
  int r1 = (int)(r*0.4);
  int b1 = (int)(b*0.4);
  int g1 = (int)(g*0.4);
  byte r2=0;
  byte b2=0;
  byte g2=0;
  for (int k=0; k<r1; k++)
    r2++;
  for (int k=0; k<b1; k++)
    b2++;
  for (int k=0; k<g1; k++)
    g2++;
  System.out.println(r+" "+g+" "+b);
  System.out.println(r1+" "+g1+" "+b1);
  if (r2>100) r2=100;
  if (b2>100) b2=100;
  if (g2>100) g2=100;
  if (r2<0) r2=0;
  if (b2<0) b2=0;
  if (g2<0) g2=0;
  System.out.println(r2+" "+g2+" "+b2);
  code[0]=r2;
  code[1]=g2;
  code[2]=b2;
}

Answer 1

My thought is that, because the audio visualizer sends color profiles dozens of times per second, the Arduino might be getting overloaded with data. Is that possible?

Close, but the source of the problem is probably this line

  Serial.flush(); //clear out the serial to prep for next time

Because your audio visualizer keeps sending data, it's quite likely that by the time you get around to claiming a record, at least some part of another one (or more) has also arrived. When you discard that with flush(), you are probably most of the time not going to be discarding exactly a record's worth of bytes, but rather some arbitrary number more or less. This introduces a synchronization problem.

You've already had some advice on solving synchronization problems. There are a variety of workable methods, binary and otherwise. One simple one that works in a lot of situations is to encode bytes textually in hex and terminate the record with a newline. It does have just over a factor of two overhead, but it has the debug/test advantage of being human readable, and you can basically cover the overhead by switching from 9600 to 19200 baud.

Answer 2

Your idea that there's a synchronization issue seems plausible. Whenever any byte gets skipped, color assignments shift from RGB to GBR or BRG, etc.

While sending a start code before the color profile is a possible approach to fixing the problem, it may be that occasionally one of the RGB values matches the start code, which could result in temporary desynchronization.

For an alternative method that doesn't require a start code, consider encoding the data into four bytes. In the first byte, send the high two bits for RGB; in the second, the next two bits of each color; etc. Set the low two bits of each byte to 11, 10, 01, 00 for first, second, third, fourth.

That is, each byte you send would be encoded BBGGRRdd where the capital letters are appropriate bits from RGB brightness levels and dd indicates which pairs of bits are being sent.

Here's some (untested) code that implements the decode:

byte code = Serial.read();
byte shif = 2*(code & 3);
byte mask = ~(3<<shif);
code >>= 2;
for (byte i=0; i<3; ++i) {
  profile[i] = (profile[i] & mask) | ((code & 3)<<shif);
  code >>= 2;
}

The following shows a function sendData3() that encodes three bytes from an array of RGB data into four bytes to be sent to the Arduino. It also shows a convenience function sendDataRGB(). A call like sendDataRGB(100, 0, 42); puts three R, G, B values into a three-byte array and calls sendData3() to send it off. [Substitute as appropriate in place of Serial.write]

void sendData3 (byte *dataIn) { // dataIn[] has R,G,B levels
  byte code;
  byte mask=3;          // mask for low 2 bits of a byte
  for (int b=4; b; ) {
    --b;
    byte shif = 2*b;    // For moving bits to align with mask
    code = 0;
    for (int k=3; k; ) { // Loop to handle dIn[2], dIn[1], dIn[0]
      --k;
      code |= (dataIn[k]>>shif) & 3; // Pick off 2 color bits
      code <<= 2;            // Roll up
    }
    Serial.write (code | b);    // Send encoded bits
  }
}

void sendDataRGB (byte R, byte G, byte B) {
  byte dataEg[] = { R, G, B };
  sendData3(dataEg);
}