Do all the processing on the Arduino, and supply the results to Matlab over serial.
I'm not 100% sure about your particular encoder, but generally encoders talk in "Grey Code". This is a series of bits which allow you to know the direction the encoder has turned.
There are a number of libraries and tutorials for Arduino dealing with encoders.
The one I use comes from here:
It would be best to use a library with interrupt routines on pins 2 & 3 (these support external hardware interrupts), so whenever the pin gets a signal from the encoder, the Arduino can act on it immediately. If you do not use these pins, and have time-consuming processing in the loop(), it's easy to miss 'ticks' from the encoder.
// Rotary Encoder (Interrupt Driven)
#define ENCODER_PIN_A 2 /* must be pin2 for interrupt */
#define ENCODER_PIN_B 3 /* must be pin3 for interrupt */
// Gentlemen, start your encoders
unsigned long start_time = millis();
unsigned long now = millis();
unsigned int encoder_value = encoder.read();
// Do whatever angular velocity processing is needed here
unsigned long ticks_per_ms = encoder_value / (now - start_time);
// Send the result back to Matlab
I think for your encoder the "A" and "B" outputs are where the grey code will be coming from. These would be connected to Arduino pins 2 & 3.
Obviously the code above is not a complete solution, and needs to take care of edge-conditions like overflow of millis() after some time, etc. etc.