I have been trying to make my Leonardo board to work as a USB device. So I started with compiling the open source LUFA USB library demo "MassStorage" for the Leonardo board settings. Since I'm using Mac OS, I downloaded the "AVR Tools" software to upload the hex file to the board(I didn't want to use the ISP protocol to flash. Because it is not possible to remove the chip from the board to make the correct ISP circuit connection). Anyway I was managed to flash the board. But afterwards it is not detectable by the computer. The serial port is not detectable. This proves it flashed something. Afterwards I flashed it again with the original bootloader file and it was identified as before. I'm a beginner to Arduino programming and forgive me if I'm asking elementary question. What am I doing wrong?
Cannot say for sure, but if you was able to flash something, then loose functionality, but was still able to flash it back, then there is possibility, that you flashed the application part (with would-be-bootloder code), not the bootloader part (and the old bootloder in bootloader part then enabled you to flash back something sensibe).
bootloader is resposible for saving application in memory in process of flashing and running the application after that, or in case you just connect the board and do not start flashing it just after (maybe software/transmit) reset.
if you save bootloader to application part of memory, it is highly possible, that it would not do anything good, as the original bootloader would still run first and setup your board and the application (with new bootloader code) would try something futile (and maybe have bad addresses already, as it is suppose to be runned in bootloader place, not in application).
There are two parts of flash memory in arduino - one for application, other for bootloader and they have different use, even if they are just different parts of one big flash area. So make sure, what you actually updated and how was the hex compiled (for which address)