The USB devices are created dynamically. You might find this question and answer from the SE Raspberry Pi helpful: How to force rescan of USB serial devices?.
You can use the dmesg(8) command on the Pi to see the messages generated by the USB sub-system. Watch the output to see if there are errors associated with the Arduino dropping off the bus. This is what the output will look like (roughly, this is from an Uno connected to a Raspbian system), note the last line which is a disconnect from unplugging the Uno from the Pi. Resetting the Uno did not do anything that the Pi saw. I would guess that that is because the reset button on the Uno doesn't reset the 16U2 (the processor handling the USB port).
[ 459.936719] usb 1-1.3: new full-speed USB device number 5 using dwc_otg
[ 460.051182] usb 1-1.3: New USB device found, idVendor=2a03, idProduct=0043
[ 460.051212] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=220
[ 460.051229] usb 1-1.3: Product: Arduino Uno
[ 460.051245] usb 1-1.3: Manufacturer: Arduino Srl
[ 460.051262] usb 1-1.3: SerialNumber: 75431343334351F061E2
[ 460.088018] cdc_acm 1-1.3:1.0: ttyACM0: USB ACM device
[ 460.089250] usbcore: registered new interface driver cdc_acm
[ 460.089270] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[ 534.212365] usb 1-1.3: USB disconnect, device number 5
From reading that question and its answers it appears that some versions of the Leonardo may have problems that can cause this behavior.
Another thing that might be worth looking into is the power supply for the Leonardos. If they are being powered by the Pi through the USB interface that might be pushing the capability of the Pi to provide power.