While you can get away with that here and there, your bin needs a proper balance of carbon-rich bedding materials and nitrogen-rich food waste.
Bedding makes the mixture more stable, absorbs moisture, buffers the pH, and adds bulk and porosity to your vermicompost to keep it aerobic.
We recommend using twice the volume of bedding to food waste during your feedings. Again, volume, not weight.
If you don't add bedding, your pH will drop, the worms may develop protein poisoning, the vermicompost can get too wet, and your bin can go anaerobic.