You can only delete a module that is inactive.
Deleting a module removes its files from your site. Deactivating it is enough to hide it: a deactivated module keeps its settings and items, and you can turn it back on later.
The delete button doesn't appear for core modules that are locked, or when the web server can't write to the module's directory.