For this you need to modify the item_albums.tpl files for the jrAudio and jrVideo module to suit your needs - those are "custom" templates for those modules.
So videolightbox is not something we use in Jamroom. Make sure you are using Firebug or your Javascript console to check for Javascript errors in your page.
I just tested this here on a new blog, and I'm not seeing any issues (see screenshot). I see you are wrapping your iframe in a div - go into the System Core -> Quota Config and make sure "div" is an allowed HTML tag and save.