I haven't played a build without destruction as my prime damage dealer. (I like mages)
The only downside is that shit gets boring pretty fast in my experience. I don't normally carry more than ten potions (role playing) so loading up on all the Fortify Destruction potions I need sorta breaks that immersion for me. I just enchant my gear with cost reduction. I love the alchemy/enchanting loop for this because I can enchant my ring/necklace with whatever percentage reduction I want and it frees up my other slots, since I often run around without armor or without an item in every slot. I think I settled on 85% for my Telvanni Wizard, which coupled with my dual-cast everything, bad aim, and excessive paralyzing and summons feels decent. I run out of mana after a few minutes of heavy casting, but I'm not fretting over running OOM after every bandit encounter and running like hell (200 health at level 55).
I highly recommend picking up illusion as a secondary means (conjuration tertiary) just because of how boring destruction gets after a while and mixing it up makes things so much better. I'm left sitting there thinking what spells would be more fun to use right now instead of what would be most effective.