"Am I doing something wrong?"

If you are, so am I. Exactly the same thing happens to me. 

Ahh, now I remember why I hate hot keying magic. Yes you're right, there isn't a way to use hot keys with this build then. Unfortunate, but other shield mages get around it.

Well... come to think of it, there is a way, but it'll take you a bit far from the build. Two hotkeys, one for potions with both Restore Health and Restore Stamina, and the second for a Staff of Healing Hand  But again, it'll require Alchemy which you don't use 

Bryn just hates everything haha

Nice build, And awesome level by level video. Makes me wanna take the dust off of the armor and use it right. +1 from me.

I used sarcasm. When you comment badly about a build because of their armor choice just for looks makes you sound ridiculously dumb. Remember these are guidelines to a build, you can change it how you like, just don't drown the build with stupid comments.

I used "haha" so I wasn't serious.