All depends on what I'm doing with it to be honest. I think a mod like this has more appeal for those taking screenshots than anything else. A few times I've had Aela, Farkas and Vilkas as followers. On a few rare occasions I've had all the Companions and Whelps as well as Lydia and a few of my own followers just to set up screenshots.
Game wise I find one is enough.
I can't comment on mods increaisng difficulty, for all I know 3 is still very hard with all those mods installed, but 1 at most in Skyrim vanilla is enough. They definitely slow down combat by drawing fire, but thats not always a bad thing, and hitting someone in the back when they're hitting your follower takes you out of the action a bit. Only in the very early game for me, until my charcater is established with the gear/ spells they need to get along by themself.