There are a lot of factors at play in controlling someone's mind. They require a lot of self-doubt to exploit, and you need to be able to predict their actions well enough to stay a step ahead of them. Essentially what you're doing is using illusion magic to trick them into following your commands, and such a process is exceptionally difficult to pull off without a lot of planning and time dedicated to it.

I could never control an entire army, no matter how much time and knowledge I have of each individual - it would be entirely too much strain on me. Most individuals aren't powerful enough to take that big of a risk either, and often the ones that are would be exceptionally difficult to control.

[See: King Garon and his wild, inexplicable behavior.]