Whenever the sim casts a spell on himself or another sim...
On one lot from Moon Base Charlie II (a world), when my witch sim uses a charm on himself or another sim, they run outside to the fenced in garden area to do it. If the gate of fence is locked to keep the zombies out during a full moon, then they don't throw the charm because they can't get through the fence.
Does this have something to do with the gate of the fence being closer to the mail box than the front door? I tried moving the garden, but the gate is still closer than the front door. Humm. Okay, I'm gonna try that but I'll post this anyway just in case someone else has this strange behavior.
I think it is rather caused by the need to find a suitably large spot for the jig this interaction requires. And if you start the spell indoors, the game will look for an empty room inside, which in your case is the fenced in area, because most probably the other rooms in your house are either too small or to cluttered to offer enough space for the jig.
I bet if you start the spell outside in a narrow area, the game will rather direct your Sims to another spot on the lot that is still outside any walls or fences and only if it fails to find one outside will look for alternatives inside. And fenced-in gardens are technically inside (a room, that is).