Time

Add to the wing-nut factor the apparent arrogance of the Tories now that they're in power. The secrecy and unaccountability of the federal Tories almost rivals that of their Alberta counterparts. Though this doesn't annoy Albertans (unless, of course, it's a non-conservative party doing it to them) but it will start to grate on the rest of Canada. The longer the arrogance lasts, the more voters it will turn away. Again time is not Stephen Harper's friend.
Time is also against Harper in that he can only keep people like Rob Anders and Myron Thompson muzzled for so long. Given enough time, the probability one of these guys is going to wander in front of a live microphone and start talking approaches 1. Any of the more nutty members of Harper's caucus speaking their "mind" to the media will do harm, possibly irreparable, to the party. This is probably why Harper has them in what appears to be a witness protection program where they're seldom seen and even more rarely heard. It's also likely why the Tories protected Anders' and Thompson's seats from being contested, payment for keeping their mouths shut.
So time is working against the Tories. Much of what the Tories say and do is out of sync with most of Canada and this will catch up with them. They have to quit governing like the support that the received from Canadians was due to vast agreement with their policies. The Tories would do well to remember as the votes that went to them were less for them and more against the Liberals. At some point in time, those votes will return to the Liberals. How fast they return is all the Tories can control. At the rate that the Tories are going, it won't take too much time at all.