Everyone hates him because he's a wizard, even though he was 800 miles away when the earthquake happened. He's also responsible for the destruction of their village, because he pretended to be the wizard. The more people believe that he is the wizard, the stronger the power he delivers to the demon god. This is why they destroyed their village in the first place: because they believed that he was the wizard and that they could trust him. The only way to stop this is to stop trying to change the "demon god order" .