A lot of the "scariness" of Varg was due to the record companies, etc. because he was 'black metal' he was supposed to be evil and scary, it was even them that told him to dye his hair black because it would make him appear more evil.
Plus besides, if he hated music he would have just stuck with burning churches instead of releasing seven full-length albums.