Say “We consistently vote very differently,” not “much differently.” But you can say “My opinion doesn’t much differ from yours.”