I've also read a huge amount of fanfic where the Dursleys are reviled, hurt and otherwise "get what's coming to them." - usually from Snape.
The fanon perspective is that they abuse him verbally, physically and emotionally, although apparently Lucius is the one pegged for sexual abuse. They also treat him like a slave in return for starvation rations and bad clothing.
And there are grounds for some of this. In SS/PS, Harry sleeps in a cupboard under the stairs, and is confined there for punishment. It isn't until he goes to Hogwarts that he is free to eat to satiation, and he wears Dudley's oversized and wornout clothing. In CoS, he is first made to complete a series of strenuous chores before being fed and later is locked in his bedroom and seemingly starved. In Book three, he has to bargain to get permission to go to Hogsmeade, and loses that possibility. In Book 4, he's forced to eat less than Dudley, who is on a reducing diet.
On the other hand, in book 5, his only restriction seems to be that he cannot come home later than Dudley.
However. I will admit emotional neglect, and possibly abuse. They consider him an unwelcome burden and make sure that he knows it. He has not been hugged or cared about since his parents died. Hating what his parents were, they do their best to stamp it out by lying to him both outright ("Your parents died in an accident.") and by omission (never saying he was a wizard.) And forcing a child to sleep in a cupboard when there is a perfectly usable bedroom is cruel and possibly abusive.
On the other hand, I'm not so sure about the food thing. Except for the diet in GoF, and an unwillingness to purchase ice cream for him in public, there's no sign that the Dursleys deny Harry food. It's more that Dudley feels free to eat Harry's share, too, and his parents allow him. Okay, his parents allow him everything, which is doing him a tremendous injustice. Even in CoS, when they seem to be starving Harry - more likely, they were starving *Hedwig* in that they were giving Harry just about enough for him, and, if they even thought about her at all, assumed he wouldn't give up his own portion for her. They're rather selfish people, so it's easy to for them assume Harry would be equally selfish.
As for physical abuse - note that Harry is not afraid of them. He's rude. He talks back. He adopts angry and sullen tones of voice. He does what he is told, but sometimes because of a bargain, and otherwise because they do let him live in their house. In CoS, especially, he had nowhere else to go. This changes after CoS, because Harry knows he will always be welcome at the Burrow. And in Book 5, not only does Harry have Grimmauld Place, which might very well *be* his (even if Lupin has a life-interest in it, I can't see him ever denying Harry the use of it) but he still has the Burrow. And, he now knows that his aunt won't let his uncle throw him out, so they can't even hold that over his head. Abused children don't talk back to their abusers, and if they duck blows, they know they'll just get another one. Petunia aimed her frying pan, missed and didn't try again. That she tried it once makes her unpleasant; that she didn't try again says they don't beat Harry up.
More than that - after Dobby/Harry screwed up his deal with the Millers, a physically abusive Vernon Dursley would have beaten Harry within an inch of his life. He *didn't*. His reaction was bad - locking a twelve year old boy in his room, potentially forever, and denying him schooling are both bad things - but he never *hurt* Harry, and Harry isn't afraid of it. And note: this occured after he learned that Harry can't use magic during the summer, so he wasn't afraid of him, either.
The chores he was expected to do in CoS came as punishment - a rather extreme one since Harry was not permitted to eat until they were done, and while none of them were out of line for a twelve year old, there were a lot of them. If, however, nothing had occured, he would have had his lunch and lousy dinner at a normal time and otherwise been free. He did cook breakfast once, and it may have been a daily thing, but that's hardly cruel - except as Dudley eats it all instead of him.
They are deeply prejudiced. That can be the only reason they would object to Hogwarts, because, look - they don't pay his tuition or school fees, they don't contribute to his school supplies at all, and they no longer have to feed him other than the summer holidays. For ten months out of the year, maybe more, he's someone else's responsibility. They should be thrilled, except that it feeds the wizarding aspect of his life, and they hate that.
And now I'm wondering. Not about Vernon, who is simply a closed minded, prejudiced, horrible man. But, rather, Petunia. See, we can pretty much figure that the Potters are all dead. Given how Wizarding families seem to be dying off anyway, and the help Voldemort gave them, this is not surprising.
But. Harry also saw people with his eyes in the Mirror of Erised - he saw his mother's family, too. And that's making me wonder - where are the Evanses? Where are the cousins? The great aunts and uncles?
Or did Voldemort attack them, too? Does Petunia remember this? How did her parents die? She's probably not even forty yet - where are her parents? These are things, I know, that may never be answered, but I do wonder.
But, if Petunia knows about the first war, and witnessed atrocities committed by Death-Eaters, she would have every reason to hate and fear wizards, and very little to not do so.
Except she seemed totally unaware of it when Lily and James died. So. I'm confused.