It all starts with Vernon's father who was not a Dursley but a very dangerous man who was sent to a school for incurable criminal boys for being a psychopath.
After leaving school, he killed his own parents in revenge and was arrested, but after being released, he raped Marge's mother and then was imprisoned again (but for the rest of his life).
The three real Dursleys never found out that Vernon was the rapist's son and he grew up as a Dursley (although the dangerous criminal is his father, Vernon is nothing like him).
Many years later, Vernon and Petunia had a son, but unbeknownst to anyone (including Dudley himself), he inherited a great deal of darkness from his biological grandfather.
After he was attacked by Dementors, it led him to see what he was becoming and changed before it was too late (had he not been attacked, he would have ended up becoming like his grandfather, an incurable criminal).
Vernon's father was also the reason they couldn't have magical offspring, as magical genes could never survive the DNA of a very dangerous Muggle.
