I think the “proper” way to simplify it is would’ve, which is pronounced the same as ‘would of’

A lot of mistakes have just become incorporated into the language in the past. Maybe ‘would of’ is just too blatantly wrong for that to ever happen though

Maybe not really a ‘mistake’, more of a normal shortening but my personal favorite english-ism is “bye” being descended directly from “god be with you”. People just kept collapsing it more and more over time.

Edit: also “a pease” -> “peas” -> “a pea”