"I just made that statement up, but I bet it's true." - You can't do that. I'm sorry. You can't base an argument around something you just made up, but which happens to fit your worldview. If you want your worldview to matter in a debate you have to base your worldview on facts. Not introduce made up "facts" which happen to fit your worldview.
And really, I'm pretty skeptical of the idea that working hard is something that varies a lot between people. Everyone is hard working when the need to be, and are motivated and a little bit lazy when they're not. I know, I know, you're an exception, you work hard all the time, good for you. While you're at it, pull the other one, it's got bells.
The fact is, I don't think it's a good idea to base social policy on some bullshit work ethic based morality invented by the religiously oppressive puritan movement from the 1600s.
"If this could be verified is the statement less valid than white people have more advantages than colored people, or the middle class have more advantages than the working class?" - The relative validity of those two statements, assuming the former one is even true (spoiler alert: It's not), is dependent on the strength of the statistical correlation between the two. And seriously, look around you, unless you happen to think that certain races of people are consistently "harder working" than others, it's pretty clear that the stronger correlation is the one about white = privilege.