People think life is like a video game. There's not an objective "doctor skills" ranking for a white guy that's manifestly at a 95/100, but some black guy gets to be a doctor instead even though his ranking is only a 90/100, so that white guy has to go be a barista.
In particular, this: "Nothing should matter beyond GPA, SAT/ACT score, entry essay/writing sample, and recommendation(s)." is a terrible idea. These things don't measure the best candidates, "best" is a mostly subjective determination, and their measurements are clearly influenced by the economic and racial inequalities of the US.
If you focus on just SAT/ACT and writing samples, you're going to undervalue candidates from certain backgrounds, and because of America's racist history (and present), a disproportionate percentage of those candidates are going to be black. Affirmative action is a clumsy way of addressing that. That people don't understand this fairly straightforward fact is very confusing to me. If you think affirmative action doesn't work, great. I basically agree. If your solution is "Get rid of it and don't do anything to replace it," then that's worse.
For a handful of programs and institutions (I am thinking of places like MIT), you can probably just focus on test scores. But I teach college history. I don't want that to be the make-up of classes I teach. There's a lot of other things that go into college and life success.