Lyndon Johnson had the right idea. However, his program was not going to pass Congress if it benefited black people only. So white women were included in affirmative action because they, too, had been adversely affected by discrimination, though, clearly, not to the degree that blacks were and are.
I don't know how old you are, but in the late 60s and early 70s when Affirmative Action was implemented, if you hired a black man, that was "one point." If you hired a white woman, you got "one point." Black women were in demand because we were "twofers."
We also need to remind people that affirmative action does not mean preferential treatment. It means considering those beyond "the usual suspects," i.e., white males. The term has been hijacked by Republicans and racists for generations now to make white people think that the "unqualified people" they so fear are getting jobs that white people "deserve."