I absolutely agree that racism and anti-Semitism are not the same thing.
But nobody should use one to diminish the other. When a kid walks into their school bathroom and sees swastikas scrawled all over the walls, that is significant and traumatic.
And while I acknowledge that I have white privilege as a mixture of European and Ashkenazic Jew, there are still white supremacists who would consider me Jewish enough to send to a camp.
It is the tribe of Israel, and the reason conversion is so hard is because it is also a form of adoption.
(And, of course, as you rightly acknowledge, it is possible to be Black and Jewish at the same time).