These people miss the fact that The Handmaid's Tale is about one very specific thing: What white supremacy does to white women.

In the epilogue it specifically states that the fertility crisis is not a drop in the human birthrate, it's a drop in the Caucasian birthrate.

When people use the symbology without understanding that basic point, it's very easy to come over as racist. (And I suspect some of them DO and don't care).