I think that the people who are having this reaction are not reacting to the person, but to the symbol, which is the downside of having somebody be a symbol.

Which means I'm going to be sympathetic. To people who have issues directly related to the British Empire, the British monarch is a symbol of those issues.

I also suspect she had another source of influence that nobody ever thinks about. Garden parties and dinners...where she controlled the seating chart *grin*.