Sometimes the context is crystal clear. For instance, I would have no shred of doubt in opposing a wealthy donor's renaming my college as the "John Birch, George Wallace and Andrew Jackson Memorial School" and erecting colossal statues of them.
On the other extreme, while I do fault the many Democrats like Obama for being late to accepting gay rights, I wouldn't refuse to shake their hand or tear down a statue of them over this.
In between these poles, there's a ton of gray area and context at play. How bad were the people, given their time? Is the status quo that their statue exists, or is it something new that's being proposed? Is the monument to them itself historically meaningful or aesthetically special? What role is the monument playing today, and what framing, if any, is it given? After all, I'm assuming none of us would visit the Smithsonian Museum of African-American history and destroy their exit exhibit that depict past racism, because the framing changes its current role.
Sometimes people argue that we cannot hold figures from the past to the moral standards of today. For instance, all the others who owned slaves. But while the rhetoric and norms of racial equality at the time were certainly very different than today, there were thousands of white people at the time who felt slavery was abhorrent. (There's a fascinating profile the New York Times ran the other day of one forgotten activist.)
As I like to mentioned, George Washington went so far as to pay to have advertisements placed describing a runaway teenage slave, to attempt to catch her and forcibly bring her back to finish her life of violence-induced servitude, permanent separation from friends and family at a moment's notice, and absence of any recourse from rape. This not genteel, gentlemanly acceptance of the conditions of the day--it was direct and brutal.
But even supposedly genteel slaveholding should be seen for what it is. When smart, reflective people people avert their mental powers from considering the implications of some of the things they do, they are just as morally responsible as if they had focused their mental powers on those things and endorsed them.
The intellectual and social circles around the founding fathers had constant debate about slavery; Massachusetts, one of the most politically powerful and influential states, adopted a constitution in 1780 (in my hometown, Cambridge, what what!) whose language about the equality of all men (though not women) created the immediate logical conclusion that slavery was unconstitutional. That is, while there were many people who were not abolitionists, the debate at the time involved them realizing the implication of their language and deciding that abolition was the natural consequence of their own moral views.
Later, there was a large faction pushing for the same language in the United States Constitution, but many of the founding fathers stopped them.
Am I really so much more moral than they were, given my time? Would I be so haughty about this if my family's wealth, and that of my friends, largely consisted of slaves? Maybe not, and in some ways I'm very "morally lucky" (lucky in a way that makes me seem moral) not to have been born into such a position. And maybe the founding fathers who opposed slavery explicitly or in practice (there were many) were just morally lucky too. But there were plenty of slave owners who freed their slaves, and plenty of children of slaveowners who chose a different walk of life, perhaps not directly confronting their own aversion to owning and managing slaves, but opt out of the system some degree all the same.
(It makes me think of Marlene Dietrich, asked in an interview once why she took such a clear political stand against the Nazis, contra her more jingoistic theater and film colleagues, and contra Hitler, who wanted her to be some kind of great Aryan star; she replied that though she didn't know much about politics at the time, it was never a complicated question for her--the Nazis were beating children. Elsewhere in the war, even ostensibly pro-Nazi German soldiers frequently infuriated their commanders by failing to fire upon innocent people when commanded to. History is full of huge amounts of opting out of immoral conduct, in the face of huge pressure to comply, although we're not very good at telling the history of ordinary people who did the right thing.)
It's a useful mental exercise to speculate about which of our current norms future generations will see as abhorrent. Eating animals? Circumcision? Would I still have a cow butchered for my pleasure if everyone around me thought I was a monster for it?
That said, I tend to think it's more valuable to add context to our memorials and monuments than to destroy them. every German schoolchild, I'm told, must visit concentration camps and come face-to-face with their country's brutal history. the presence of a concentration camp surely causes pain to many who see it or even just know it's there, but we do not oppose that kind of pain -- we see it as crucial.
Part of what makes Confederate monuments so much more tricky is that there are still so many people that celebrate them or at least treat them with respect, rather than just seeing them as disgraceful or sobering. if the country as a whole treated Confederate sites and monuments like concentration camps, I don't think you would see people clamoring to have them torn down.
The clamor is more about the present day that the past; is an attempt to force a choice from people who like to have it both ways. so many white Americans are free to be tacit, casual supporters and beneficiaries of white supremacy. I see the current clamor as a demand that the country make good on its surface position that white supremacy is wrong and must be destroyed. it's a shame that we still need to voice that demand, but the current state of injustice is the construction of generations of the same people who led the Confederacy and who have celebrated it ever since. In a sense it is they who are truly tearing down these monuments-- their poisonous betrayal of their fellows, their collusion and violence and theft, is a rot they are forcing others to deal with instead of them.