Politics is a tricky business. You'll never find anyone who agrees with all of your views on everything. The best you can hope for is to address issues that you care about, and hope there is enough political support around your position on that issue. More often than not, that support may come from people you have severe disagreements with on other issues. If you had to abandon your position on any issue that someone reprehensible agreed with you on, or because someone did something reprehensible at a political rally you were there to support, you'd be left with nothing at all. Should everyone have left Trump rallies during the election because a small number of yahoos showed up? If they did, you'd be giving more power to these fringe groups than they could ever dream of! These fringe groups would be the ONLY groups out there demonstrating!Again, if I was at a protest that was suddenly rife with open racism, I would leave no matter how important I felt the cause. The fact that those people stayed showed they were quite comfortable hanging around with open white supremacists.
p.s. The cause itself was hero worship for those who fought for slavery. Not exactly a good fall-back argument.
p.s. The basis for the Civil war and the motivations of those who fought for the South and/or have been memorialized are a lot more complex than your summary. Some others, like Aardie, have already posted good material on this, which I would encourage you to read.