One man in my family tree back then was, as I recall being told, a doctor. He never enlisted in the Confederate army, which was something of a source of public shame. I don't know his full reasons, but when asked why he hadn't fought (or served as a field doctor), he apparently would answer, "Well, I wasn't mad at anybody."
This was met with scoffs, scorn, and eyerolls. Of course, an even better reason would be to recognize that the cause was white supremacy. But when deeply limited people are called to war, the world would be a better place if more people would refuse, who at least don't actively believe in the war cause.
"I'm not mad at anybody" is a perfectly good reason not to wage war, and I wish it would were felt, and cited, more often.