Again, you're not looking at the proper context.
Lets say I was a dictator of my own country and I insisted in keeping black people in ghettos, and white people in affluent suburbs based on just their skin colour alone, then yes, that would be highly racist.
However, if I was dictator of that same country, and I built walls and fences to keep my enemy out of my country because otherwise they would start terrorizing and killing my people, then thats no longer racist, thats just smart policy and would fall under self-defense