The freedom philosophy is intimately related to ethical, political, and methodological individualism. Therefore, the philosophy should be expected to detest any kind of collectivism — and particularly its “lowest, most crudely primitive form” — even in its nonviolent manifestations.
To put it more concretely, if a libertarian observed a growing propensity to embrace (nonviolent) racism, that person, qua libertarian, ought to be concerned. Why? Because that attitude and resulting conduct can be expected to eat away at the values conducive to libertarianism. It’s the same sort of reason that a libertarian would be concerned by, say, a growing acceptance of Keynesian ideas, even though merely holding and advocating those ideas does not require the use of force.