The word has two senses; only one is pejorative
It seems he [John McWhorter] is not aware that 'prejudice' does not necessarily require a negative attitude towards that concerning which one is prejudiced and is therefore actually not an ideal replacement for 'racist'. Surely, 'bigoted' would be better.
I agree. 'Prejudice' admits of pejorative but also non-pejorative uses. 'Bigot' does not. Note also that racial prejudice is not the only kind. That is why a careful writer and speaker does not use 'prejudiced' sans phrase, but always adds the appropriate qualifier unless the context makes the addition unnecessary.
As for 'prejudice,' it could refer to blind prejudice: unreasoning, reflexive (as opposed to reflective) aversion to what is other just because it is other, or to an unreasoning pro-attitude toward the familiar just because it is familiar. ("My country right or wrong.") We should all condemn blind prejudice. It is execrable to hate a person just because he is of a different color, for example. No doubt, but how many people in fact do that? How many people who are averse to blacks or whites or Asians are averse because of their skin color as opposed to their behavior patterns? Racial prejudice is not, in the main, prejudice based on skin color, but on behavior.
'Prejudice' could also mean 'prejudgment.' Although blind prejudice is bad, prejudgment is generally good. We cannot begin our cognitive lives anew at every instant. We rely upon the sedimentation of past experience. Changing the metaphor, we can think of prejudgments as distillations from experience. The first time I serve my cats whisky they are curious. After that, they cannot be tempted to come near a shot glass of Jim Beam. They distill from their unpleasant olfactory experiences a well-grounded prejudice against the products of the distillery. They know what is good for them and what isn't.
My prejudgments about rattlesnakes are in place and have been for a long time. I don't need to learn about them afresh at each new encounter with one. I do not treat each new one encountered as a 'unique individual,' whatever that might mean. Prejudgments are not blind, but experience-based, and they are mostly true. The adult mind is not a tabula rasa. What experience has written, she retains, and that's all to the good.
So there is good prejudice and there is bad prejudice. The teenager thinks his father prejudiced in the bad sense when he warns the son not to go into certain parts of town after dark. Later the son learns that the old man was not a bigot after all: the father's prejudice was not blind but had a fundamentum in re. The old man was justified in his prejudgment.