Many of us have occasion to argue against particular types of prejudice. But John Stuart Mill unmasks all prejudices in the 6th paragraph of the “Introductory” to On Liberty:

But though this proposition is not likely to be contested in general terms, the practical question, where to place the limit—how to make the fitting adjustment between individual independence and social control—is a subject on which nearly everything remains to be done. All that makes existence valuable to any one, depends on the enforcement of restraints upon the actions of other people. Some rules of conduct, therefore, must be imposed, by law in the first place, and by opinion on many things which are not fit subjects for the operation of law. What these rules should be, is the principal question in human affairs; but if we except a few of the most obvious cases, it is one of those which least progress has been made in resolving. No two ages, and scarcely any two countries, have decided it alike; and the decision of one age or country is a wonder to another. Yet the people of any given age and country no more suspect any difficulty in it, than if it were a subject on which mankind had always been agreed. The rules which obtain among themselves appear to them self-evident and self-justifying. This all but universal illusion is one of the examples of the magical influence of custom, which is not only, as the proverb says, a second nature, but is continually mistaken for the first. The effect of custom, in preventing any misgiving respecting the rules of conduct which mankind impose on one another, is all the more complete because the subject is one on which it is not generally considered necessary that reasons should be given, either by one person to others, or by each to himself. People are accustomed to believe, and have been encouraged in the belief by some who aspire to the character of philosophers, that their feelings, on subjects of this nature, are better than reasons, and render reasons unnecessary. The practical principle which guides them to their opinions on the regulation of human conduct, is the feeling in each person’s mind that everybody should be required to act as he, and those with whom he sympathizes, would like them to act. No one, indeed, acknowledges to himself that his standard of judgment is his own liking; but an opinion on a point of conduct, not supported by reasons, can only count as one person’s preference; and if the reasons, when given, are a mere appeal to a similar preference felt by other people, it is still only many people’s liking instead of one. To an ordinary man, however, his own preference, thus supported, is not only a perfectly satisfactory reason, but the only one he generally has for any of his notions of morality, taste, or propriety, which are not expressly written in his religious creed; and his chief guide in the interpretation even of that. Men’s opinions, accordingly, on what is laudable or blameable, are affected by all the multifarious causes which influence their wishes in regard to the conduct of others, and which are as numerous as those which determine their wishes on any other subject. Sometimes their reason—at other times their prejudices or superstitions: often their social affections, not seldom their antisocial ones, their envy or jealousy, their arrogance or contemptuousness: but most commonly, their desires or fears for themselves—their legitimate or illegitimate self-interest. Wherever there is an ascendant class, a large portion of the morality of the country emanates from its class interests, and its feelings of class superiority. The morality between Spartans and Helots, between planters and negroes, between princes and subjects, between nobles and roturiers, between men and women, has been for the most part the creation of these class interests and feelings: and the sentiments thus generated, react in turn upon the moral feelings of the members of the ascendant class, in their relations among themselves. Where, on the other hand, a class, formerly ascendant, has lost its ascendancy, or where its ascendancy is unpopular, the prevailing moral sentiments frequently bear the impress of an impatient dislike of superiority. Another grand determining principle of the rules of conduct, both in act and forbearance, which have been enforced by law or opinion, has been the servility of mankind towards the supposed preferences or aversions of their temporal masters, or of their gods. This servility, though essentially selfish, is not hypocrisy; it gives rise to perfectly genuine sentiments of abhorrence; it made men burn magicians and heretics. Among so many baser influences, the general and obvious interests of society have of course had a share, and a large one, in the direction of the moral sentiments: less, however, as a matter of reason, and on their own account, than as a consequence of the sympathies and antipathies which grew out of them: and sympathies and antipathies which had little or nothing to do with the interests of society, have made themselves felt in the establishment of moralities with quite as great force.

People often have feelings about the conduct of others. If person A is able to insist that another person B should do X because not doing X makes A feel bad, this could imply a tyrannical domination by A of others like B unless the scope of A’s legitimate sphere for concern and insisting about B’s and others’ actions is limited somehow. It won’t work to give each person a sphere of decisive influence over all the things that shehe cares about because those spheres of decisive influence would be overlapping–meaning they wouldn’t be spheres of decisive influence at all. Somehow, the legitimate sphere of decisive influence for each person must be delimited in a way that doesn’t lead to too much overlapping. 

Of course, it is possible to have decisions made jointly by several different people. But in cases of stubborn disagreement, there must be some mechanism for deciding–even if that mechanism is flipping a coin. Dividing up as much as possible into separate spheres of decisive influence in which one person is dominant has been a very useful strategy to minimize the need to use other social choice mechanisms like voting that tend to constantly raise the possibility of conflict. John Stuart Mill’s advocacy of both civil and social liberty can be seen as advocating an equitable division of spheres of decisive influence more or less in line with the physical body of each person and (for the most part) the fruits of each person’s labors.  

Economists have done some work on subtle social choice mechanisms that are not in such common use as the division of spheres of decisive influence and voting. In the case of each such social choice mechanism, a key question is how it could be made a staple of everyday social choice as voting, flipping a coin or the division of spheres of decisive influence are. Or to put it another way, a social choice mechanism has only fully come into its own as a general purpose method if children on the playground often use it to resolve their disputes.