In his radical book Fair Play (p. 16), Steven Landsburg puts forward this radical idea:
… we should care about other people’s liberty as well as our own.
In On Liberty, Chapter IV, “Of the Limits to the Authority of Society over the Individual” (paragraph 12), John Stuart Mill puts forward the same idea, and explains just how radical it is:
But the strongest of all the arguments against the interference of the public with purely personal conduct, is that when it does interfere, the odds are that it interferes wrongly, and in the wrong place. On questions of social morality, of duty to others, the opinion of the public, that is, of an overruling majority, though often wrong, is likely to be still oftener right; because on such questions they are only required to judge of their own interests; of the manner in which some mode of conduct, if allowed to be practised, would affect themselves. But the opinion of a similar majority, imposed as a law on the minority, on questions of self-regarding conduct, is quite as likely to be wrong as right; for in these cases public opinion means, at the best, some people’s opinion of what is good or bad for other people; while very often it does not even mean that; the public, with the most perfect indifference, passing over the pleasure or convenience of those whose conduct they censure, and considering only their own preference.
Part of the problem is in the limitations of the golden rule: “Do unto others as you would have them do unto you.” That is all well and good if they have the same preferences, but not if they want something different from what you would want in the same circumstances. Until further advances in technology, we are shut out of knowing directly what the world looks like from inside someone else’s mind, and have to guess based on how we would feel. But that sometimes steers us badly off target. Giving everyone personal liberty is a safeguard against our blind meddling.
Of course, applying the golden rule at a meta-level would say “I want liberty, so I should give others liberty as well.” But there are many times when liberty is a higher law than the applying the golden rule at the detailed level.