The US Constitution guarantees freedom of religion, but what we often need is not just freedom of religion, but freedom from religion. It is very common for religions to have prohibitions that seem a very weighty matter to believers, while violating those prohibitions seems a matter of much smaller weight to the non-believers in that particular religion. Yet, prohibitions that at first seem easy to bear for the non-believer grate more and more as time goes on. And in a pluralistic society, accommodating the prohibitions of many religions would add up to a great constriction of life.

Before things get to that point, John Stuart Mill argues that imposing each religious restriction on non-believers (or on believers unwilling to impose it on themselves) is, in any case, illegitimate. Here is what John Stuart Mill writes in  On Liberty, Chapter IV, “Of the Limits to the Authority of Society over the Individual” paragraphs 13-16:

The evil here pointed out is not one which exists only in theory; and it may perhaps be expected that I should specify the instances in which the public of this age and country improperly invests its own preferences with the character of moral laws. I am not writing an essay on the aberrations of existing moral feeling. That is too weighty a subject to be discussed parenthetically, and by way of illustration. Yet examples are necessary, to show that the principle I maintain is of serious and practical moment, and that I am not endeavouring to erect a barrier against imaginary evils. And it is not difficult to show, by abundant instances, that to extend the bounds of what may be called moral police, until it encroaches on the most unquestionably legitimate liberty of the individual, is one of the most universal of all human propensities. 

As a first instance, consider the antipathies which men cherish on no better grounds than that persons whose religious opinions are different from theirs, do not practise their religious observances, especially their religious abstinences. To cite a rather trivial example, nothing in the creed or practice of Christians does more to envenom the hatred of Mahomedans against them, than the fact of their eating pork. There are few acts which Christians and Europeans regard with more unaffected disgust, than Mussulmans regard this particular mode of satisfying hunger. It is, in the first place, an offence against their religion; but this circumstance by no means explains either the degree or the kind of their repugnance; for wine also is forbidden by their religion, and to partake of it is by all Mussulmans accounted wrong, but not disgusting. Their aversion to the flesh of the “unclean beast” is, on the contrary, of that peculiar character, resembling an instinctive antipathy, which the idea of uncleanness, when once it thoroughly sinks into the feelings, seems always to excite even in those whose personal habits are anything but scrupulously cleanly, and of which the sentiment of religious impurity, so intense in the Hindoos, is a remarkable example. Suppose now that in a people, of whom the majority were Mussulmans, that majority should insist upon not permitting pork to be eaten within the limits of the country. This would be nothing new in Mahomedan countries. 1 Would it be a legitimate exercise of the moral authority of public opinion? and if not, why not? The practice is really revolting to such a public. They also sincerely think that it is forbidden and abhorred by the Deity. Neither could the prohibition be censured as religious persecution. It might be religious in its origin, but it would not be persecution for religion, since nobody’s religion makes it a duty to eat pork. The only tenable ground of condemnation would be, that with the personal tastes and self-regarding concerns of individuals the public has no business to interfere.

To come somewhat nearer home: the majority of Spaniards consider it a gross impiety, offensive in the highest degree to the Supreme Being, to worship him in any other manner than the Roman Catholic; and no other public worship is lawful on Spanish soil. The people of all Southern Europe look upon a married clergy as not only irreligious, but unchaste, indecent, gross, disgusting. What do Protestants think of these perfectly sincere feelings, and of the attempt to enforce them against non-Catholics? Yet, if mankind are justified in interfering with each other’s liberty in things which do not concern the interests of others, on what principle is it possible consistently to exclude these cases? or who can blame people for desiring to suppress what they regard as a scandal in the sight of God and man? No stronger case can be shown for prohibiting anything which is regarded as a personal immorality, than is made out for suppressing these practices in the eyes of those who regard them as impieties; and unless we are willing to adopt the logic of persecutors, and to say that we may persecute others because we are right, and that they must not persecute us because they are wrong, we must beware of admitting a principle of which we should resent as a gross injustice the application to ourselves. 

The preceding instances may be objected to, although unreasonably, as drawn from contingencies impossible among us: opinion, in this country, not being likely to enforce abstinence from meats, or to interfere with people for worshipping, and for either marrying or not marrying, according to their creed or inclination. The next example, however, shall be taken from an interference with liberty which we have by no means passed all danger of. Wherever the Puritans have been sufficiently powerful, as in New England, and in Great Britain at the time of the Commonwealth, they have endeavoured, with considerable success, to put down all public, and nearly all private, amusements: especially music, dancing, public games, or other assemblages for purposes of diversion, and the theatre. There are still in this country large bodies of persons by whose notions of morality and religion these recreations are condemned; and those persons belonging chiefly to the middle class, who are the ascendant power in the present social and political condition of the kingdom, it is by no means impossible that persons of these sentiments may at some time or other command a majority in Parliament. How will the remaining portion of the community like to have the amusements that shall be permitted to them regulated by the religious and moral sentiments of the stricter Calvinists and Methodists? Would they not, with considerable peremptoriness, desire these intrusively pious members of society to mind their own business? This is precisely what should be said to every government and every public, who have the pretension that no person shall enjoy any pleasure which they think wrong. But if the principle of the pretension be admitted, no one can reasonably object to its being acted on in the sense of the majority, or other preponderating power in the country; and all persons must be ready to conform to the idea of a Christian commonwealth, as understood by the early settlers in New England, if a religious profession similar to theirs should ever succeed in regaining its lost ground, as religions supposed to be declining have so often been known to do.

Let me point out that the same principle applies to freedom from secular religions and non-supernatural religions. If one avoided all actions that could offend someone according to some deeply felt belief, one would have very little room to move. 

One problem I have with John Stuart Mill’s argument here is that I don’t see the boundary between self-regarding and other-regarding actions as being quite so clear as he does. I discussed some of the tricky issues in “John Stuart Mill on Being Offended at Other People’s Opinions or Private Conduct, but without any resolution. 

If one says that one should be free to choose one’s own self-regarding conduct, the next question is what counts as "self-regarding conduct.” And it would be circular to then say that self-regarding conduct is conduct we should be allowed to choose freely. 

Here is one possibility for the boundary between self-regarding and other-regarding conduct: self-regarding conduct is conduct that even if one fully understood the consequences of the conduct, one would only care about in someone else if one was close to that person emotionally, or one was imbued with a particular meme–what the Wikipedia article “meme” calls “a unit for carrying cultural ideas, symbols, or practices that can be transmitted from one mind to another through writing, speech, gestures, rituals, or other imitable phenomena with a mimicked theme.” Later on, the article says this:

Proponents theorize that memes are a viral phenomenon that may evolve by natural selection in a manner analogous to that of biological evolution. Memes do this through the processes of variation, mutation, competition, and inheritance, each of which influences a meme’s reproductive success. Memes spread through the behavior that they generate in their hosts. 

Genes respect some boundaries between self and other. Relatives end up being counted as in-between self and other as far as genes are concerned, but at least some people are counted as definitely other. Memes respect no such boundaries. The propagation of an untestable and unverifiable idea, religious or otherwise, can be affected by almost anyone’s actions. So memes care (in the sense of having their replication depend on) what everyone does.

To draw a magic circle of liberty around each person, it is necessary to stop at the radius that his genes would care about, and that other people’s genes would not care about, or would care only a tiny bit about.

The upshot of this rule is that people are allowed to do anything that helps out their own genes–or any interest that ultimately derives from the interests of genes, even if it has wandered far afield from direct gene interests (such as the desire of an infertile person for sexual activity)–unless it interferes with the genes or gene-outgrowth interests of others of others. We then have social rules (hopefully fair) about what to do if my gene or gene-outgrowth interests conflict with yours. But if my gene or gene-outgrowth interests conflict your memetic interests, then I have the right to decide what will happen, not some social mechanism. And if your gene or gene-outgrowth interests conflict with my memetic interests, then you have the right to decide. And these rules tend to provide enough guidance on boundaries to help decide conflicts between my memetic interests and yours. 

I don’t know if this account of “self-regarding” versus “other-regarding” actions really works, but it is all I have right now. I would be glad for any other non-circular definitions of what counts as “self-regarding” versus “other-regarding” that you can show give a different answer about the boundary than the boundary I am proposing that accords with the limits of gene and gene-outgrowth interests.