Because the government so often makes mistakes, good parents aplenty have been afraid of “Child Protective Services” taking their children away for no good reason. But John Stuart Mill argues for the appropriateness of at least some government intervention in parent-child relationships in paragraph 12 of On Liberty “Chapter V: Applications”: 

I have already observed that, owing to the absence of any recognised general principles, liberty is often granted where it should be withheld, as well as withheld where it should be granted; and one of the cases in which, in the modern European world, the sentiment of liberty is the strongest, is a case where, in my view, it is altogether misplaced. A person should be free to do as he likes in his own concerns; but he ought not to be free to do as he likes in acting for another, under the pretext that the affairs of the other are his own affairs. The State, while it respects the liberty of each in what specially regards himself, is bound to maintain a vigilant control over his exercise of any power which it allows him to possess over others. This obligation is almost entirely disregarded in the case of the family relations, a case, in its direct influence on human happiness, more important than all others taken together. The almost despotic power of husbands over wives needs not be enlarged upon here, because nothing more is needed for the complete removal of the evil, than that wives should have the same rights, and should receive the protection of law in the same manner, as all other persons; and because, on this subject, the defenders of established injustice do not avail themselves of the plea of liberty, but stand forth openly as the champions of power. It is in the case of children, that misapplied notions of liberty are a real obstacle to the fulfilment by the State of its duties. One would almost think that a man’s children were supposed to be literally, and not metaphorically, a part of himself, so jealous is opinion of the smallest interference of law with his absolute and exclusive control over them; more jealous than of almost any interference with his own freedom of action: so much less do the generality of mankind value liberty than power. Consider, for example, the case of education. Is it not almost a self-evident axiom, that the State should require and compel the education, up to a certain standard, of every human being who is born its citizen? Yet who is there that is not afraid to recognise and assert this truth? Hardly any one indeed will deny that it is one of the most sacred duties of the parents (or, as law and usage now stand, the father), after summoning a human being into the world, to give to that being an education fitting him to perform his part well in life towards others and towards himself. But while this is unanimously declared to be the father’s duty, scarcely anybody, in this country, will bear to hear of obliging him to perform it. Instead of his being required to make any exertion or sacrifice for securing education to the child, it is left to his choice to accept it or not when it is provided gratis! It still remains unrecognised, that to bring a child into existence without a fair prospect of being able, not only to provide food for its body, but instruction and training for its mind, is a moral crime, both against the unfortunate offspring and against society; and that if the parent does not fulfil this obligation, the State ought to see it fulfilled, at the charge, as far as possible, of the parent.

I think the government policy questions here are quite hard. But on the much more basic question of whether it is appropriate for us to take an interest in the way other people treat their children, I would give a resounding “Yes!”

Even there, one should beware of the mistakes one could make because one has not walked in another parent’s shoes. Many a parent, after being blessed with an easy child for their first, and falling prey to a certain arrogance as a result, have been brought up short by how hard a time they had with a later child.

Nevertheless, at the end of the day, it is the children’s interests that must come first, not the parents’ interests. It is because it generally makes sense to trust parents to have a child’s interests more deeply at heart than the government that it makes sense to give parents the latitude society does give them. (A good illustration of an area where the government frequently does not have children’s interests very deeply at heart is where educational quality in the public schools comes into conflict with the interests of teachers.)