It has been argued that the social circumstances of many children of prisoners goes against established principles of social justice. In this paper the proper allocation of responsibility for remedying this social injustice is discussed. Through a discussion of four principles for allocating remedial responsibility, it is argued that the responsibility for children of incarcerated parents is shared among several actors, including the incarcerated parent, remaining caregivers, prison officials, social work professionals, and, to some extent, members of the wider community. While incarcerated parents are partially responsible for remedying the harm caused to their children, prison officials have the responsibility to uphold the types of prison conditions under which incarcerated parents are able to fulfill their responsibilities to their children and socially contribute to their well-being. Similarly, whereas the main responsibility to care for the children of incarcerated parents falls on the caregivers (such as the remaining parent or other family relatives), states are at the same time responsible for implementing social welfare policies of the sort that can help caregivers fulfill their responsibilities for the well-being of these children. As for individual members of the wider community, they have an obligation not to contribute to the stigmatization and marginalization of the families of incarcerated individuals.