Increased knowledge about sex differences is important for development of individualized treatments against many diseases as well as understanding behavioral and pathological differences. This review summarizes sex chromosome effects on gene expression, epigenetics, and hormones in relation to the brain. We explore neuroanatomy, neurochemistry, cognition, and brain pathology aiming to explain the current state of the art. While some domains exhibit strong differences, others reveal subtle differences whose overall significance warrants clarification. We hope that the current review increases awareness and serves as a basis for the planning of future studies that consider both sexes equally regarding similarities and differences.