Series elastic actuation (SEA) has become prevalent in applications involving physical human-robot interaction, as it provides considerable advantages over traditional stiff actuators in terms of stability robustness and force control fidelity. Several impedance control architectures have been proposed for SEA. Among these alternatives, the cascaded controller with an innermost velocity loop, an intermediate torque loop, and an outermost impedance loop is particularly favored for its simplicity, robustness, and performance. In this article, we derive the necessary and sufficient conditions for passively rendering null impedance and virtual springs with this cascade-controller architecture. Based on the newly established conditions, we provide nonconservative passivity design guidelines to haptically display these two impedance models, which serve as the basic building blocks of various virtual environments, while ensuring the safety of interaction. We also demonstrate the importance of including physical damping in the actuator model for deriving the passivity conditions, when integrators are utilized. In particular, we prove the unintuitive adversary effect of physical damping on the passivity of the system by noting that the damping term reduces the system Z-width, as well as introducing an extra passivity constraint. Finally, we experimentally validate our theoretical results using an SEA brake pedal.