‘It’s hard going into this deep anger and grief, and then go back,’ says the star, who re-teamed with director Joachim Trier for the emotional family drama