Seems the recently released version 9.1.1 has regressed, and isn't actually rerunning failed tests. Pinning it to 9.1 seems to work however.