The transform back is not correct yet.
See how to make the fit and rotation with respect to point centers,
then translate ellipse back at the right coordinates.