Off centre projection matrix breaks cascaded shadows

I have an off centre projection matrix in my game but while it works if the offset is on the x axis if there is any y offset then cascaded shadows get offset in the opposite direction, and example can be seen in the picture, can anyone offer a solution to fix this?