Your Zero Wars game didn't affect your Ladder Elo, the FFAs did. The ladder elo change is also never a direct result from a single battle, but most likely in this case it was rather direct from the FFAs.
Your elo was so disparately high that you had a 50%+ win chance, even though there were 4-5 players in the game. So these games are the equivalent of losing a 1v1 to a slightly worse player. In the rating formula, you can approximate the weight of a game to be linear with the number of players, so each of these FFAs was roughly equivalent to losing 10 balanced 10v10s in a row.
If you want your casual rating to be less impacted by such single events you can either play more 1v1/FFA or play a lot more team games. Otherwise there is always the problem that 1v1/FFA contains a lot more "information" per battle than big teams and the rating will thus cling to those results.