refactor intermediate saving

- right now, we save the full eval results (convert all states to eval results + metadata and write and overwrite file) at each saving step which makes this unnecessary wasteful (for large evals, i have seen ridicilous hf cache sizes as well)
- i think we should rather append the newly finished results to the eval results, it wouuld feel natural to make this part of the new `on_progress` hook