Description
At the moment, writing a checkpoint causes acid-state to realise the entire serialised representation in memory before it gets written to disk. This can be a significant memory cost for server applications with a
large state. It seems unavoidable with the existing archive backend, because the format consists of (length, CRC, bytestring) where the first two fields are not known until the bytestring is fully evaluated. However, we should be able to do better with an alternate backend (given #96) that stores chunk lengths rather than a single overall length, and moves the CRC to the end.
This raises a question: what should we do if an exception is thrown during checkpoint serialisation? In particular, this can happen if user code stores an unevaluated error thunk in the state. We already fail to handle this case gracefully (see #38). Should we simply document that the state must never contain pure exceptions? Given the possibility of multiple checkpoints per file, it seems hard to recover from a partially-written checkpoint.