SLF, which stands for Simple Lua Format, aims to be a human readable serialization format and implementation for what I call the Lua data model: tables and values that can build directed graphs, DAGs1, trees, lists, etc.
The format doesn't aim to be a standard, but it aims to be portable and durable.
The format is designed to be used as an immutable file format2 and is mostly about the storage of small and precious data.
My concrete use cases leading to the creation of this project are the storage of
Lua data in a Fossil repository, in a LÖVE archive (.love) or in a game's save
file.
Principles and Constraints:
- Simple: The format and the implementation should be simple to use and understand to improve the durability of the data. It is only about plain Lua data, i.e. no metatables or external dictionary support. Decoding/encoding speed is a secondary concern (cf. small and precious data).
- Compressible: By delta compression (Fossil) and by general algorithms, mostly by Deflate (Fossil's zlib, LÖVE's zip and gzip).
- Exact: It must output exact values, i.e. no rounding or loss of precision. When decoding, inability to decode a value exactly must be reported as an error by default.
- Deterministic: The same Lua data should produce the same SLF output. It is a constraint for the encoder, the decoder can accept any ordering to allow for easier human edits, and a best effort; new versions may alter the output.
- Stable: In addition to deterministic output, a slight change to the Lua data should yield a slight change in the SLF output. Both allow for easier human analysis of the data, clean diffs computation and better delta compression.
Although the format is human readable and can be directly modified with an editor, it is first and foremost a serialization format and not something designed to be edited manually. The human readability aspect is mostly about helping with the inspection of the data and its changes, and to increase its resilience.
This module aims to be a fundamental component of my methodology, which lies in the context of video game creation. Contrary to other methodologies where it could be considered harmful to have the application's data bound to a specific programming language (in that case I would use SQL/SQLite), I consider Lua as fundamental to mine. The Lua data model can be represented by human readable Lua sources, in memory by a Lua state or by this project's format.
Format Specification
The format must be backwards compatible and should be frozen.
Forbidden Ideas
This section is about ideas that should never be done.
- Format customization (options to customize the output). This would impair the ability to understand and exchange the data, making it less stable and more brittle. It would also increase the complexity of the decoder.