Lua Data Format (LDF) is a binary format for what I call the Lua data model: tables and values that can build directed graphs, DAGs1, trees, lists, etc.
The format doesn't aim to be a standard, but it aims to be portable and durable. It is similar to MessagePack.
Note: This module aims to be a fundamental component of my methodology, which lies in the context of video game creation. Contrary to other methodologies where it could be considered harmful to have the application's data bound to a specific programming language (in that case I would use SQL/SQLite), I consider Lua as fundamental to mine. The Lua data model can be represented by human-readable Lua sources, in memory by a Lua state or by this project's format. This allows to use the same data model for all purposes, with use cases ranging from immutable file formats, e.g. game state data or SCM2 versioned data, to network messages.
The format is documented and specified here. Benchmarks can be found here.
The implementation aims to be safe when unpacking from untrusted data, i.e. to deal with malicious inputs, whether it is about format parsing exploits or denial of service attacks. Of course, this is only about the format, not about the semantics of the decoded Lua values.
LuaJIT is the only target at the moment.
API
Configuration
The configuration table is used by the format functions. It can be set globally or passed to a specific function call. The table will be modified (prepared) on first use and must not change afterwards.
Fields:
objects: External object dictionary. Can containtable,string,userdata,functionandthreadvalues.metatables: External metatable dictionary. Tables only. When unpacking, each metatable is assigned to its table after the decoding of the latter.stable: If true, packing will try to produce stable output. There are no deterministic guarantees; it is a best effort to facilitate the use of delta compression (e.g. with Fossil). It is likely to have a significant performance cost. Default is disabled.strict_metatables: If true, packing a table that has a metatable which is not part of the metatable dictionary will throw an error. Default is disabled.interrupt_interval: Heuristic to control the frequency of interruptions in number of packed/unpacked bytes (period). Default is 4096.interrupt_pack(i): Callback called on packing interrupt.iis the current number of packed bytes (progress).interrupt_unpack(i, size): Callback called on unpacking interrupt.iis the current number of unpacked bytes (progress) andsizeis the total amount of bytes to unpack.
The two external dictionaries are sequences of external resources; each resource
will be associated to its index. The maximum number of entries is 2^32. Holes
are allowed, which can be useful to remove an entry or to reserve specific
indices. Lower indices use less bytes. Lean more about dictionaries in the
format documentation.
External dictionaries must match to decode the original value. Other options have no impact.
Interrupt callbacks have been designed to be able to yield the packing/unpacking process when it has been called from a coroutine. It is about concurrency. When packing, the input value must not change. They can also be used to display progress.
Functions
ldf.set_conf (conf)
Set the global configuration.
ldf.pack (value [, conf]) -> data
Pack/encode a Lua value using the global or conf configuration. Returns a
data string (LDF).
ldf.unpack (data [, conf]) -> value
Unpack/decode a data string (LDF) using the global or conf configuration.