LDM

LDM (Lua Data Model) aims to be a module to help working with what I call the Lua data model: tables and values that build directed graphs, DAGs¹, trees, lists, etc. It is mostly about persisting and exchanging said data.

This module aims to be a fundamental component of my methodology, which lies in the context of video game creation. Contrary to other methodologies where it could be considered harmful to have the application's data bound to a specific programming language (in that case I would use SQL/SQLite), I consider Lua as fundamental to mine and don't want to make a distinction between code, runtime and persistent data. What I call the Lua data model can be represented by the LDM Binary Format, by human-readable Lua sources or in memory by a Lua state.

Project scope (implemented, aimed or potential features):

• Binary Format: Portable and durable, with use cases from immutable file formats (e.g. game state data or SCM² versioned data) to network messages. It should be safe to exchange.
• State: Delta, snapshots, transactions, concurrency.
• Utilities: Content-based object comparison, clone, merge, etc.
• Schema: Definition and validation.

LuaJIT is the only target.

The binary format is documented and specified here. Benchmarks are available.

The binary format implementation aims to be safe when unpacking from untrusted data, i.e. to deal with malicious inputs, whether it is about format parsing exploits or denial of service attacks. Of course, this is only about the format, not about the semantic of the decoded Lua value.

API

Format and Compare Configuration

The configuration table is used by the format and compare functions. It can be set globally or passed to a specific function call. The table will be modified (prepared) on first use and must not change afterwards.

Fields:

• objects: External object dictionary. Can contain table, string, userdata, function and thread values.
• metatables: External metatable dictionary. Tables only. When unpacking, each metatable is assigned to its table after the decoding of the latter.
• stable: If true, packing will try to produce stable output. There are no deterministic guarantees; it is a best effort to facilitate the use of delta compression (e.g. with Fossil). It is likely to have a significant performance cost. Default is disabled.
• strict_metatables: If true, packing a table that has a metatable which is not part of the metatable dictionary will throw an error. Default is disabled.
• interrupt_interval: Heuristic to control the frequency of interruptions in number of packed/unpacked bytes (period). Default is 4096.
• interrupt_pack(i): Callback called on packing interrupt. i is the current number of packed bytes (progress).
• interrupt_unpack(i, size): Callback called on unpacking interrupt. i is the current number of unpacked bytes (progress) and size is the total amount of bytes to unpack.

The two external dictionaries are sequences of external resources; each resource will be associated to its index. The maximum number of entries is 2^32. Holes are allowed, which can be useful to remove an entry or to reserve specific indices. Lower indices use less bytes. Lean more about dictionaries in the format documentation.

External dictionaries must match to decode the original value. Other options have no impact.

Interrupt callbacks have been designed to be able to yield the packing/unpacking process when it has been called from a coroutine. It is about concurrency. When packing, the input value must not change. They can also be used to display progress.

Use ldm.set_conf(conf) to set the global configuration.

Binary Format

ldm.pack (value [, conf]) -> data

Pack/encode a Lua value using the global or conf configuration. Returns a data string in the LDM format.

ldm.unpack (data [, conf]) -> value

Unpack/decode a data string in the LDM format using the global or conf configuration.

Utilities

ldm.compare (a, b [, conf]) -> r

Compare two values by their content using the global or conf configuration. Returns -1, 0 or 1 when a is less, equal or higher than b.

Content-based comparison is about comparing the structure of two different objects. For example, this function is internally used to sort the pairs of a table when packing with the stable option enabled and to check (in testing) that ldm.unpack is the reciprocal of ldm.pack.

The configuration is useful to compare values that depend on external resources.

1. ^{^} Directed Acyclic Graphs
2. ^{^} Source Code Management