Hi @sliderule, curious if you have any more info on the document format you could share. I’m interested in contributing to adding USB disk functionality to Ansible, and from the work on your branch on usb_disk_mode.c it looks like a lot of what remains is to work out the JSON serialization code.
I figure that one component needed to use the Ansible preset save/load feature would be a tool for loading a pre-USB-support ansible.hex image and writing the presets from the firmware dump to a JSON file, so that users can carry over their existing presets. I wrote a Python program that does this conversion on a hex file, with a goal in mind of making it simple to add support for reading presets from older firmware versions or alt firmware like Ansible Earthsea.
I also thought this kind of tool could prove useful for hashing out the exact JSON format to use in a language where that’s easier to prototype than with a low level embedded-friendly C library, as well as giving me an example document I can put on a USB stick and try to parse from Ansible. Right now the format this program writes is a very straight-ahead dump of all the structs/arrays for the Ansible apps, basically as they’re named in the source code. A couple questions I’ve been thinking about for the format include:
- Would this simple approach to building the JSON objects cause problems for backward-compatibility, or otherwise pose difficulties for supporting this serialization format on other modules or apps? For the broader I2C ecosystem and a possible future desire to create presets that operate in a multi-module context?
- The structure I started with lets each app have its own section of the JSON document, within which I suppose a given app can do whatever it wants. There’s also a
shared section for scales and a meta section for preset version, etc. Should apps be versioned independently, or is firmware version sufficient/more desirable?
- Is storage space a concern for all the string literals that the parser would need to match, to the point that strings like
alt_note or meta_lswap should be abbreviated? If so, how best to trade this off with human-readability?
- This is a pretty verbose format. If you’ve got an array of
uint8_ts, each byte gets its own number in the JSON representation (edit: realized you can reduce this ~2x by just storing byte arrays as hex strings, ~3x if you’re willing to decode base64). The file I converted from my Ansible Earthsea dump is about 1.15 MB, with the Earthsea part using a little more than half of that - a little less than 10x larger than the nvram_data_t struct in ROM. This might make parsing/emitting individual values from structs on the target easier, and less prone to fiddly issues with byte order/packing, as well as making the format more human readable (though I probably wouldn’t want to do a whole lot of hand editing of a JSON file this size). Storage size is probably not a huge issue given the size of USB disks, but perhaps you don’t want to try to parse/stringify a document that size all at once. So maybe we need to select an incremental parser? (<-- now it’s a question!)
As far as selecting a JSON library goes, I found this benchmark comparing a number of C and C++ options in various ways. Tables comparing all these libraries where you can sort by specific metrics are here. There’s also Frozen, which is not included in this benchmark but says it specifically targets embedded systems. I haven’t done a lot of filtering this list down yet on criteria like how actively maintained they are or whether they have features like incremental parsing.
Apologies for a bit of a brain dump, I realize this is all kinda open ended but would love to get some feedback from folks with some experience working on the Monome firmware.
