+1 on human readable json!
very excited that you’re taking this one @csboling!
Thanks for all the feedback and good vibes! I’ve started a prototype program for seeing what works as far as dealing with this JSON format on the target. So far it basically just checks the top level version and loads scales, but I think the patterns generalize and from here it will be easier to hammer out the rest of the sections. An example of what the “client code” might look like for describing the format for a particular firmware/app is in ansible_sections.c.
I decided to go with hex strings for encoding buffers, this reduces the size a lot and hopefully remains relatively human editable. A little fiddlier for types that are signed or not 8-bit, but hopefully not too painful. (Since I believe the target is big-endian I think this should still work out okay for types like uint16_t[], but I do need to remember to make sure the Python code creates hex strings with the right byte order.)
I looked into a number of different JSON libraries and wound up going with JSMN. Small code footprint, simple API, but importantly it seemed more capable than others I found of dealing with small text buffers that might contain only partial JSON constructs, and it doesn’t do any allocations. I did have to edit out one line that would flag an error if it saw a closing brace/bracket and couldn’t find the matching open brace in its token buffer, and I also have to “rewind” the parser’s state between reads from the file stream. Otherwise the incremental behavior seems to be working quite nicely. JSMN doesn’t really provide any mechanism for stringifying objects, since it doesn’t really have any type of DOM data structures, so some patterns for serialization still need to be worked out - for now I’m kinda hand-waving this as being roughly the inverse of what works for deserialization.
From the app author’s point of view what this means is that you register callbacks which handle a single JSON token (jsmntok_t) at a time. You return PRESET_READ_OK when you’re done, PRESET_READ_INCOMPLETE if you need more tokens, and PRESET_READ_MALFORMED if you want to abort the whole parse. If the last token is incomplete and you need to process a whole token, returning PRESET_READ_INCOMPLETE will cause it to be relocated to the beginning of the text buffer so you can try again. The deserialization engine will pass you opaque pointers that you can keep whatever state you need in, and it’s the caller’s responsibility to make sure the state pointers are valid before calling preset_deserialize. It’s okay to reuse the same state object for multiple callbacks though, since the sections that they represent are just called one at a time. As I go through the different sections I’m trying to refactor reusable functions and state chunks out to reduce the tediousness of writing and maintaining these handler functions.
Next up I definitely want to write some more formal test cases for making sure that various kinds of malformed JSON documents don’t cause buffer overruns and stuff. Not sure yet what the right UI signal should be on Ansible that the preset file couldn’t be read.
1 Like