#1

Development Discussion – Read at your own risk!

In preparation for the feature TL , AKA Timeline, AKA F, I am beginning design of a better USB disk mode.

To recap, the current disk mode has the following characteristics:

  • Automatically launches when USB flash drive
  • Writes all scenes in flash to ttXXs.txt
  • Reads all scenes to flash from ttXX.txt

To begin designing the new feature, I think it would be wise to list how a USB disk might be used:

  • Selectively save / load scenes
  • Manipulate directory structure
  • Rename / copy / move / delete files
  • Paste scripts / patterns from other scenes

Obviously, this list is too broad to implement safely and too complex for non-power users. Direct access to the filesystem is inelegant, confusing, and error-prone.

Facet 1: To combat this, I think Scenes should be organized in banks, with 32 (eventually 16?) scenes available at a time in the user interface. This would permit us to keep the existing file name conventions and present a simple UI.

When a user wishes to copy a script from a scene in another bank, it will need to be parsed and loaded into memory. We should not have to load the whole bank, nor should we have to load the scene into the current bank.

Facet 2: There needs to be scratch space available in RAM for one full scene.

When a user is navigating scenes, it may be difficult to remember where each known script is in the bank layout.

Facet 3: The UI should not list blank scenes in read select operations.
Facet 4: The UI should not list empty banks in read select operations (or there are no empty banks which a user has not explicitly allocated / created.)
Facet 5: The UI maybe should list the first line of the description for scene identification when listing scenes.

Points of Entry

We need to consider how a user might access data on the USB drive. Here are some possible use cases:

  • A user accesses the existing Scene Write mode and uses a keystroke to display a bank/scene selector and saves to USB.
  • A user accesses the existing Scene Read mode and loads from USB as above.
  • A user accesses a new, dedicated interface for all USB filesystem transactions.
  • A user is in edit mode and presses a keystroke to display a bank/scene/script selector to source a script from another scene into the current script.
  • A user is in pattern mode and presses a keystroke to source a pattern from another scene as above.

(Edit: There’s no keyboard connected when a USB stick is inserted, DUH!)

Dedicated Filesystem Mode

A new display mode will need to exist to perform the following operations:

  • Save / Load a bank to flash memory
  • Save / Load current scene bank with a selectable bank slot
  • Save / Load individual scene with a selectable bank / scene slot
  • Copy one bank to another bank slot
  • Copy one scene to another bank / scene slot
  • Delete (clear) a scene
  • Delete (clear) a bank

And, depending on design decisions, it may also need to perform:

  • Copy a script from one scene to another
  • Copy pattern data from one scene to another

(Edit: no keyboard, will be an edit mode feature)

Simplified Operation

For more mundane usage, a user may wish for a way to use the USB filesystem that is as simple as the current method.

Facet 6: The default selectors for save and load should match the existing save/load idiom so that simplified behaviour is presented in as few keystrokes as possible.

That’s all I have on my mind at the moment. I will not edit this post. Further iterations of the filesystem idiom will be condensed into a future post in this thread to prevent confusion.

Like this post if you agree with it wholeheartedly and don’t have a preference on any of the decisions, reply (with relevant quote, please!) if not.

Edit: Other Facets

Facet 7: The USB UI will need to be entirely operated with the PARAM knob and the button.

Facet 8: Script / pattern copy will need to either be performed from the current bank in edit mode , or from any bank in USB FS mode, or both.

Facet 9: USB FS mode menu options should have sane defaults.

Facet 10: USB mode must automatically launch when a USB stick is inserted and must automatically close when a USB stick is removed.

Tentative menu structure

6 Likes
Teletype 3.0
Monome ecosystem firmware development backlog
A user's guide to the wonderful world of teletype ⌨
Scala files in Teletype + Telex ecosystem
Teletype 3.+ feature requests and discussion
(Teletype) 2.2 Release Status
(Teletype) expanding scripts: length and amount
#2

All this sounds great. I’ll be happy as long as it doesn’t automatically overwrite saves like it does now. The whole editing the file and adding s Has killed me too many times to count. If it was as simple as the scene saves I’d be elated. Plugging in the USB drive, scrolling with the parm knob and pressing the button to load would be sweet.

4 Likes
#3

And with that, you’ve reminded me that the keyboard will not be available during USB filesystem operation.

My goodness, how could I have forgotten:

Facet 7: The USB UI will need to be entirely operated with the PARAM knob and the button.

This also helps to clarify questions about script / pattern copies:

Facet 8: Script / pattern copy will need to either be performed from the current bank in edit mode, or from any bank in USB FS mode, or both.

1 Like
#4

This facet means that the USB FS mode will be menu-driven. Here’s my first crack at a menu:

Save Bank to USB
    - Bank Selector
    - Save (default selected)
    - Cancel
Load Bank from USB
    - Bank Selector
    - Load (default selected)
    - Cancel
Save Scene to USB
    - Bank Selector
    - Scene Selector
    - Save / Cancel
Load Scene from USB
    - Bank Selector
    - Scene Selector
    - Load / Cancel
Copy Script
    - Source Bank / Scene / Script Selectors
    - Target Scene / Script Selector
    - Copy / Cancel
Copy Pattern
    - Source Bank / Scene Selectors
    - Target Scene Selector
    - Copy / Cancel

The menu-driven idiom is unfortunate, but inevitable I think. Its downside can be somewhat ameliorated by good design.

Facet 9: USB FS mode menu options should have sane defaults.

Also, scratch Facet 2:

2 Likes
#5

Yeah. I kept forgetting that too :smiley:

Perhaps some sort of hierarchical menu, param to scroll through the entries, the button to select.

  • Save all scenes to bank
    • Bank 1
    • Bank 2
    • …
    • Bank 32
  • Load all from bank
    • Bank 1
    • Bank 2
    • …
    • Bank 32
  • Save individual scene
    • From preset 1
      • To bank 1
        • To preset 1
        • To preset 2
        • …
        • To preset 32
      • To bank …
    • From preset …
  • Load individual scene
    • From bank 1
      • From preset 1
      • From preset 2
      • …
      • From preset 32
    • From bank …

(Ah, just seen your post with something similar)

Have you given much thought to the actual code that serialises and deserialises a scene to a text file? Would it be okay for me to share some thoughts on it? I actually think that it’s more important to get that code right, then it is to get the UI right as the UI is more easily changed going forward.

1 Like
#6

Not anything yet, but I have broken out the read and write operations to operate on a by-scene-and-filename basis.

I was taking the “If it ain’t broke, don’t fix it” approach, but as we can see on the issue tracker, it is broken in at least one way (# in scene description breaks deserialization).

#7

curious to hear your thoughts on serialization! this really ought to be a part of libavr32 so we could add it easily to other modules as well. have you taken a look at orca code for that? pretty simple and forgiving to the syntax (which among other things means a user has more freedom to format it to whatever is more readable for them).

re: keyboard not being available - @sliderule have you thought of taking a stab at writing USB hub support? :slight_smile:

#8

I doubt the serialisation / deserialisation code will ever be generic enough for inclusion in libavr32, but if anyone knows of a parser combinator library for C that doesn’t heap allocate I’m all ears.

Rather we need to separate the serialisation code from the IO code, that way we can move it from module where it’s incredible hard to test and debug to src where we can write unit tests and use gdb.

(Does everyone know what I mean when I say module and src?)

This is particularly important as we will need some sort of forward compatibility1 for presets, and maintaining that will be impossible without some sort of automated testing. And that in turn will give us the freedom and confidence to adjust the preset format for new features going forwards.

My suggestion is that we add 2 functions to src:

void tele_serialise_scene(tele_scene_t *scene, char[MAX_TEXT_SIZE] *output);
bool tele_deserialise_scene(char *input, tele_scene_t *scene, char[MAX_ERROR_SIZE] *error_text);  // returns false on error

Neither function is allowed allocate or do any IO, that will be handled by the caller. AFAIK scene text files aren’t that big, we may choose to statically allocate some space for the buffer for the module code rather than deal with malloc.

The error text can either be displayed to the user, or saved to an error.log file on the USB stick (or both).

But most importantly it will allow us to build up a corpus of presets to test against. The easy path would be to just assert that they must parse. The harder route would be to assert that they parse and check the value. It’s possible that we could build some tools to turn a script into a C file with the structs in it so that we may quickly generate the data.

Anyway, that’s what I would do if I had the time. Alas, I don’t. So it isn’t for me to say whether you should go this way or not!

1 I suggest we do place some limitations on forward compatibility, e.g. we only support import presets saved under the previous X releases.

Serialisation of Presets
#9

It really wasn’t on my radar, and may be significantly more complex than I’m willing to tackle given the breadth of USB devices that might be simultaneously connected (multiple USB drives, a keyboard, a grid).

Do I ever! :slight_smile:

That sounds good, including the corpus of presets. The versioning will require that we change the file structure, so it will be breaking.

1 Like
#10

i disagree that we can’t have some generic enough functions for serializing / deserializing common types, which is where the bulk of the work is.

heap allocation - could you point where it is?

#11

Maybe, it depends how @sliderule ends up implementing it. I would suggest letting it mature in the Teletype repo for a while before it gets moved though. It’s that much harder making changes to shared code.

I don’t follow this.

#12

In my years, I have found that this sort of effort is not worth the time, at least in C. (Trying to make generic operations instead of specific ones)

1 Like
#13

i thought you were referring to the existing code, sorry.

1 Like
#14

i’m not convinced still, could you elaborate on the reasoning?

and take a look here: https://github.com/scanner-darkly/monome-mods/blob/master/orca/main.c#L3203

#15

Excellent. You’ve still got the really messy bit of trying to demangle the existing functions.

Are you going to stick a magic number and a format version in the new presets going forwards?

I kind of agree with this. I’d vote for copying and pasting the code between modules.

Unless we can find a pre-existing library that targets embedded devices.

#16

sorry, but i find this very strange reasoning. why do we have libavr32 at all then?

edit: maybe i didn’t make myself clear. i’m not advocating writing a full blown serializer/deserializer method, i would agree it couldn’t be done efficiently. but i think helper methods could certainly be re-used by module specific implementations of serialization.

#17

Well. For sharing AVR32 specific code…

But it’s so hard to write generic parser code. Your one doesn’t look like it checks for the end of the buffer1, which might be fine for your use case, but not for something generic.

Also, the src bit of the Teletype code shouldn’t be limited to only running on the modules. If someone enterprising wanted to make it run on a Raspberry Pi with MIDI instead of CV and a Curses interface, that should be pretty doable.

1 Or rather it’s based on null terminated strings. Which isn’t necessarily the same as a file could contain a null character. (edit: or maybe I’m talking out of my bottom)

#18

I see that the linked code is bound-constraining, but that will just happily siphon from an invalid file and produce a bad result instead of abandoning, so I might redesign this to include an error return value, which will lead to every output being pointer-driven. At that point, I don’t know why I wouldn’t use sscanf / whatever and do external bounds-constrainment.

Essentially, these type of exercises lead to replacing 4-5 lines of code with 2 using compromises to get there.

I understand the benefits of modular code and the downsides of long function bodies (and sscanf!), but at the end, you haven’t really saved yourself much of anything.

That’s just my opinion. I will take a good hard look at the teletype scene loading, which is much less complex than that orca preset load, and see if there is any hay to be made by modularizing any of the code.

#19

it’s not just teletype code. it would be nice to make USB storage universally available on all, mp/ww/es/ansible. as it happens, teletype serialization is the simplest of them all, it’s mostly text with no actual parsing other than pattern data.

#20

But do they need text based serialisation like Teletype does? Wouldn’t a structured file format like JSON/BSON/ProtoBuf/etc/etc (or even a binary dump with a magic number and version) be better suited?

1 Like