#41

a few things against a plaintext version:

  1. people can’t actually edit the text in a computer editor-- there is no syntax checking, so when the files import there will inevitably be typos and it’ll just fail, without any warnings, which is a completely terrible system-- hence useless
  2. manual editing means files get saved by various editors/os’s with different LF tactics, hence the parser breaks (granted, this just means the parser needs to be smarter-- currently it is not-- if you edit these files with a noncompliant editor (not sure which works) you’ll simply basically corrupt the file.

a js scene-to-text converter would be very lightweight and would live online, no dependencies needed.

so, what’s worth examining are what are the advantages to plaintext storage? yes, it’s human readable, but the only time you’ll ever care about that is to either 1. see what the file is while sorting your usb disk or 2. copy-paste the text to a forum post.

the forum post will be unformatted, but looks ok in a codeblock. if we repair the linefeed issues someone could copy-paste it back into a file on disk. but otherwise it’d be smarter to just attach the source file to the thread.

perhaps the reason i’m pushing for something like msgpack is that it can set up a sensible framework for saving preset data for other things, ie, kria presets, which absolutely need no plaintext, and have have values that are not 8 bit (hence would need more characters and manual typecasting) or arrays which would be better handled with a library.

ie, the pattern read/write code is super clunky as is with plaintext, where with a serializer it’d quite possibly be one line of code.

i know it seems like i’m pushing hard on this, but i absolutely can be swayed, please don’t take what i’m saying as what should happen!

3 Likes
#42

This exists currently as a github repo, although there’s not much activity in it yet.

The ability to do this is predicated on the scripts being plain text, and I’d like to see that continue.

Yes, agreed 100%

3 Likes
#43

this is a very good point.

the essential meta-data (name of scene, description) are immediately readable, whereas they’d be chunked between bin blobs with a serializer. we don’t want to have a meta-file to identify the contents of the file. so yes, this is a strong issue.

1 Like
#44

These are excellent points!

I don’t have any real objection to moving to a serialization library like msgpack, honestly.

If there are userland tools to convert to and from a text and/or markdown version of the script, I can’t see this being major, especially if those tools are included in the codex repo and the repo only gets commits with all three versions. This adds the advantage that all the scripts in the codex are known to be valid and free of syntax errors.

Another alternative is to write both files to the USB drive. It’s not like anyone would be short on storage space.

1 Like
#45

one more thing to consider - editing files manually could allow for copying scales / patterns between different firmwares, if we standardize on how they are serliazed.

i agree such editing can be errror prone, but this could be addressed by introducing a simple key/value format, with one line per value (arrays would also be done in one line, space or comma separated). the first word on a new line would always be considered the key, unrecognized keys could be ignored.

edit: i would like to point out the irony of us discussing it for a module whose ethos is translating human readable commands into triggers and control voltages :slight_smile:

5 Likes
#46

Presumably, you are referring to serializing to a text document, right?

As someone who has fought the line ending war with myself for years, I have to find that factor a compelling argument against text documents as a sharing vector over USB. I have to presume that nobody is using notepad on Windows to view these files, or that modern notepad is better with line endings.

Of course, this won’t be a problem on the repo or on lines, only USB to Windows PC. I’ll have to see if the parser in Teletype handles CR okay.

The other thing that’s sticking with me is the Markdown aspect. Having to indent and deindent can introduce corruption. If the raw text or Markdown version can be easily converted to a valid scene, you no longer need to attach the original file to your lines post as the easy vector.

I’m really coming around to the idea of a web-based interface, too. I was looking at this from the wrong angle before: if the goal is to share, then loading up a browser is a natural fit. Mea culpa, @jlmitch5.

Now I can see it: a Teletype scene editor and simulator in your browser with a community library of scripts and scenes and local filesystem access.

Guess I better brush up on my Dart. :smile:

1 Like
#47

isn’t it trivial to accept both styles of line endings, unless i’m missing something? i was looking at the teletype code today (working on storing button and fader state for grid integration, as it happens) and it looks like it would be easy to change it to accept both (although i would probably refactor that code altogether). as for indents just ignore leading and closing whitespace in scripts or ignore whitespace altogether if using key/value pairs.

yeah, storing presets from, say, kria and copying scales to a white whale preset by editing files manually. or you make a nice pattern on white whale, transfer it to pattern data in teletype and put the turtle on it.

3 Likes
#48

Thinking about this, it’s something we can tackle when and if hub support is in view. For instance, you could just not load the interface if a keyboard or other suitable control surface were still connected.

1 Like
#49

Yeah, absolutely. I’m just not sure if it currently does.

As mentioned earlier, however, writing both (or all three) files is trivial. It’s the reading side that’s critical. Text parsers are okay, but if we’re willing to do something new, we might as well choose to be efficient and safe on the hardware, then spin out text parsing to a better machine / language, IMO.

1 Like
#50

there were some threads on adding USB hub support and sounds like it would be a major task… i wonder if it would be almost easier to have a dedicated merger/switch module (that could also communicate with tt via i2c)

i don’t think it does but could be simply fixed with if (c == '\r') continue; and a couple other places that expect only 1 char before a new line begins.

1 Like
#51

It’s late here, but just wanted to say I’ve thought about it more and definitely think that going the direction of your design is definitely the way to go here. The web is a good platform for enabling sharing and such, but probably would just introduce headache and complexity when dealing with management.

:+1: @sliderule. I feel like we can figure out some cool tools on the sharing side of things (and some sort of interpreter/linting is a really neat idea!)

The sharing/community stuff with scripts can pretty much be completely dependent on most of the technical details here so far as I understand them…the serialization stuff does come into play a bit tho, especially when considering something like github. It always feels a bit weird and opaque when you see files that aren’t “code” being managed in source control…because history can be pretty opaque. An example would be something like a photoshop file, or a max patch.

That being said, I feel like I see some other ux benefits to keeping things in a blob format as mentioned here…and that’s not even considering some of the technical details that are definitely over my head.

—

USB hub mentioning was more of just a like, something to think about when implementing (and sounds like it might not be something even feasible based on previous discussion and work). I think you got the UX right in your design for the time being :+1:

1 Like
#52

USB hubs… there are a bunch of threads on here with me discussing how it could be done by offloading the multiplexing to a Raspberry Pi or a Beaglebone Black (or even a normal computer). The AVR32 USB port is supposed to be able to switch between device and host mode at runtime. IMO this would be much easier than real USB hub support. Anyway, no more talking about it on this thread, if someone wants to implement this then bump an existing thread or start a new one.

Serialisation formats…

I’m generally in favour of text based. Remember we already have parsing and validation code from Teletype code. But I can understand not wanting to as it’s a lot of work to maintain.

IMO, that’s because the saving and loading code is too hard to work with. If we made a better version it should be a lot easier to make it more robust. As well as giving it the ability to save errors to an error.log file on the USB stick.

Using Javascript for the converter would force the data format to include potentially redundant metadata (e.g. OP names), otherwise you’d have to duplicate that information from the Teletype source base. A better idea would be to output a Markdown file with a read only copy of the scene in text format, and then a base64 encoded version of the binary data at the bottom: e.g.

<!-- WARNING the following content is READ ONLY -->

# First line of scene description

More scene description

## Script 1

`` 
ADD 1 1
``

## Script ...

## Pattern data

....


## Binary data

This is the actual data that is read by the Teletype. **Do not edit.**

``
----------------------- BEGIN TELETYPE DATA: v1 ------------------------
TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxp
dCwgc2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9y
ZSBtYWduYSBhbGlxdWEuIFV0IGVuaW0gYWQgbWluaW0gdmVuaWFtLCBxdWlzIG5vc3RydWQg
ZXhlcmNpdGF0aW9uIHVsbGFtY28gbGFib3JpcyBuaXNpIHV0IGFsaXF1aXAgZXggZWEgY29t
bW9kbyBjb25zZXF1YXQuIER1aXMgYXV0ZSBpcnVyZSBkb2xvciBpbiByZXByZWhlbmRlcml0
IGluIHZvbHVwdGF0ZSB2ZWxpdCBlc3NlIGNpbGx1bSBkb2xvcmUgZXUgZnVnaWF0IG51bGxh
IHBhcmlhdHVyLiBFeGNlcHRldXIgc2ludCBvY2NhZWNhdCBjdXBpZGF0YXQgbm9uIHByb2lk
ZW50LCBzdW50IGluIGN1bHBhIHF1aSBvZmZpY2lhIGRlc2VydW50IG1vbGxpdCBhbmltIGlk
IGVzdCBsYWJvcnVtLg==
------------------------ END TELETYPE DATA: v1 -------------------------

``

(I’ve used double backticks rather than triple because Markdown.)

On the subject of binary serialisation formats. Why do we need something like msgpack? If the Teletype ends up being the only device that can read the data then we are not working in the heterogeneous environment that those formats are designed for.

Why not just a raw dump of a versioned struct1, with code to forward translate versions? We’re going to have to do that anyway msgpack or not.

If we do have a binary serialisation format, then we need to take special care that the format is independent of the layout of tele_ops. If we do get it right, then a potential future benefit would be saving the internal scenes in the same binary format. Then we could let internal saved scenes persist between firmware flashes!

Of course, if we switch to a binary format, someone will need to break the news that everyone will need to retype all their presets…

1 With some header bytes to indicate the version.

2 Likes
#53

this is precisely the solution i came up with in my half-sleep last night.

other good points you’ve made, thank you.

i’m happy to go either way. text or binary. should we do an exhaustive pro-con?

one more point-- the read/write cycle actually takes a long time. i expect this would be optimized writing binary. but also, if blank scenes were not written/read it’d automatically be fast. so perhaps null on this.

2 Likes
#54

I think so. It’s a much bigger issue than the UI.

The UI is relatively simple given that it’s driven primarily by control limitations. It seems like we’ve generally reached consensus on some sort of hierarchic menu structure, even if we haven’t gotten the exact details sorted out yet.

Is there a way to insert a new post above the first post that can be made a wiki? Or does someone want to take responsibility for keeping a track of the list of pros and cons while we discuss them? (@cmcavoy?)

@sliderule given that you’re doing the work, I would be inclined to give added weight to your opinion. Do you have a preference?

1 Like
#55

Correct me if I’m wrong (I just woke up), but wouldn’t using a raw binary struct make backwards compatibility difficult? Wouldn’t you need to keep around old versions of the struct just to read in old scenes, then port them with explicit member assignment to the new struct?

I’ll parse the rest of this thread over my coffee.

2 Likes
#56

I hardly understand what all of this is about but following the discussion I wanted to say that the way it is discussed and how it seems to develop out of the collective process by those who have that understanding makes me feel very confident that it will be as good as it can be in the end!

:slightly_smiling_face:

5 Likes
#57

There’s a bunch of topics in this thread,

  1. A proposal to change USB stick support in the top post.
  2. USB hub support
  3. Serialization / deserialization implementation.
  4. Sharing TT scripts
  5. Human readable v. Binary serialization
  6. A bug in current TT USB implementation that prevents serialization of # being discussed in Github issue 104.

My suggestions for each topic:

A proposal to change USB stick support in the top post.

I like that @sliderule has been summarizing new ideas to his original proposal (top post) through calling out additional Facets. That makes sense. @sliderule can you edit your top post to reflect the additional facets in addition to calling them out in the thread? When ideas are invalidated or removed, can you also remove them from the top post? I’m thinking of the keyboard issue specifically.

The editor of the content has some power over the content, which is what you’re suggesting in the quote above. :+1:?

USB hub support

Start a new thread, top poster gets a chance to drive the conversation.

Serialization / deserialization implementation.

The discussion led to some key points about how users want to interact with USB storage, which is great - ideas around sharing and human readability came out. So - :+1: on not splitting them too early.

Sharing TT scripts

Given that there’s already a pattern for exchange any discussion that breaks that pattern should be treated as a breaking change. Meaning - it’s possible that this discussion is quickly running into TT 3.0 territory based on semvar.

Human readable v. Binary serialization

See above - I’d suggest making any changes here part of a 3.0 release.

bug 104

Given that it’s already in Github issues, and human readable / binary is a breaking change, I’d suggest fixing 104 without making changes to the 2.x serialization output format.

2 Likes
#58

Previous post was me being objective and trying to clarify the thread. This post is from @cmcavoy the user of Teletype who is also interested in features. I don’t want to mix the two @cmcavoy’s. BTW - I love that this is all going down on Sunday morning. It’s like Open Source Church. Amen

I have no idea if this is possible from an implementation perspective - but could TT act as a USB drive? Both the OP-1 and Octatrack become USB drives when connected to a computer. Content in both cases is human and machine readable via XML (I think? I can’t remember for sure). Not suggesting XML, but the pattern of ‘musical instrument sharing configuration as a USB filesystem’ is pretty well proven.

1 Like
#59

AFAIK msgpack has no schema (ProtoBufs do though). So in effect we’d have to keep multiple versions of something around.

I’ve been thinking about the process of (de)serialisation…

Consider we have a hypothetical type: scene_flash_t, which is the data stored in flash for 1 scene. No guarantees are provided about it’s format between firmware versions. It’s just for internal use only.

If we want things to be composable and testable we’d want to have some functions that transformed scene_flash_t into new structs that are then passed onwards in a pipeline that eventually ends up with the data on a USB stick.

What’s the pipeline going to look like with and without msgpack

Let’s define 2 structs representing 2 different versions of the binary format:

typedef struct {} scene_serialisation_v1_t;   // e.g. no timeline data
typedef struct {} scene_serialisation_v2_t;   // e.g. with timeline data

So saving to USB might look like:

  1. Use scene_flash_t to create scene_serialisation_v2_t
  2. Serialise scene_serialisation_v2_t
  3. Save to USB stick.

Loading from USB for a v2 preset:

  1. Read data from USB
  2. Deserialise scene_serialisation_v2_t
  3. Convert scene_serialisation_v2_t into scene_flash_t

Loading from USB for a v1 preset:

  1. Read data from USB
  2. Deserialise scene_serialisation_v1_t
  3. Either convert scene_serialisation_v1_t into scene_flash_t or convert it into scene_serialisation_v2_t and then use the exiting conversion from scene_serialisation_v2_t to scene_flash_t.

The question is, what does using msgpack in the above scenario gain over and above just dumping out the struct?

1 Like
#60

Mainly I was suggesting that @sliderule is the one giving up his free time to tackle a difficult problem. Ultimately he has to want to write the code, and thus in a material sense, his opinion on everything matters more than mine.

Essentially the logic goes…

  1. We want timelines…
  2. Then we need to make some changes to USB loading and saving.
  3. Well… if we’re going to put the effort in, might as well get it right.
  4. Drat. Now we need a new file format.
  5. Ah, bump the version to 3.0 :sunglasses:

Yes it could behave like a USB drive. Is that really what you want the very limited developer time spent on? There would probably be more effective ways of doing it without the “drive” bit. But the entire discussion belongs on the “USB hub” topic…

2 Likes