A big update on my parser branch…

https://github.com/samdoshi/teletype/tree/parser

The entire parser is now running via Ragel, both scanning for tokens and matching them.

Code sizes, master branch first, parser branch second:

section               size     size
.reset                8204     8204
.rela.got                0        0
.init                   26       26
.text                55728    66080
.exception             512      512
.fini                   24       24
.rodata              14732    15140
.lalign                  4        4
.dalign                  4        4
.ctors                   8        8
.dtors                   8        8
.jcr                     4        4
.got                     0        0
.data                 7492     7492
.balign                  0        0
.bss                  8136     8136
.heap                74456    74456
.comment                47       47
.stack                8192     8192
.flash_nvram        147076   147076

That makes the flash ROM 10.5kb bigger:

(66080 - 55728) + (15140 - 14732)  = 10352 + 408  = 10760 bytes

I have tested the code on device, but not thoroughly. I have tested loading in a slightly complex scene from USB. I also haven’t benchmarked it, but my gut (and brain) tell me that it should be several orders of magnitude faster.

What’s the next step then? Anyone want to help test? Or should I open up a PR and invite comments from there?

2 Likes

amazing! i’d say 10k is no big deal at all given the improvement.

i’ll do some testing! it doesn’t seem that there are too many points of failure though?

1 Like

Apart from the usual segmentation faults, etc. (which I don’t think there are too many opportunities for), it tends to be weird things, e.g. for a while TADD 1 1 would be the same as ADD 1 1. It would parse TADD as T followed by ADD. I added some code to count how many operators were matched. It’s possible that a better way would be to return after the first match, whilst also asserting that the match has consumed the entire token.

I would also feel more comfortable if you had a look through scanner.rl and match_token.rl and make sure you feel confident with how they work. I plan on sticking around to provide help and support, but you never know…

I’ll also freely admit to not really knowing how most of the bits of Ragel work. We’re in a special use case (the => operator for scanning) which feels quite orthogonal to all the other ways to use Ragel.

Hi @tehn I noticed on another thread that you’re currently working on some teletype & ansible code.

Would it be possible to coordinate somehow as I think my parser changes might be a tricky rebase if you’re going to make the teletype changes I think you’re going to make (i.e. new ops for ansible).

Would you consider forking monome/teletype to tehn/teletype and working on it there? There would be a couple of advantages…

  • It’s socially acceptable to force push to your own fork, when you can’t to the main repo.
  • I could manually grab your changes and reimplement them on my branch before the merge to master on monome/teletype.
  • If you open up a PR from your branch to master you’ll get the benefit of the automated tests running and checking your branch. (You can also get this working on your fork for any push.)
  • If you wanted I’d be happy to review any code if you did open a PR.

Alternatively, could you make the changes on a new branch in the main repo.

Also while I’m “needy mode”, would it be possible to adopt semantic versioning going forward (i.e. 1.2.1 instead of 1.21).

yes to all of the above!

though i’m afraid i’ve already added quite a bit of code-- i’ll figure out how to move files around to prevent any trouble and not commit directly. FYI i’m really only editing hardware.* and ii.h to get more ansible ii ops in there-- not sure that collides with your edits.

semantic versioning, yes! so after i add these ops we’ll be at 1.3.0

ok, i’ve successfully moved all my changes over to my fork. i’ll from now on behave like a responsible community member on this project-- thanks for the push.

1 Like

It’s not so much the edits to hardware.c as the edits to op.c to enable the extra ops that will mess with it.

Do you want to make your changes on tehn/monome and we can have a look and figure the best order to do things

i’ll submit a PR when the new ops are ready-- hoping to finish this all by the end of the day. honestly not doing anything breaky, i think. also i’ll be sure to stay on top of my formatting.

Okay, open the PR, and I will have a quick go at rebasing my changes this evening (or tomorrow morning) to make sure it’s not going to cause me too much grief. Sounds like it should be straightforward though.

Okay I’ve added the PN versions of all the ops in. patterns.c has been heavily rewritten to reduce duplication, I’m really hoping I haven’t introduced any bugs, but it’s a lot tider now.

I’ve also added some tests to check that all ops are unique and that each op really manipulates the stack as it says it should. (FYI, while I was playing with merging the code, the test told me that LV.RES has the wrong .param size, it’s set to 0, it should be 1).

@tehn are your changes on tehn/teletype finished? I’ve had a quick go at trying to merge your branch and mine, and we’ll end up with a merge conflict. It’s not a particularly hard one to fix, and I’m happy to do the merge. We just need to decide which branch we merge to monome/teletype first.


Also FYI, the tests are broken on your master branch, you need to add the following to tests/main.c

void tele_ii_tx_now(uint8_t addr, uint8_t *data, uint8_t l) {}

And cough make format

3 Likes

thanks for the tips. will have a minute later tonight to wrap it up with a PR

late late late, but PR is in.

2 Likes

Nice one, it’s (past) my bedtime now. I’ll have a quick look in the morning and make sure there is nothing that will cause me too many problems. Then you can merge the PR? And I will merge those changes into my branch.

Looks good to me. (Also left the same message on PR.)

I’ll merge the changes into my parser branch this week. I’ll probably rename some of the structs to match the op name (I’ve made similar changes in my branch for other ops).

Do you have any more short term plans for the teletype code? I’m trying to decide if I should PR my changes soon, or wait until I’ve taken a stab at the ; code.

no short term plans-- must push through more production work first. but also regroup with you all as to what the priorities are with the TT codebase and feature extensions.

4 Likes

I’ve merged monome/master into my parser branch.

I’m going to try and do the ; multi command code over Christmas, and then go for a big PR in the new year. It would be nice to get most of it wrapped up before 2017 gets going.

4 Likes

I have a brand new TT arriving tomorrow and very interested in the progression of the firmware and making life easier to add custom ops (one of my driving factors in picking one up) - so having the ragel-based parser is awesome. I’ve already built a version from your parser branch and plan to give it a run out of the gates, sam.

I’ll let you know if I run into anything via github. Good work on this!

4 Likes

Good to have you on board @adammokan

If you do want to get started adding your own custom ops there are some instructions in the readme on my parser branch. Do get the tests running as there are some that check that all the values are correct.

I’m going to have a think about the best way to make custom ops easy to maintain, ideally we want to be able to easily git rebase our custom ops branch. I think I’ll need to modify op_enum.h to drop the explicit numbers, and I might add a custom.c file to store them in too

2 Likes

Just forked your branch to snoop around.

I was thinking about implementing a STOP operator to halt script execution. You’d have to interrupt the run_script loop to do so:

process_result_t run_script(size_t script_no) {
    process_result_t result = {.has_value = false, .value = 0 };
    exec_state_t es;
    es_init(&es);
    for (size_t i = 0; i < tele_get_script_l(script_no); i++) {
        result = process(&es, tele_get_script_c(script_no, i));
        if (result.value == MAGIC_STOP_NUMBER)
            return result;
    }
    return result;
}

Which comes from process():

if (cs_stack_size(&cs)) {
    process_result_t o = {.has_value = true, .value = cs_pop(&cs) };
    return o;
}
else {
    process_result_t o = {.has_value = false, .value = 0 };
    return o;
}

As far as I can tell, nothing cares about the final return value for the command state stack, which makes sense, as there is nothing more to consume the value at the end of the command.

Would it be safe to use a magic number here? What is the scope of the .value returned from any of the current operators, i.e.: is there a safe magic number?

I’ll keep familiarizing myself with the operators to try to grok the situation, but some of the bounds are going to be elusive.

1 Like

Of course the other solution is to add a stop flag to the state and then check for (and clear) that.

Magic numbers are dirty, but they’re cheap. :slight_smile: Not that we have a memory problem that couldn’t support one more state flag.

1 Like