Ii bus board, v2, with pullups

bpcmusic · 2017-04-04T05:23:15.418Z

@trickyflemming - interesting. But, what hat were you wearing??

My gut feeling is this isn’t bus related, but something else. Especially since your Q3 test came up clean.

trickyflemming:

I just tried out Q2, and it failed after about 15 minutes.

I had worried that there was something with the TO.TR.M implementation in the expander that was unhappy with the high frequency manipulation (updating every 50ms) - it might be able to hang the TT with a catastrophic error. That you still got the hanging with the simpler command (via Q2) suggests that this isn’t the case (as well as the fact that I couldn’t get it to hang).

Might want to try the tests with one of those vizors with the integrated spiky blonde hair and see if that helps / hurts.

–

I should point out that the expander’s implementations of metronomes and oscillators run externally to the Teletype once initiated. The TT can hang and the oscillators and trigger metronomes will continue to go on forever. They also do not reset on preset swaps on the Teletype. You need to either turn the features off or initiate a reset command to turn them off.

trickyflemming · 2017-04-04T06:07:11.396Z

No hat, but I switched from striped socks to solid. My Teletype is currently facing East, if that matters.

What I meant by the TXos continuing to function was not that they should have crashed, but that they strangely changed scenes, if that makes sense. It’s almost as though the I script on the previous TT scene was called when it locked up. They went from simultaneous blinking with random CV (scene 9) to clock division and static CV (scene 8). I’m wondering if something is overflowing somewhere, and scene 8’s I script is getting triggered as a weird side effect.

Is Q1 still useful at this point, or should I investigate Q4?

Leverkusen · 2017-04-04T10:35:42.416Z

I haven’t had time to build mine by now but I wonder if there are any positive experiences too where the new bus board solved issues that have been there before using it.

We could tell then if this case is some kind of singularity and an individual solution has to be found or if there still are issues that don’t get solved by the changed pull up resistance.

Anyone tried changing the resistors on teletype yet?

sakul · 2017-04-04T14:29:04.704Z

I don’t understand why you are suggesting this, when the powered bus board has been developed by people that understand more about the issue

Leverkusen · 2017-04-04T16:18:26.497Z

I do not understand what kind of suggestion I made to anybody with my post?

The bus boards have been introduced as not tested but expected to solve the remote issues. So what is wrong about asking for more feedback to understand if they really do?

trickyflemming · 2017-04-06T03:48:55.260Z

@bpcmusic and @sam: I tested Q4 (downgrading to 1.4.1) tonight. It crashed after 8 minutes of running the TO + Jumpy Edges script.

There’s not an official 1.4.1 binary on the Github release page, so I used the one posted by @tehn here: Teletype Firmware i2c Debugging

Galapagoose · 2017-04-06T03:57:21.472Z

The most helpful you could be is to build your bus board and provide another data point! That will certainly push us towards clarifying that the bus board does indeed solve the problems people have been having.

Leverkusen · 2017-04-06T04:50:16.913Z

Yes, of course - I am sorry if I was being stupid.
Just finished my mouser order.
It was a bit tedious and I hope everything fits on the board when it arrives…

sam · 2017-04-06T08:32:01.784Z

Another test (sorry!). You’ve got an Ansible connected up right?

Can you modify a failing script to use the Ansible outputs instead of the TXo outputs and see what happens?

bpcmusic · 2017-04-06T14:29:53.719Z

@Leverkusen - bus boards passed with flying colors for me. I was able to turn off the inelegant hardware pull-up in one of my expanders and run a fairly large configuration solidly. I’m not sure what is going on with @trickyflemming right now - but he does have quite a lot of variables at play. I don’t think it is his bus - or at least not in the way that I’ve seen the bus fail. Those issues were very replicable and were all cured by the bus board for me.

@trickyflemming - how are you patching and triggering that script? also, have you tried running it at a slower M? Perhaps you are overloading the TT slightly and it is going into an overrun condition after time. (@sam - i like your suggested test too!)

trickyflemming · 2017-04-06T14:36:45.023Z

@sam: I’ll run the Ansible test later today.
@bpcmusic: I’ve posted the I and M scripts. For 1-4, they’re copied from Jumpy Edges. For the patch itself, I’m taking the first four TR outputs from the TO and plugging them back into the Teletype. Essentially, I & M setup a clock division patch on the TO, M & PARAM work together to control the speed of that patch, and then that patch is used to control Jumpy Edges.

My plan with learning the expanders was to go through the Teletype Studies a second time, but this time with an eye on how to use the expanders to build upon each concept.

trickyflemming · 2017-04-06T14:50:58.796Z

@sam: How should I use the Ansible in your proposed test? I’m using the TO’s metronome capabilities to set up a clock, and the metronome speed is updated on the M script (at 75 ms refresh). Should I just use, say, the clock outputs from Levels and ignore M? Should I use Kria and update KR.PERIOD from the TT’s PARAM knob?

sam · 2017-04-06T15:16:02.247Z

Can you post a txt file of the script you are using please.

trickyflemming · 2017-04-06T22:47:50.979Z

tt00s.txt (762 Bytes)

Had to re-update to 2.0b to save >_<

Just connect the 4 gate outs from the TO expander to the TT script inputs 1-4. The order doesn’t matter. Any level of PARAM works, whether super fast or slow. It will crash around 10 minutes or so. The record was about 20 seconds I think.

trickyflemming · 2017-04-08T19:15:06.284Z

I have free time tonight and tomorrow. Anything else that I should test?

sam · 2017-04-09T08:10:25.079Z

Sorry for not getting back to you sooner, the school holidays are just starting here in the UK, and the last week of school before the break is always busier than you anticipate.

Anyway, first a little background… what we’d like to do is create the simplest script that fails, but we also want to be open minded. It’s possible (probable even) that the bug has nothing to do with i2c, my vote is either a race condition or memory corruption in either the Teletype or the TXo firmware.

So, what I’d like to do is starting removing bits of script / functionality until we stop getting lock ups.

This is the order of operations I’d like you to follow if possible (all based on the CRASHTEST script you posted).

Make sure that CRASHTEST as is still fails.
Remove all the cables from the TXo to the Teletype. Does the Teletype still lock up?
Remove the contents of scripts 1-4 and test:

#M
X SUB 320 RSH PARAM 6
L 1 8 : TO.TR.M I MUL I X

#I
L 1 8 : TO.TR.M.ACT I 1
M 50
M.ACT 1

Test with just 1 TO.TR.M.ACT

#M
X SUB 320 RSH PARAM 6
TO.TR.M 1 MUL 1 X

#I
TO.TR.M.ACT 1 1
M 50
M.ACT 1

Remove PARAM:

#M
TO.TR.M 1 <what's a sensible value here?>

#I
TO.TR.M.ACT 1 1
M 50
M.ACT 1

Can you do those and report back please?

trickyflemming · 2017-04-09T19:16:23.816Z

Thank you for the detailed response!

Okay, I’ve tested the following so far. For each test, I’ve marked it as “stable” if it ran for 30 minutes without a lock-up

CRASHTEST as-is: Crash. (Test 1)
Remove all cables from the Teletype: Stable (Test 2)
Remove all cables and remove scripts 1-4: Stable (Test 3)
Keep I, M, 1-4. Trigger scripts 1-4 from Varigate: Crash (Just to test a different external trigger source)
Keep I, M, 1-4. Trigger scripts 1 & 3 from Varigate: Crash (Since 1&2 and 3&4 have command overlap, I wanted to see what happened if only 1&3 were triggered)
Eliminate I & M, Keep 1-4. Trigger scripts from Varigate: Stable.

One test I’m running right now is keeping Scripts 1-4 and adding an M script that gets called every 50 ms. This script simply calls “TR.TOG 4”. So far, it has been stable for 15 minutes. I’ll update this thread later after running a few more tests. After this, I’m going to call TR.TOG on the Ansible, followed by TO.TR.TOG. I want to see if updating those from the M is unstable. Finally, I’ll run your proposed tests 4 and 5. Since Test 3 passed, I imagine that 4 and 5 will be stable, though.

sam · 2017-04-09T19:41:14.322Z

trickyflemming:

Remove all cables from the Teletype: Stable (Test 2)

wasn’t expecting that!

So, let’s start from CRASHTEST again…

Can you try the following sequence (again we want to minimise our failure case)

Keep CRASHTEST as is. With the 4 cables connected, but on each test pass, remove a cable. Until you have no cables. At what point does it go from crashing to stable. (Please power cycle between tests.)
Connect all 4 cables, but delete scripts 1-4.

And then on a separate track, use the varigate to trigger the Teletype. Start from CRASHTEST but modify M and I as such:

#M
X SUB 320 RSH PARAM 6

#I
M 50
M.ACT 1

If that still crashes, then modify M to: (keep I as is)

#M
X 0

and then if it still crashes delete the contents of M (but keep I as is).

I hope that’s not too hard to follow.

trickyflemming · 2017-04-09T19:51:57.209Z

Great. About to test that, but one more data point…

I just changed my M script to TR.TOG 5 (with the Ansible in Teletype mode) while using the rest of the Jumpy Edges script (no TO code in there… that was the only line in the M script). It crashed within ten seconds.

EDIT: As an aside, here’s my thought as to what is potentially happening. It seems that the Teletype is internally stable. I was smacking it pretty hard with the Varigate while the M script was flipping one of the main toggles. Everytime I do anything involving the ii bus on the M script while using the main TT script triggers it seems to crash.

I think there may be some weird race condition involved with the M, ii, and trigger sampling schedulers, but only when using all three.

With my “No Triggers” test (M script triggering an ii message every 50 ms), things were stable.
With my “Lots of Triggers” test (M script triggering internal message every 50 ms), things were stable.
As soon as I have triggers, M, and ii together, that’s when it seems to crash.

trickyflemming · 2017-04-09T20:05:16.272Z

Okay, just ran all of the #1 tests. It crashes with any number of cables (4, 3, 2, or 1). So, I suppose it’s only stable with no cables there.

On to test #2.

EDIT
Test #2, part 1.
Deleting scripts 1-4 but leaving the cables in leads to a crash. This is making me believe more in my scheduler race condition theory.

ANOTHER EDIT:
I’m about to test the M thing for Test #2, although I think it’s pretty similar to the TR.TOG M test that I ran above. I looked at the STACK_SIZE and STACK_OP_SIZE defines at https://github.com/samdoshi/teletype/blob/2.0/src/state.h and noticed that they’re defined to a constant 8. I’m not too familiar with the Teletype code, so this may be way off. However, my M script is sending 8 messages to the ii bus each tick. Is it possible that there’s a situation where the various schedulers are trying to push to an already full stack?