#82

this is a visual defect bug, i’ll track it down when i have a spare second (and feel like the functional stuff is ironed out). previously reported by @Leverkusen

may be too much, polled at 10ms. but of course, polling slower than this will introduce quantization. we’re coming up against a technical constraint, so it’s time to choose musical constraints that fit within the technically possible.

i’m not certain, also, that we can completely crash-proof tt. certain scripts are going to be possible which will certainly take the system down. ie, L 0 V 10 : CV 0 I takes almost a full second to execute. stick that in a metro with a time shorter than 1000ms and bonk

short of revising the event system and tracking down every crash scenario, we might need to just accept that user error (or overconfidence) might break tt in its current hardware implementation.

#83

I haven’t really been keeping up with this thread, but… if there is always going to be a danger of locking up the TT would it be wise to enable a watchdog timer of some sorts (I’m assuming one is available)? That way the TT just reboots rather than forcing a power cycle of the whole case.

4 Likes
#84

yeah, i’ve been thinking that even if/when we resolve the i2c issues the next big one will be timing. doing things syncronously helps to fix weird i2c issues but at the cost of timing accuracy. this is unavoidable unless we have a dedicated CPU for each task, but we don’t, so we need to consider this constraint (and this is where i’d like to reiterate again that we are speaking about more extreme use cases, and TT is already incredibly capable as it is).

with that some things to consider:

  • timing will never be 100% perfect but we could try and get it to be as close as possible
  • what is more important? tigher timing or not skipping commands? if M execution takes longer than the metro interval, or if a trigger comes in before a script finished executing, should we skip the rest of the script or skip the trigger? perhaps this could be configurable. and if you choose the option of not executing scripts fully, and a script is cut short we could indicate it on both live and script pages with an icon of some sort, so then you’ll know when you are hitting the limits and can adjust your scripts or your trigger/metro rate.
  • we could also limit the number of commands per script but this feels like the least ideal option as in order to make limits work they’d need to account for worst case scenario, and it feels like an unnecessary limitation.
  • regardless of the implementation i think we should try and get it to the point where it’s able to recover gracefully.

in regards to that last point i’ve been reading up on i2c recovery and looks like there are drivers that account for a possible i2c lock (which by descriptions sound very similar to what we see, SDA is pulled down and is not released likely because the slave is waiting for SCL) and have mechansims to recover from that, usually by implementing a timeout and getting the bus into a healthy state again. ASF implementation does not seem to have that. we could implement something similar but this means tinkering with the ASF code, and i don’t know if we want to open that can of worms…

1 Like
#85

honestly i don’t need to scrub the patterns that fast for my musical needs. most often i’m using the the pattens for quantization. if i want un-quantized scrubbing i’ll use the ansible cv outs. that said i was still able to lock up TT polling at 250ms.

#86

@scanner_darkly - I was able to put a timeout in the infinite wait loop in the ASF library to bail out of unanswered commands. I was not able to figure out the series of voodoo commands required to restore the bus to a happy state. I tried a number of things there but was unable to get success. I’m just not familiar enough with i2c and with the ASF library to see the answer.

I do think we need to implement something like that to keep users from mis-typing a command and losing their work. With no Ansible connected, hitting CV 5 will lock your TT and you will loose all of your unsaved work. This is a pain in the butt for a studio session and a nightmare if you are live coding in a performance.

An easy way around this would be to “register” bus devices in the TT. The TT stores that registration to flash memory and then bounds checks any addresses at the time of the command - skipping devices that are unknown. Registration and deregistration would be a one-time affair.

Even if we do that type of thing, the aforementioned timeout and recovery would also be a good idea for the ecosystem so that it is more resilient.

Around timing, I’ve been able to pummel the bus with write commands without problem. 24 Triggers and 24 CV outputs writing values at 10ms intervals with only slight timing discrepancies. I put that in the “wow” category for what the TT and its 400mHz bus can do.

The reads are another story. They are slow and eat up bus time. I’ve avoided supporting read operations for the TXo for this reason; you don’t get a value from TO.CV 1 as it doesn’t bother calling to the device to get its state. I always figured that if this was desired, it would be far easier to cache the value on the TT and simply return it from there.

For the TXi, reads are unavoidable. One level of caching would be to remember values for the duration of a script so that subsequent calls for the same parameter would return from memory as opposed to another bus read. A second level of caching would have us move reads to a lower priority background thread and have it poll at a safe and reasonable rate (and for a single device at a time as opposed to input by input). Scripts can then read at any rate they want without causing havoc on the i2c bus and the values can update in the TT without affecting timing.

Lots of options here - thoughts?

#87

this just isn’t happening for me. i can have a standalone TT, no cables attached, and attempt to read/write via i2c and it aborts just fine?

do they work in non-metro territory? do reads work to ansible correctly on your setup? (mine do, at this point).

this feels like it falls into the bad-user-script category. i think any caching would make operations more murky-- meaning you’d potentially be unsure if you were getting a live or cached value-- at the expense of trying to prevent bad user scripts.

having a constant-poll system isn’t a bad idea, but it requires management as to what values the user “cares” about. so you’d need some sort of INPUT.ACT status bits, like metro being on or off. i feel that this should also simply be managed by the user. particularly once the v2 feature AUTOTRIG gets implemented, where we have a timer per input channel, hence are less bounded by the existence of only one metro for user processes.

so at this point, is reading working at all for you?

#88

this just isn’t happening for me. i can have a standalone TT, no cables attached, and attempt to read/write via i2c and it aborts just fine?

This is a given for me. Try to send to an II device that isn’t connected, the TT locks up. This has happened for me with my expanders and with the Ansible. If you disconnect a device and forget that the current preset has calls to that device, you have to reconnect the device or re-flash your TT in order to get back in.

do they work in non-metro territory? do reads work to ansible correctly on your setup? (mine do, at this point).

Reads now work both in Metro and non-Metro environments at this point if I have 3 or fewer devices connected. More than that and they become unreliable - locking up the TT after an indeterminate amount of executions. I’m planning to experiment more with this in the evening as the additional devices are my expanders. (I’m going to hard-set their i2c rate to 400 and see if that helps; I had been letting them autonegotiate down.)

IN and PARAM polling / caching

Yeah - I’ve struggled with this conceptually quite a bit as I try to think around my read difficuties. What is happening now is clean and clear and nice. Completely under the control of the user writing the script. I don’t think we would need to go to the extent of caching and polling unless we just don’t get the performance we want when reading from multiple sources at a reasonable rate.

1 Like
#89

re: i2c - i saw the sequence for getting the bus back into healthy state somewhere, i’ll post if i find it. i think addressing a non existing device is a separate issue and should be easy to fix (since in this case no ACK coming from a slave whatsoever, vs a slave keeping SDA down).

pretty sure i was able to get TT to lock when addressing a non existent CV too. also when trying CY.RES i managed to get TT to lock immediately - i think this was because ansible was in the TT mode and not Cycles, but not sure 100% since i only had time for a very quick test. i’ll check again when i get a chance.

so what about the idea of interrupting a script if it gets triggered again before it finishes executing, and indicating it with a flashing icon or something?

1 Like
#90

fyi, ansible changes i2c addresses based on what mode it’s in. so these are potentially the same issue.

that’s probably a rational thing to check out, good call

1 Like
#91

Interrupting the script seems wrong to me – who knows what you’re interrupting and what effect it will have on the scene? I get that it’s in the territory of bad-user-script, but it seems wrong to not treat a script as atomic, at least to itself. Triggers from a random source aren’t always predictably sparse.

I’d rather see the second trigger dropped, or maybe queuing up the script to run a second time with a maximum queue depth of 1.

Just my 2 cents.

(Of course you can effectively give your scripts self-atomicity by muting itself at the top and unmuting at the bottom.)

#92

that’s why i suggested a visual indication as well. you will know your scripts are not executing fully and have a chance to fix it by lowering the rate or editing the script.

1 Like
#93

found a description of a method to clear the i2c bus:
http://www.forward.com.au/pfod/ArduinoProgramming/I2C_ClearBus/index.html

potentially related to the issue we’re seeing:
http://processors.wiki.ti.com/index.php/I2C_Tips#External_Slave_Device_Hanging_the_Bus_by_Holding_SDA_Low

2 Likes
#94

Since I am on the latest teletype firmware teletype occasionally gets in a state where the trigger pulses don’t go down to zero anymore. It is still running scripts (relatively simple ones w/o i2c commands) and it is possible to turn the pulse off with TR.TOG but it will get high with the next input trigger running the script and then stay high again. It’s a bit strange since it is running for quite some time before this suddenly happens…

Ii bus board, v2, with pullups
#95

I think that you guys nailed the code for i2c; it is solid for both reads and writes with a small number of devices. In this configuration, if you type the wrong address for an output or input - no foul. Things just keep on running.

As the device count grows - the performance starts to diverge. I’m assuming that this has to do with the aggregate bus resistance and the pull-up resistor value that is in the Teletype. Here is what I am seeing:

  • Writes are solid at high rates up to a large number of devices. I was able to PUMMEL 6 TXo devices while 8 total TX devices were on the bus (+2 TXi). Writes were fast with hardly any perceivable timing discrepancies. We’re talking 24 Triggers and 24 CV values updating every 10ms. Runs and runs and runs. Wow.

  • The stability of read operations decreases with the addition of devices to the bus.

  • The odds that the TT will freeze on the input of an address that is not available (for read or write) goes up precipitously with the number of devices on the bus.

TESTS (all TX units hard-set to 400mHz i2c bus)

2xTXi + 1xTT [AWESOME FOR READS]

  • no hanging on bad ports
  • 10ms reads from 4 pots to CV 1-4

2xTXi + 1xTXo + 1xTT [GOOD FOR READS]

  • no hanging on bad ports
  • 10ms pulses to TO.TR 1-4
  • 10ms reads from TI.PARAM 1-4 to TO.CV 1-4
  • 10ms reads from TI.PARAM 5-8 to CV 1-4
  • froze only after ~5 min

2xTXi + 2xTXo + 1xTT [FAIL FOR READS AND MISTAKES]

  • hangs on bad ports
  • only got to test 1000ms reads from TI.PARAM 1-4 to CV 1-4
  • froze after 2s

I am running the shortest cables I have between units.

This doesn’t seem to be a software thing anymore - it is rock solid for 2 external devices plus the TT. It is only when you add units on the bus that things go downhill. I saw the same behavior with the Ansible the other day; I don’t think it matters what is connected - just how many devices.

6 Likes
#96

how is the situation with less frequency? I’m asking because TT + three monome modules already isn’t a rare scenario… adding txi+txo… hmm

nevertheless, great progress everyone, thank you!

#97

Frequency doesn’t seem to matter - just # of units. Frequency is just a good way to get it to lock up faster in that “almost stable” state. It’s also a good way to identify if a certain configuration is stable.

I should be super clear by the way that I’ve only been testing with configurations that have my expanders, the TT and Ansible in the mix. The expanders are a completely different platform then the trilogy modules. It could be that they are the wildcard causing the problem.

We should have some more definitive answers soon as more tests are performed by more people. I will probably pull my other trilogy moduses and bring them into the mix.

2 Likes
#98

do you see any significant drop in voltage level when adding more devices?

1 Like
#99

I’ll scope it tonight with my junkyScope. Picoscope hasn’t arrived yet. :wink:

BTW: playing around before leaving to work - added Ansible back into the mix. TT still borked out on nonexistent ports (eg CV 9 V 10). With Ansible + TT + Expanders on the Bus, the TT seemed to be more resilient when reading from Ansible - but I was still able to get it to lock without making any calls to the expanders (just Ansible).

2 Likes
#100

You could try hanging an extra set of 10k pull-up resistors onto the bus lines. Of course would be best to have a definitive test-case for “this particular set of commands takes x-minutes to crash when n-devices are connected” and see if that changes.

1 Like
#101

Good idea; I’ll give it a shot. :slight_smile:

1 Like