#6

Just a little vote of support & encouragement here! I know some of these issues relate to the Just Type firmware for JF as well, so excited to see things move forward.

On my side, I’ve noticed TT can freeze when sending multiple commands to JF from the same script. The receive code is triggered by interrupt, grabs the data on the i2c line and dumps it in a buffer. That interrupt is on a higher priority IRQ than the audio DSP loop, yet still when the DSP is running a more intensive mode, the freeze (on TT) is more likely – I can’t explain why!

Hope that’s not too OT.

2 Likes
#7

Also words of encouragement from me, hoping for the fix!

I have experienced TT freezing with my trilogy on many occasions in the past, and I never thought that this might have been related to Metro functionality. But now that I think of it it could very well have been the case, especially with Earthsea, when I was hoping to clock it from TT…

#8

I didn’t mention my half-baked, harebrained theory above for the freezing on read issue. My gut is telling me that the issue is related to a contention between interrupts which causes the TT to miss out on the receipt of the response and sit (relaxed) waiting for it.

I could be biased to think this because I’ve spent a lot of time looking here trying to figure out how to recover from calls to modules that aren’t connected:

Miss out on that return value or accidentally call to a module/output not on the bus and it is nappytime … forever. That is, at least until you power cycle. I’ve been able to tweak that library file to keep the unit from locking up, but I’ve not figured out how to bring the i2c bus back to life in a way where it will function and not lock up on future calls. My ignorance of the AVR TWI implementation (and the all over the place documentation) have kept me from solving this.

EXPERIMENT 1

This helps bolster my theory. I am able to read values for a much longer period of time if I disable the metronome and trigger the i2c input sampling from an external clock source. The TT will still lock up, but it takes a lot longer and happens more randomly than the near-instantaneous freeze when using the TT’s internal metronome interrupt. See the example video below.

I’m triggering SCRIPT 1 from the clock out of the 0-coast and am sampling its rise/fall generator via TI.IN 1 and pushing that value to CV 1 .

I’ve monitored the TXi and it gets and responds to all of the requests for data up until the moment of the freeze.

1 Like
> teletype : code exchange
#9

i2c reads need to happen conditionally-- we need to be checking if the i2c write (just prior) succeeded. otherwise the system will lock. should be an easy first fix. but there are other things lurking that are clearly causing trouble.

2 Likes
#10

So great to see you are having progress with this. I made a “yip” when I saw the notification email. :slight_smile:

Quick question: I’d played around with (and reconfirmed) that if I comment out the the tele_ii_tx_now just before the read, it will still lock up immediately during the metro event and eventually during a triggered script. Is this the write you are referring to?

Thanks!

b

1 Like
#11

been looking into this, got a picoscope recently and remembered it does i2c decoding so thought i’d try that. confirmed first it decodes properly with orca remotes. then tried reading CV from ansible. no freezing when triggering manually (red is SDA, blue is SCL):

graph1.jpg719×328 150 KB

^ this is the last part, the 2nd byte in the ansible response to a write request which is not shown here. it returns data properly (in this case 0F). then i tried triggering it repeatedly by using WIN+1 to trigger a script and holding the buttons so it would keep calling it, and in a few seconds it froze. the read request from TT looks fine, as does the first byte sent by ansible but on the 2nd byte here is what happens:

graph2.jpg722×328 155 KB

looks like in the first graph it properly lets SDA go high after SCL is high to signal the end of transmission but in the 2nd one something pulls SDA back down and it stays down.

i’ll try more scenarios tomorrow - let me know if you want me to check anything specifically, thought i’d post this first in case it proves helpful. i’ve been reading up on i2c and looking at the code over the holidays and so far it does seem to be some sort of race condition… but that’s just a SWAG at this point.

5 Likes
"Just Type" Beta Testers
#12

I started to experiment with trying to use a bus pirate board to snoop the i2c bus last night but don’t yet understand how to use the bus pirate properly so it hasn’t yields valuable results yet. If there is anyone out there with bus pirate experience I’m all ears.

…I’ve started work on sketching out some new ansible arp ideas (configurable via tt) so I expect to be pounding on the ii code during testing. Hopefully another set of eyes on all things ii will be a positive. Now off to further familiarize myself with the code and the spec.

5 Likes
#13

good data points. i’m on this also this week, but i must admit the issues are not straightforward so any extra opinions and investigation are most welcome

fyi, SDA/SCL pullups:

ansible : 0
trilogy: 0
tt: 10k

1 Like
#14

also just noticed looking at the graphs there is no ACK from TT, not sure it’s related to the issues we’re seeing though, and don’t remember if it ACK'd the first byte or not - should’ve posted the full sequence, will do that once i get home.

1 Like
#15

sorry, just an info dump at this point, going through the code and cross referencing it with the graphs.

CV 5 16383
seems fine, can’t figure out why the address is 20 when judging by the code it should be II_ANSIBLE_ADDR 0xA0

image.png1459×815 17.1 KB

CV 5
also seems fine. TT sends II_ANSIBLE_CV | II_GET, then requests a read, and ansible sends 2 bytes. the only weirdness i see is a delay between the 2 bytes sent by ansible and TT not ACKing the 2nd byte:

image.png1567×757 16.8 KB

image.png693×723 8.36 KB

image.png690×720 8.84 KB

CV 6 CV 5
the start is similar to the above, TT sends a request to read CV, ansible returns 2 bytes.

image.png1178×729 15.3 KB

image.png690×723 8.76 KB

image.png771×736 9.83 KB

now the interesting part is that after TT receives the 2nd byte (same as doing CV 5) it starts a new transmission to set CV 6, but see how the 1st data package after the address package is stretched and it doesn’t transmit the clock for a while (either that or perhaps it’s ansible stretching the clock):

image.png738×730 11.2 KB

image.png702×731 10.2 KB

the behaviour with clock stretching seems pretty consistent.

2 Likes
"Just Type" Beta Testers
#16

one more observation: when trying L 5 8 : TR.PULSE I TT does send 4 transmissions. the first seems fine but in the last 3 i’m seeing similar clock stretching. interestingly enough L 5 8 : TR I 1 works (the outputs get set) even though i’m seeing same clock stretching, so might not be a factor in itself. and then if i try TR.PULSE after that outputs 1 and 3 pulse and 2 and 4 stay on.

#18

@ngwese added your patch

pretty reliable read/writes by commenting out lines 95-102 in ansible_tt.c (on the ansible)

void ii_tt(uint8_t *d, uint8_t l) {
	// print_dbg("\r\ni2c:");
	// print_dbg_ulong(l);
	// print_dbg(" ");
	// for(int i=0;i<l;i++) {
	// 	print_dbg_ulong(d[i]);
	// 	print_dbg(" ");
	// }

but, i’m still seeing NACK on all the read requests also (with the analyzer). this is perplexing. i want to trust the ASF, but it’s worrying me.

but, it’s working, with the NACK and everything. but does not feel good.

i can still eventually overwhelm the event queue, but it takes a ton of at-once commands in succession.

hope to have more to report later.

1 Like
#19

@tehn i’m assuming this is the patch you are talking about. i think a bunch of it is not needed so i’ll generate another more minimal PR instead.

i haven’t run into any problems yet writing from tt -> ansible during development. …going to implement a few more commands today which require more invasive changes then take things out for a test drive and try to break it.

…i had that same feeling while trying to figure out why the usb stack was locking up back a month or so ago…

1 Like
#20

Are the reads working on the TT’s Metro event for you with this change in the Ansible?

#21

looking at the ASF code it appears NACK on reads is intentional to stop the slave from transmitting any more. which seems to go against i2c protocol but makes sense from a practical point of view (apologies if my interpretation is naive, i have a very limited knowledge of i2c).

  twi_masterBufferIndex = 0;
  twi_masterBufferLength = length-1;  // This is not intuitive, read on...
  // On receive, the previously configured ACK/NACK setting is transmitted in
  // response to the received byte before the interrupt is signalled. 
  // Therefor we must actually set NACK when the _next_ to last byte is
  // received, causing that NACK to be sent in response to receiving the last
  // expected byte of data.
#22

i’m able to fix L 5 8 : TR.PULSE I by using multiple read buffers:

#define I2C_RX_BUF_COUNT 4
uint8_t rx_buffer_index = 0;

uint8_t rx_buffer[I2C_RX_BUF_COUNT][I2C_RX_BUF_SIZE];
static uint8_t rx_pos[I2C_RX_BUF_COUNT];

void twi_slave_rx( U8 u8_value )
{
  if (rx_pos[rx_buffer_index] < I2C_RX_BUF_SIZE) {
    rx_buffer[rx_buffer_index][rx_pos[rx_buffer_index]] = u8_value;
    rx_pos[rx_buffer_index]++;
  }
}

void twi_slave_stop( void )
{
  uint8_t counter = rx_buffer_index;
  if (++rx_buffer_index >= I2C_RX_BUF_COUNT) rx_buffer_index = 0;
  
  process_ii(rx_buffer[counter], rx_pos[counter]);
  rx_pos[counter] = 0;

this proves skipped commands are likely due to the fact that the buffer is overwritten while process_ii is executing. the solution can either be the above or having process_ii make a copy of the buffer data before it starts processing.

still getting freezing with CV 6 CV 5 in M script…

one more thing in the code above - added a check for rx_pos not to go over I2C_RX_BUF_SIZE

2 Likes
#23

i haven’t gotten to that yet.

but as another data point, i can’t get non-metro command to lock up with a non-existent i2c address, ie calling II WW.POS 2 without a WW attached.

@bpcmusic it’d be crazy helpful if you could scope/analyze your extender to see if you get an ACK after a TT read request.

1 Like
#24

well, what. ok. ah.

good find, i was thinking this but couldn’t rationalize it being an issue-- so what this suggests is the I2C interrupt (for rx) is jumping in and plowing the data before the previous interrupt’s process function has finished, augh.

i’ll stress-test with this fix and then jump into the metro issues once it feels solid (i’m keeping the metro issue separate for now)

3 Likes
#25

It’s normal for a master receiver to indicate the end of reception by sending a NACK. It needs to do this so it can take control of the bus and issue the STOP condition. If it sent an ACK instead, the state machine of the transmitter slave may attempt to continue to drive the bus.

3 Likes
#26

makes sense, didn’t see it mentioned in any of the i2c materials i read (admittedly didn’t go deep enough) so wondered if it was ASF interpretation of the protocol.

1 Like