MIDI SysEx 7bit decoding help (bits and bytes)!

General
1

Hi,

I’m trying something unusual, the Retrokits RK006 is a MIDI hub device, and has its own SysEx to program the settings. I’m trying to figure it out but I ran into some problems.

Here’s how messages are constructed:

GENERAL RK006 SYSEX FORMAT:

F0 00 21 23 00 06 <CMD/RSP> <args...> F7

  • each command will be acknowledged with a response from the 005.
  • are 7-bit packed: 7 bytes of 7-bit data is prequelled by a byte
    containing the MSBs

I used the available browser-based manager to get a SysEx example of how the commands are set up but I came across the following command:

F0 00 21 23 00 06 03 02 08 7B F7

which shouldn’t be there according to the documentation. I mean the pdf says that you have set parameter requests which are supposed to be set up like this

F0 00 21 23 00 06 03 00 <paramnr> <paramval> F7

but I did not expect to see anything with … 03 02 so I was a bit confused by that part, and I reached out to Retrokits to ask why there was 02 there. Here’s what I got back from them:

“The 02 is accumulating bits from the next messages. SysEx can only hold 7 bits. The 8th bit is stored in a preceding byte so you have one 7bit value which has all 8th bites from the next 7 Sysex value bytes”

I’m getting confused. Can anyone help clarify what the above means?

Happy to share more info on my decoding of the messages so far if that helps.

1 Like
2

The data between the F0 and the F7 must all have the high bit set to 0. This is MIDI.

Since most devices (but not all) want to transmit data to/from memory that uses all 8 bits, there has to be some way to encode the 8-bit data into 7-bit data.

One way is to take all the high bits of the next 7 bytes and collect them into a single byte.

So data like this:

byte 0: [a7][a6][a5][a4][a3][a2][a1][a0]
byte 1: [b7][b6][b5][b4][b3][b2][b1][b0]
byte 2: [c7][c6][c5][c4][c3][c2][c1][c0]
byte 3: [d7][d6][d5][d4][d3][d2][d1][d0]
byte 4: [e7][e6][e5][e4][e3][e2][e1][e0]
byte 5: [f7][f6][f5][f4][f3][f2][f1][f0]
byte 6: [g7][g6][g5][g4][g3][g2][g1][g0]

Gets transmitted as:

byte 0: [0][g7][f7][e7][d7][c7][b7][a7]
byte 1: [0][a6][a5][a4][a3][a2][a1][a0]
byte 2: [0][b6][b5][b4][b3][b2][b1][b0]
byte 3: [0][c6][c5][c4][c3][c2][c1][c0]
byte 4: [0][d6][d5][d4][d3][d2][d1][d0]
byte 5: [0][e6][e5][e4][e3][e2][e1][e0]
byte 6: [0][f6][f5][f4][f3][f2][f1][f0]
byte 7: [0][g6][g5][g4][g3][g2][g1][g0]

So every 7 bytes becomes 8 when transmitted. It is up to you to do the “unshuffle” after getting the data.

Sadly, there is no standard here, so different devices sometimes have variants of this scheme: Is the byte of “extra bits” at the start or the end of the block? Are the “extra bits” in the order I showed, or reversed? What happens when there are fewer than 7 bytes of data to transmit?

From the looks of your one example packet, I’d say that Retrokits is using the scheme as I showed, and if there are fewer than 7 bytes in a message, they are just omitted and the bits in the byte of “extra bits” set to zero.

3 Likes
3

Thank you so so so much for helping me here, your example does help somewhat, as I now get what you mean at least at some level. It seems that the bytes still consist of 8 bits. That’s a given, right?

I’m still rather confused, I’m afraid.

Here’s a concrete example that might help as a good basis to clear things up:

The manual states

GENERAL RK006 SYSEX FORMAT:

F0 00 21 23 00 06 <CMD/RSP> <args...> F7

and

4 = BOOT_INT_TEMPO ; tempo of internal clock generator at boot
;(in bpm: 0=off, 120=120.0bpm)

So when I run into the following line I’m sure this is the BOOT_INT_TEMPO parameter

F0 00 21 23 00 06 03 00 04 00 F7

And since HEX(00) is 0 that means it’s off. But if I set it to 207bpm here’s the line I get:

F0 00 21 23 00 06 03 02 04 4F F7

This is supposed to be 207 bpm, but 4F is 79 so the MSB is to be found in the 02 in some manner. But how? And how do I decipher that?

EDIT

Also, sorry @mzero I just noticed that typing < args > gets considered as an argument here so it disappeared above and it might have misled you somewhat. This is what the manual actually states for the 7-bit packing

F0 00 21 23 00 06 <CMD/RSP> <args...> F7

  • < args > are 7-bit packed: 7 bytes of 7-bit data is prequelled by a byte containing the MSBs

Which means that only the < args > part works like you described, or would there be differences in this case?

Also, what would, for example, [a0] contain? A hex of two digits?

4

I actually think I got what’s going on.

That 02 is the 1 missing from the 8bit number at the beggining!

If I wanted an 8bit representation of 207 bpm I would’ve need to use:

1 1 0 0 1 1 1 1

instead I got

(0) 1 0 0 1 1 1 1

which is 79 bpm

and that 1 missing from 207 became

0 0 0 0 0 0 1 0

which is the 02 earlier in the message.

Correct?

5

Yes, only the args part is encoded.

In my diagram all the [xx] things are individual bits.

So looking at F0 00 21 23 00 06 03 02 04 4F F7:

  • F0 00 21 23 00 06 is the sysex header
  • 03 is the command
  • 02 04 4F are the args
  • F7 is the sysex trailer

Now, the args are transmitted as three bytes:

byte 0: 0 0 0 0  0 0 1 0
byte 1: 0 0 0 0  0 1 0 0
byte 2: 0 1 0 0  1 1 1 1

The high bits are all zeros in this transmitted block, as they must be. So to extract the original data, we must “fill in” the missing high bits. We’ll take them from byte 0:

  • The first missing high bit, for byte 1, will come from the lowest bit in byte 0: 0
  • The second missing high bit, for byte 2, will come from the next lowest bit in byte 0: 1
  • There are no more bytes in the transmitted args, so we’re done.

The resulting decoded args is:

byte 1: 0 0 0 0   0 1 0 0
byte 2: 1 1 0 0   1 1 1 1

This is 04 CF in hex, and CF is indeed 207.

2 Likes
6

Perfect! My explanation did not contain the byte 1 part.

Thank you so much @mzero it’s so much clearer now!

One thing that got me confused is that in the manual it states that

SETPARAM_REQ
F0 00 21 23 00 06 03 00 F7

so I always thought that the 02 is part coming right after the 03 was part of the actual command, and not part of the arguments.

I’ll be returning to your explanation again and again as I try to create a custom setup for myself.

Again, thank you so much!

7

This was such a great explanation… I’ve just started trying to decipher some sysex files from some gear I use and was struggling with the whole 8bit to 7bit and back again process. This gave me that “aha” moment I needed!

1 Like