This document is primarily intended for developers only.
A major change is the support of variable length frames between host and Proxmark3.
This is a step especially important for usage over FPC/USART/BT.
## Old format
Previously, frames were, in both directions like this:
uint64_t cmd;
uint64_t arg[3];
union {
uint8_t asBytes[PM3_CMD_DATA_SIZE];
uint32_t asDwords[PM3_CMD_DATA_SIZE / 4];
} d;
with PM3_CMD_DATA_SIZE = 512 and there was no API abstraction, everybody was forging/parsing these frames.
So the frame size was fixed, 544 bytes, even for simple ACKs.
When snooping the USB transfers, we can observe the host is sending 544b Bulk USB frames while the Proxmark3 is limited by its internal buffers and is sending 128b, 128b, 128b, 128b, 32b, so in total 5 packets.
## New format
Even if we make the payload part variable in the old format, we've still a minimum of 32 bytes per frame with fields arbitrarily large.
So we designed a new format from scratch:
For commands being sent to the Proxmark3:
uint32_t magic;
uint16_t length : 15;
bool ng : 1;
uint16_t cmd;
uint8_t data[length];
uint16_t crc;
*`magic`: arbitrary magic (`PM3a`) to help re-sync if needed
*`length`: length of the variable payload, 0 if none, max 512 (PM3_CMD_DATA_SIZE) for now.
*`ng`: flag to tell if the data is following the new format (ng) or the old one, see transition notes below
*`cmd`: as previously, on 16b as it's enough
*`data`: variable length payload
*`crc`: either an actual CRC (crc14a) or a Magic placeholder (`a3`)
For responses from the Proxmark:
uint32_t magic;
uint16_t length : 15;
bool ng : 1;
int16_t status;
uint16_t cmd;
uint8_t data[length];
uint16_t crc;
*`magic`: arbitrary magic (`PM3b`) to help re-sync if needed
*`length`: length of the variable payload, 0 if none, max 512 (PM3_CMD_DATA_SIZE) for now.
*`ng`: flag to tell if the data is following the new format (ng) or the old one, see transition notes below
*`status`: a field to send back the status of the command execution
*`cmd`: as previously, on 16b as it's enough
*`data`: variable length payload
*`crc`: either an actual CRC (crc14a) or a Magic placeholder (`a3`)
We used to send an anonymous ACK, now we're replying with the corresponding command name and a status.
CRC is optional and on reception, the magic `a3` is accepted as placeholder. If it's different then it's checked as a CRC.
By default CRC is user over USART and is disabled over USB, on both directions.
Internal structures used to handle these packets are:
* PacketCommandNGPreamble
* PacketCommandNGPostamble
* PacketCommandNGRaw
* PacketResponseNGPreamble
* PacketResponseNGPostamble
* PacketResponseNGRaw
But they are abstracted from the developer view with a new API. See below.
## Transition
Because it's a long transition to clean all the code from the old format and because we don't want to break stuffs when flashing the bootloader, the old frames are still supported together with the new frames. The old structure is now called `PacketCommandOLD` and `PacketResponseOLD` and it's also abstracted from the developer view with the new API.
## New format API
So the new API is a merge of the old and the new frame formats, to ensure a smooth transition.
The boolean `ng` indicates if the structure is storing data from the old or the new format.
Old format can come from either old 544b frames or mixed frames (variable length but still with oldargs).
After the full transition, we might remove the fields `oldarg` and `ng`.
`PacketCommandNG` and `PacketResponseNG` are the structs used by the API, as seen in a previous section, there are other variable-sized packed structs specifically for the data transmission.
So cmds should make the transition from `SendCommandOLD` to `SendCommandNG` to benefit from smaller frames (and armsrc handlers adjusted accordingly of course).
`SendCommandBL` is for Bootloader-related activities, see Bootrom section.
`SendCommandMIX` is a transition fct: it uses the same API as `SendCommandOLD` but benefits somehow from variable length frames. It occupies at least 24b of data for the oldargs and real data is therefore limited to PM3_CMD_DATA_SIZE - 24 (defined as PM3_CMD_DATA_SIZE_MIX). Besides the size limitation, the receiver handler doesn't know if this was an OLD frame or a MIX frame, it gets its oldargs and data as usual.
Warning : it makes sense to move from `SendCommandOLD` to `SendCommandMIX` only for *commands with small payloads*.
* otherwise both have about the same size
*`SendCommandMIX` has a smaller payload (PM3_CMD_DATA_SIZE_MIX <PM3_CMD_DATA_SIZE)soit'sriskytoblindlymovefromOLDtoMIXifthereisalargepayload.
Internally these functions prepare the new or old frames and call `uart_communication` which calls `uart_send`.
### On the Proxmark3, for receiving frames:
(`armsrc/appmain.c`)
PacketCommandNG
`AppMain` calls `receive_ng`(`common/cmd.c`) which calls `usb_read_ng`/`usart_read_ng` to get a `PacketCommandNG`, then passes it to `PacketReceived`.
(no matter if it's an old frame or a new frame, check `PacketCommandNG.ng` field to know if there are `oldargs`)
`PacketReceive` is the commands broker.
Old handlers will still find their stuff in `PacketCommandNG.oldarg` field.
So replies should make the transition from `reply_old` to `reply_ng` to benefit from smaller frames (and client reception adjusted accordingly of course).
`reply_mix` is a transition fct: it uses the same API as reply_old but benefits somehow from variable length frames. It occupies at least 24b of data for the oldargs and real data is therefore limited to PM3_CMD_DATA_SIZE - 24. Besides the size limitation, the client command doesn't know if this was an OLD frame or a MIX frame, it gets its oldargs and data as usual.
Example of a handler that supports both OLD/MIX and NG command styles and replies with the new frame format when it receives new command format:
`uart_communication` calls `uart_receive` and create a `PacketResponseNG`, then passes it to `PacketResponseReceived`.
`PacketResponseReceived` treats it immediately (prints) or stores it with `storeReply`.
Commands do `WaitForResponseTimeoutW` (or `dl_it`) which uses `getReply` to fetch responses.
## API transition
In short, to move from one format to the other, we need for each command:
* (client TX) `SendCommandOLD` ⇒ `SendCommandNG` (with all stuff in ad-hoc PACKED structs in `data` field)
* (pm3 RX) `PacketCommandNG` parsing, from `oldarg` to only the `data` field
* (pm3 TX) `reply_old` ⇒ `reply_ng` (with all stuff in ad-hoc PACKED structs in `data` field)
* (client RX) `PacketResponseNG` parsing, from `oldarg` to only the `data` field
Meanwhile, a fast transition to MIX frames can be done with:
* (client TX) `SendCommandOLD` ⇒ `SendCommandMIX` (but check the limited data size PM3_CMD_DATA_SIZE ⇒ PM3_CMD_DATA_SIZE_MIX)
* (pm3 TX) `reply_old` ⇒ `reply_mix` (but check the limited data size PM3_CMD_DATA_SIZE ⇒ PM3_CMD_DATA_SIZE_MIX)
## Bootrom
Bootrom code will still use the old frame format to remain compatible with other repos supporting the old format and because it would hardly gain anything from the new format:
Sending multiple commands can still be slow because it waits regularly for incoming RX frames and the timings are quite conservative because of BT (see struct timeval timeout in uart_posix.c, now at 200ms). When one knows there is no response to wait before the next command, he can use the same trick as in the flasher:
// fast push mode
conn.block_after_ACK = true;
some loop {
if (sending_last_command)
// Disable fast mode
conn.block_after_ACK = false;
SendCommandOLD / SendCommandMix
if (!WaitForResponseTimeout(CMD_ACK, &resp, some_timeout)) {
....
conn.block_after_ACK = false;
return PM3_ETIMEOUT;
}
}
return PM3_SUCCESS;
Or if it's too complex to determine when we're sending the last command:
// fast push mode
conn.block_after_ACK = true;
some loop {
SendCommandOLD / SendCommandMIX
if (!WaitForResponseTimeout(CMD_ACK, &resp, some_timeout)) {
....
conn.block_after_ACK = false;
return PM3_ETIMEOUT;
}
}
// Disable fast mode and send a dummy command to make it effective
conn.block_after_ACK = false;
SendCommandNG(CMD_PING, NULL, 0);
WaitForResponseTimeout(CMD_ACK, NULL, 1000);
return PM3_SUCCESS;
## Reference frames
For helping debug...
* OLD command and reply packets are 544 bytes.
* NG & MIX command packets are between 10 and 522 bytes.
* NG & MIX reply packets are between 12 and 524 bytes.
On linux USB
* sent packets can be 544
* received packets are max 128, so 544 = 128+128+128+128+32
