Addressing: (cat_id, cmd_id)
Every application message carries a one-byte category and a one-byte command.
This tuple is used on both the BLE link and the ESP-NOW mesh — but the same
(cat_id, cmd_id) maps to a different wire layout on each transport (see
Two format registries).
[BLE] DataXfer cat_id: {} | cmd_id: {},
[BLE] ConnStatus cat_id: {} | cmd_id: {} | len: {},
A cmd_id must be provided for demi-god messages.
Two format registries — do not conflate
Proof they differ for the same key: BLE
gen_live_data (0x03, 0x01) packs
<bfi3fb4Bi3b2hbiffb3ibHBBb (69 B), while TOTEM_MSG_MAP (3,1) = <BBHH4b4BHHBb
(20 B). Same key, different bytes. (Confirmed)
Categories
Confirmed BLE(cat_id, cmd_id) pairs and their labels, taken directly from the
send_data_v2 transmit logs:
The receive path dispatches app→device commands on
cat_id. Writes to …-0002 go to
recv_data_msgs; writes to …-0001 go to recv_status_msgs. The complete map, decoded from
both handlers (Confirmed; identical in v5.0.2 and v5.0.3 except where marked):
Data characteristic (…-0002)
Conn-status characteristic (
…-0001)
The legacy transmit loop logs its sends per category (
Sending static data,
Sending BLE | DataTransfer (0x06, 0x07), …), and the half-duplex loop logs a confirmation per
record: BLE ACK Static Data, BLE ACK Live Data, BLE ACK Peer Ping, BLE ACK Peer Sync,
BLE ACK Comms Handoff.
BLE message formats (gen_*) — what a client reads
Each gen_* builder writes buff[0:2] = (cat_id, cmd_id) then packs the payload from
live device state; the LOAD_ATTR names are the field semantics. These are the formats a
BLE client actually reads (Confirmed from the struct.pack_into opcode stream).
Live Data — (0x03, 0x01) gen_live_data
struct <bfi3fb4Bi3b2hbiffb3ibHBBb packed at offset 2 (calcsize = 69). 30 values,
in this exact order:
Flags byte #28 =
pack_flags(is_sos, is_eco_mode, led_brt≥GLOBAL_BRT, gnss_location_set, power_level==2, is_charging, 0, is_mag_cal_needed) in that bit order. Bit 2 is set at normal brightness
(GLOBAL_BRT = 0.6; eco dims to 0.1). Bit 4 means low battery (change_power_level logs
Changing battery health to: {}). Byte #8 power_bits encodes config.power_mode in bits[0:3].
The reserved slots are constants in every published firmware. 3.2.12, 4.1.3, 5.0.2 and
5.0.3 all pass the literals above to struct.pack_into. The official app (2.3.0,
useLiveDataParser) reads each one at the offset shown and discards it. It also runs offset 69
through unpackFlags and uses no bit. Neither side gives them a name. gen_live_data still
computes len(config.peers), config.closest_peer, config.furthest_peer, the ms since
config.last_peer_msg and gc.mem_free() and never packs them, which suggests these slots once
carried such statistics. (The app labels flag bits 2–4 isDimLeds, isGnssLock and
isLowBatt; the firmware sets bit 2 at full brightness.)
Static Data — (0x01, 0x02) gen_static_data
buff[2] = total_len & 255; buff[3:9] = MAC (6 bytes); then
struct <biHBBBbBBBbhhiiibbb at offset 9 (calcsize = 34); then three
UTF-8 strings concatenated from offset 43 (no per-string prefix; the lengths are fields
17–19). 19 packed values:
Then, from offset 43, the three strings back to back:
device_name, git branch
('N/A' if none), wifi_ssid ('').
Field 9 is a capability byte. 4.1.3, which has no send_data_v2, packs 0; 5.x packs 1. The
official app (2.3.0, useStaticDataParser) reads bit 0 as isHalfDuplex and switches to the TX
handoff when it is set. The app reads fields 1 and 11–16 and discards them; every published
firmware packs them as 0.
Peer Ping — (0x06, 0x02) gen_peer_ping
buff[0:2] = (0x06, 0x02); buff[3:9] = peer MAC (hex_to_bin, 6 bytes); buff[9] =
peer.mesh_hops clamped to 0..255. A 40-byte record struct <ffbbh4BHbb4BiihBbf
(calcsize = 40) is then packed at offset 10, followed by the UTF-8 peer name (its byte
length is field #11 below, name bytes from offset 50) and a trailing struct <bH =
(batt_pct, release_id). buff[2] holds the total length, written last. This is a full
~53 + name-length-byte peer record, not just MAC + two flag bytes — the two peer-flag
bytes are fields 6 and 20 inside this struct. (Confirmed from the struct.pack_into
opcode stream; field labels are the LOAD_ATTR names, so semantics are Confirmed where a
name is given.)
Log:
Sending BLE | DataTransfer (0x06, 0x02) for {}.
Peer Sync — (0x06, 0x07) gen_peer_sync
A peer-MAC list: [0x06, 0x07, total_len & 255, peer_count, mac0(6), mac1(6), …]
(6-byte MACs, first at offset 4, +6 each).
ESP-NOW registry: TOTEM_MSG_MAP
TOTEM_MSG_MAP in espnow_conn_v2.py is a (cat_id, cmd_id)-keyed map (2-byte bytes
key) whose values are the ESP-NOW / mesh payload struct formats. Recovered in full
with the corrected >>3 decode (Confirmed):
EXTENDED holds the longer variants, (cat, cmd) → {frame length → fmt}:
The keys are total frame lengths including the SyncWord, not schema ids:
Parser.aread
picks EXTENDED[key][len(frame)] on an exact match. Failing that, if the frame’s cmd_id
is 0 and it is at least 72 bytes long, the 72 entry is used; otherwise aread logs
Non-extended payload being used and falls back to TOTEM_MSG_MAP. compass_mesh only
upgrades a (2,0) frame to the 45 variant on an exact length match. Commands 1 and 2 of
category 0 have no EXTENDED entry, so a receiver decodes only the first 23 bytes of a bond
frame. The struct is unpacked from frame[2:], so its leading BB is the echoed
(cat_id, cmd_id) at frame offsets 2-3.
The
cmd_id == 0 fallback is keyed on the command byte alone, so in principle it also covers
(2,0), whose EXTENDED dict has no 72 entry. In practice no (2,0) frame reaches aread:
EspConn.recv handles category 2 command 0 inline and continues. Confirmed
(espnow_conn_v2.dis:7515-7540 and :6698-6790).calcsize(fmt) + 2 — except EXTENDED[(1,6)][5], whose format
<BBbbB also has calcsize 5 and so would need a 7-byte frame. Nothing emits either length;
see demi-god.
The category 1 (demi-god) map entries describe only the fixed head of the frames the
firmware actually sends. Each espnow_msg.demigod_gen_* packer extends its buffer past the
mapped fields: (1,2) builds 30 bytes for a 26-byte entry, (1,7) builds 12 for 7, and
(1,0) builds 16 plus four variable-length UTF-8 strings plus an optional trailing byte.
struct.unpack ignores the surplus. Field tables are on the
demi-god page.
The field tables below give frame offsets (SyncWord at 0), as the firmware’s pack_into
calls use them. All are Confirmed from the builders in espnow_conn_v2.Messages and the
readers in Parser, and the peer and locate layouts are exercised on hardware by the
ESP32 emulator, whose Go codec (mesh package) matches frames
packed by CPython’s struct with these formats byte for byte.
Peer frame (0, cmd)
ENOW_PEER_BUFF = bytearray(108) is a module-level buffer in project_data; Messages
holds a memoryview of it and always sends all 108 bytes. Command 0 is the status, 1 a
bond request or confirmation, 2 an unbond notice.
Four methods write it, at different times — a caller that only calls gen_peer_msg sends
whatever the others last left behind:
The name.
update_device_data writes it with
pack_utf8_str(buff=peer_msg, start=71, text=get_device_name()), which is a bare
buff[71:71+n] = data — its max_size argument is accepted and never used. The buffer is a
memoryview of 108 bytes, so a name of 33-37 bytes fits the slice but overwrites the field
region after it, and gen_peer_msg’s struct.pack_into('<bbHb', …, 71+n) then raises; a
name of 38 bytes or more makes the slice assignment itself raise. Either way nothing goes
out. 32 bytes (108 − 71 − 5) is the largest name that works. The buffer is reused, so bytes
after the tail can hold leftovers of a longer earlier name. Confirmed
(f_lib_bitwise.dis, pack_utf8_str; espnow_conn_v2.dis, update_device_data).
The battery voltage is a struct 'e' half float. MicroPython’s software
mp_encode_half_float (py/binary.c, used on ESP32 because the port has no native
_Float16) rounds half up, lets a mantissa carry spill into the exponent field, flushes
values that should become the largest subnormals to zero, and — the part that matters —
has no range check. Past the half’s ~65504 limit the exponent runs off the end of its
five bits and into the sign bit, so a voltage of 131072 encodes as −0 and 100000 as a NaN.
The Go codec in mesh/half.go is a port of both directions and matches a Totem byte for
byte; its encoder refuses a finite magnitude above 65504 rather than silently changing it.
The encoder behaviour is Inferred — it is MicroPython runtime code, not in the frozen
bytecode — but the field being 'e' at offset 43 is Confirmed from
'<4BhHehhffbB' at espnow_conn_v2.dis:1647.
Locate (2,0)
gen_mesh_msg fills ENOW_MESH_BUFF = bytearray(45): the 45-byte EXTENDED variant.
Smart Group beacon (7,0)
gen_auto_bond_grp builds 21 + 16 × n bytes: <ffbbbHbbH at offset 4, then one 16-byte
record per member.
The member MAC is written as a raw 6-byte slice assignment from the key of the host’s
modes.auto_bond_grp dict, and the four packed values are that entry’s items 0-3 — so a
member’s position is whatever the host last heard, not a live value. Offsets 12/13 (the
host’s own accuracy and colour) come from the host’s GNSS and the clr_id argument; the
host sends clr_id = 0 while advertising and config.color_id on finalize. Confirmed
(espnow_conn_v2.dis:1296-1447, gen_auto_bond_grp).
Smart Group reply (7,1)
16 bytes built in Parser._smart_group: <Hbffb at offset 4 = group UID, join flag (+1 join,
−1 leave), latitude, longitude, position accuracy.
ESP-NOW frame envelope
On the radio these payloads sit inside an ESP-NOW application frame with a fixed SyncWord prefix and no CRC (Confirmed):0xA7 0x74 match + len ≥ 4 + per-(cat, cmd)
payload-size check. There is no checksum or CRC on the ESP-NOW frame. (The rodata
string Invalid Checksum value for: {} belongs to the u-blox UBX GNSS parser, not this
path.) A mesh dedup UID (uint16) sits at mesh-frame offset 20.
Message identity & lifetime
Only the locate frame(2,0) floods the mesh, so only it carries an identity. Peer frames
have no UID or sequence number; a receiver keys them on the ESP-NOW source MAC and the receive
time.
gen_msg_uid, MSG_IN_BUFF and MSG_OUT_BUFF belong to the chat module (chat_msg), which
no module imports in 5.0.3.
This lets a message flood the mesh once and be dropped as a duplicate on re-receipt
(Demi-god command ignored, already received is the same idea in the command path).
Chunking
Files larger than one GATT write are moved by a common chunking layer (f_ble/chunking.py,
used by BLE OTA and, since v5.0.3, by the log uploader). Two headers are involved:
In v5.0.2
f_ble/file_upload.py imported CHUNK_HDR_FMT and CHUNK_HDR_SZ from
chunking, but chunking never defined them, and nothing imported the uploader. Earlier
versions of this page inferred CHUNK_HDR_FMT = '<HBBBiHiB' from that. v5.0.3 defines
both names explicitly, and they describe the 12-byte chunk header. <HBBBiHiB is the
transfer header packed inline by gen_transfer_buff.Transfer header (gen_transfer_buff)
buff[0] = 2, buff[1] = 2, then struct.pack_into('<HBBBiHiB', buff, 2, …)
(Confirmed, both versions):
After the 16-byte header:
buff[18:50] = 32-byte SHA-256 (when set); buff[50] =
name_len (v5.0.3 writes 0 when there is no name), then the UTF-8 file name, then a
trailing err_no byte.
Chunk header (v5.0.3, FileUploader._stream)
[0x00, 0x02] + struct.pack_into('<HHiH', buff, 2, file_id, n, byte_pos, chunk_no) + n
data bytes, written to …-0003 with send_update=True. n is the bytes read for this chunk
and byte_pos its offset (Confirmed from the operands); chunk_no counts from 1
(Inferred from the counter variable). The payload per chunk is
min(ble.mtu_payload(), 247) - CHUNK_HDR_SZ (20 − 12 before the MTU is known). v5.0.3
tracks the negotiated MTU in BleLite.mtu for this purpose.
Header enums (Confirmed)
file_id is derived from the file’s SHA-256: file_id = sha256[0] | sha256[1] << 8
(u16). The transfer destination enum is separate: SAVE_TO_VFS = 1, SAVE_TO_OTA = 2, and
v5.0.3 adds UPLOAD_TO_APP = 3.
File transfer
f_ble/file_upload.py (FileUploader) pushes device log files (events-* files) to the
app so the app can forward them to the cloud. It is new in practice in v5.0.3: v5.0.2
shipped the module but never imported it. In v5.0.3 ble_manager starts it as the
ble_file_upload task, and it only runs when every gate is open:
Uploads mark themselves as the half-duplex non-critical owner (
noncrit_owner = 'upload'),
which may delay the next TX handoff by up to 2 s. After MAX_FILE_RETRIES failures a file is
suspended until BLE is next turned on. v5.0.3 also toggles nav-log fast rotation depending on
whether a backlog exists.
App→device reply header (FileUploader.on_header)
The app replies with an 18-byte header, struct <BBHbBBiHiB (Confirmed, guarded by
len(data) >= 18), starting (0x02, 0x03). In v5.0.3 it is written to …-0001:
recv_status_msgs routes (2,3) to uploader.on_header. Used fields: file_id = t[2],
status = t[3], action = t[4], chunk = t[7].
Log: [Upload] App header | status: {} | action: {} | chunk: {} | err: {}.
Control flow
Result codes (Confirmed)
RES_DONE = 1, RES_NO_APP = 2, RES_RETRY = 3, RES_FAILED = 4, RES_CANCEL = 5,
RES_ABORT = 6 (logged as result: {} (1=done 2=no app 3=retry 4=failed 5=cancel 6=abort)).