Methodology & next steps

How this firmware was analyzed, the tools used, and how to close the remaining gaps.

Approach

The firmware findings come from static analysis of public artifacts: the v5.0.2 and v5.0.3 firmware images and the companion Android app. No network service was attacked; the releases API and S3 objects are served openly. The BLE protocol was additionally checked end to end with the totemctl client (cmd/totemctl in this repository) against the author’s own device on firmware 4.1.3 and 5.0.3. That test found the transmit-mode behaviour on macOS, which static analysis alone had missed. The ESP-NOW mesh was checked the same way with the ESP32 emulator (cmd/totememu): it bonded with the author’s Totem on 5.0.3 and exchanged peer status with it, which confirmed the radio settings, the peer frame layout and the pairing handshake.

Tools

Tool	Use
esptool	parse the ESP-IDF image header, segments, and app descriptor
Python + struct	parse headers, carve segments, read the esp_app_desc_t
capstone 6.0 (CS_ARCH_XTENSA)	disassemble the native Xtensa code
rizin / rabin2	validate the rebuilt ELF
strings + grep	recover the qstr symbol table and string constants
hermes-dec	decompile the Android app’s Hermes bytecode (for the download URL)

Steps

1. Parse the image

   Confirm ESP32, 7 segments, entry point, and read the app descriptor (IDF v5.4, build date, ELF hash).

2. Rebuild an ELF

   Map the 7 segments to their load addresses with correct R/W/X perms in an EM_XTENSA ELF for Ghidra/IDA/rizin.

3. Identify the runtime

   Native strings reveal MicroPython v1.25.0 — the app is Python, frozen as bytecode.

4. Recover the symbol table

   Extract the qstr pools and rodata strings: the application’s method, attribute and class names and every log line. The first v5.0.2 pass counted 2,479 identifiers and 11,857 strings by string scanning. Walking the pools directly gives 4,086 qstrs for v5.0.2 (228 core + 972 const + 2,886 frozen) and 4,162 for v5.0.3 (228 + 975 + 2,959).

5. Reconstruct behavior

   Map modules, classes, protocol categories, and subsystem behavior from those symbols and the {}-style (Python str.format) log messages.

Coverage audit

Symbol recovery tells you what exists. It does not tell you what anyone has actually read, and the two are easy to confuse — a module can be named, categorised and given a confident one-line role on the strength of nothing but its filename.

Every module was therefore audited against re/v5.0.3/mpy/*.dis and assigned one of four ratings, published per module on the module reference:

Rating	Bar
DECODED	Behaviour documented from the bytecode: named functions, constant values, control flow
PARTIAL	Some verified internals documented, main logic not
NAME-ONLY	Role is a hypothesis from the module name and a few strings; never read
STOCK	Upstream MicroPython or micropython-lib; out of scope

How the ratings were derived

1. Import graph. Every IMPORT_NAME opcode in all 94 disassemblies was extracted and resolved against the module list, giving each module’s importers. This is what identified apa106.py and neopixel.py as dead code, chat_msg.py, f_ota/hotspot.py and peer_auto_bond.py as unreferenced, and f_lib/logger.py (41 importers) and f_lib/task_mgr.py (28) as the load-bearing modules.
2. Distinctive-symbol coverage. For each module, the identifiers in its qstr_table that appear in at most two modules across the image — its distinctive symbols, which excludes shared names like config or Event — were checked against the full text of docs/. The fraction of them that appear is a mechanical proxy for decode depth. When the audit ran, f_lib/logger.py, f_lib/bitwise.py and f_lib/async_helpers.py each scored zero; all three have since been read and are written up on f_lib.
3. Manual adjudication. The proxy is unreliable for small modules (a module with one distinctive symbol scores 0% or 100%), so every rating was set by reading what the docs actually claim about that module and checking it against the disassembly. The proxy ranked the candidates; it did not decide them.

A “documented” module is not necessarily a correct one. This audit found several confident claims that the bytecode contradicts: apa106.py presented as the LED driver (it is unreachable; leds.py imports f_lib/neopixel_v2.py), a deep-sleep power state (the firmware never calls machine.deepsleep()), f_lib/rtc_v2.py described as RTC-memory storage (it is a machine.RTC subclass doing calendar maths), and BleCtrl listed as a class (it is a log-line prefix; the class is BleController).

Each of those came from reading a name or a log string and not the code. That is exactly the failure mode the NAME-ONLY rating exists to flag.

Why not a full source decompile?

The application is frozen MicroPython bytecode. Unlike CPython (which has mature decompilers such as uncompyle6/decompyle3), MicroPython has no mature bytecode-to-source decompiler. The best available is bytecode disassembly (opcodes plus qstr references), not clean Python source.

However, the frozen format keeps every identifier as an interned string (qstr) and retains all string literals. That symbol table plus the log messages reconstruct each module’s behavior at close to source fidelity — which is how this documentation was built.

Closing the remaining gaps

Three items were graded “needs bytecode disassembly” in the protocol completeness matrix:

1. The complete per-category cmd_id enumeration (TOTEM_MSG_MAP).
2. The exact binding of each struct format to a specific message.
3. The field order of CHUNK_HDR_FMT.

The frozen-bytecode disassembly has now been carried out and all three are recovered. The values below are literals present in the binary — every struct-format string appears verbatim in the DROM rodata — and the method that produced them is recorded afterward.

The /protocols/overview completeness matrix now marks these three as Recovered and reproduces the full decoded map; this page records how they were obtained.

TOTEM_MSG_MAP — confirmed

Defined in espnow_conn_v2.py, built by a BUILD_MAP 13 / STORE_NAME TOTEM_MSG_MAP opcode sequence (fully decoded). The map is flat, keyed by a 2-byte bytes value (cat_id, cmd_id) (byte 0 = cat_id, byte 1 = cmd_id); every value is a payload struct-format string.

An earlier pass decoded the map’s qstr-immediate values with >>2 and reported “handler/type names” (disconn_animation, dev_total_lightsleep_ms, dev_info, device_power, disabled). Those were decode artifacts and are wrong. On ESP32 MicroPython a qstr-immediate is tagged by (o & 7) == 2 (REPR_A), and the qstr number is o >> 3, not o >> 2. Under the correct o >> 3 decode every entry resolves to a struct format string — the values below.

(cat_id, cmd_id)	struct format	Size
(0, 0) / (0, 1) / (0, 2)	<BBffbbhbbb6B	23 B
(1, 0)	<BBB7bHbb	14 B
(1, 2)	<BBiffHii	24 B
(1, 5)	<BB9B8bhh4B	27 B
(1, 6)	<BBbbB	5 B
(1, 7)	<3Bbb	5 B
(2, 0)	<BB6BffbbHbbbhhBBi	33 B
(3, 1) / (3, 2)	<BBHH4b4BHHBb	20 B
(7, 0)	<BBffbbbHbbH	19 B
(7, 1)	<BBHbffb	14 B

struct format → message bindings — confirmed (for listed)

Each payload format begins with BB = the echoed cat_id/cmd_id header:

(cat_id, cmd_id)	struct format	Size
(1, 0)	<BBB7bHbb	14 B
(1, 2)	<BBiffHii	24 B
(1, 6)	<BBbbB	5 B
(1, 7)	<3Bbb	5 B
(7, 1)	<BBHbffb	14 B

Transfer header — format & field order confirmed

CHUNK_HDR_FMT = <HBBBiHiB, CHUNK_HDR_SZ = 16 bytes — packed by f_ble/chunking.py’s gen_transfer_buff via struct.pack_into at buffer offset 2. The format string, field order, and field names are confirmed from the live pack_into opcode stream: gen_transfer_buff calls struct.pack_into('<HBBBiHiB', buff, 2, self.file_id, self.status_id, self.action_id, self.file_type_id, self.byte_pos, self.chunk_no, self.file_size, 0) (LOAD_ATTR order in the bytecode). The first pass also bound the name CHUNK_HDR_FMT to this format, because f_ble/file_upload.py imports CHUNK_HDR_FMT and CHUNK_HDR_SZ from chunking. That inference was wrong. v5.0.2’s chunking never defines those names, so the import could not have succeeded. v5.0.3 defines them explicitly, as the separate 12-byte chunk header: CHUNK_HDR_FMT = '<HHiH', CHUNK_HDR_SZ = 12, UPLOAD_TO_APP = 3. See chunking. v5.0.3 also turns the last byte below into a flags byte.

struct code	Field	Type
H	file_id	uint16 LE
B	status_id	uint8
B	action_id	uint8
B	file_type_id	uint8
i	byte_pos	int32 LE
H	chunk_no	uint16 LE
i	file_size	int32 LE
B	literal 0 in v5.0.2; pack_flags(is_last_chunk, is_origin_compass) in v5.0.3	uint8

How it was done

1. Locate the frozen data

   In the image (file offsets): 0xf3d3 points into the qstr identifier-string data, but mid-pool — the sorted run of identifier strings starts much earlier (around 0xe000) and flows continuously through 0xf3d3 with no sort reset. The concrete anchors are the frozen qstr pool header at seg0 0x5de0c (prev=0x3f4213d0, total_prev_len=1200, len=2886) with its qstrs[] pointer array at 0x5de24. The frozen module-name registry begins near 0x27b00; the DROM segment maps file 0x18 → vaddr 0x3f400020.

2. Parse the frozen module table

   Reconstruct the MicroPython v1.25 (.mpy version 6.3) mp_raw_code_t tree for the two comms modules that actually carry these constants: espnow_conn_v2.py (home of TOTEM_MSG_MAP and the per-message struct-format registry) and f_ble/chunking.py (home of CHUNK_HDR_FMT).

3. Disassemble the bytecode

   Walk the bytecode with a v1.25-matched reader. Reuse the opcode tables from py/bc0.h / tools/mpy-tool.py, but note mpy-tool.py -d itself only reads standalone .mpy blobs; frozen firmware modules (emitted as proto_fun_* / fun_data_* / const_qstr_table_data_* C structs, not .mpy blobs) require a custom frozen-aware reader. Small integers -16..47 load inline as single-byte opcodes 0x80–0xBF; larger ones via LOAD_CONST_SMALL_INT (0x22, signed var-int) — both embed the value in the bytecode. The TOTEM_MSG_MAP dict build and the CHUNK_HDR_FMT string literal are then readable.

4. Cross-check

   Validate recovered (cat_id, cmd_id) values against the pairs already visible in the logs. (0x01, 0x02) and (0x03, 0x01) validate directly against TOTEM_MSG_MAP. (0x06, 0x07) is a Peer-Sync BLE control frame handled outside TOTEM_MSG_MAP, so cat 6 is absent from the map — its absence is expected, not a failed cross-check.

An alternative dynamic approach — running the exact MicroPython build under an ESP32 emulator or on hardware and dumping TOTEM_MSG_MAP / CHUNK_HDR_FMT from a REPL — would yield the same values faster if a device or matching build is available.

Diffing firmware versions

The v5.0.3 image was processed with the same reconstruction and compared to v5.0.2 function by function:

1. Carve and find anchors

   Carve the 7 segments from the image header. Then locate the structures the reconstructor needs by their shape rather than by fixed addresses:

   • qstr pools: headers {prev, total_prev_len, alloc, len, hashes*, lengths*, qstrs[]}, validated by the NUL terminator at each declared length.
   • Type objects: mp_type_type is the only type object whose base points to itself; str, bytes, tuple, float and int point to it and carry their name qstr at +6.
   • Frozen module table: the longest run of pointers to {qstr_table, obj_table, proto} records whose first qstr ends in .py.

   On v5.0.2 this reproduces every hand-found address. On v5.0.3 it finds the frozen pool at seg0 0x5f59c (base 1203, 2,959 qstrs), the const pool at 0x22208 (975) and the module table at 0x3f4287b0–0x3f428924 (94 modules).

2. Reconstruct and disassemble

   Re-serialize every frozen module as a standalone v6.3 .mpy and disassemble it with MicroPython’s own mpy-tool.py -d: 96/96 modules for v5.0.2 and 94/94 for v5.0.3, with no unknown opcodes, truncated functions or dangling children.

3. Diff

   Split each disassembly into functions, drop the byte-offset column, and compare added, removed and changed functions plus the qstr and constant tables. 20 modules changed, 74 are identical and 2 were removed. See What changed in 5.0.3.

   That page says 21 changed, and both numbers are right: it compares the .mpy files themselves, and a .mpy also carries the source line-number table, which moves whenever a file is recompiled. ble_controller.py is the one module that differs that way and no other — same qstrs, same constant table, same instructions, every line-info entry shifted by exactly +2. Twenty-one modules were rebuilt; twenty had their code changed.

Core-qstr off-by-one in the first v5.0.2 pass. The reconstructor looked up core qstr g at qstrs + 4*(g-1). The core pool’s qstrs[0] is the null qstr, so this read the name of qstr g − 1, the alphabetical predecessor. Every core name in the old disassembly was shifted by one: keys stood for len, find for format, sep for set, values for write, from_bytes for get, step/split for stop/start, and Ellipsis for Exception. Frozen and const-pool names were unaffected, and so were all constants, struct formats and control flow. The protocol findings on these pages were re-checked against the corrected disassembly.

Immediate-object mapping: True disassembled as Ellipsis. In REPR_A an immediate object is (v << 3) | 6, and py/obj.h assigns none = 0, false = 1, true = 3 — the gap at 2 is deliberate, so that (v >> 3) & 1 is the boolean. The reconstructor read 0x16 (v = 2, unassigned) as True and 0x1E (v = 3) as Ellipsis, so every True in a constant table came out as Ellipsis, in the rebuilt .mpy as well as the printed disassembly. Ellipsis is not an immediate at all; it is a ROM object pointer, and none occurs in these images.

compassing’s constant table showed it plainly — a family of (bool, int) return tuples reading (False, 3), (False, 1), (False, 0) and (Ellipsis, 0) — and f_lib/requests carried the scheme table (80, False), (443, Ellipsis), where the second entry is the HTTPS branch.

Fixed, and every version regenerated: across all four images only compassing, f_lib/requests, f_ota/f_assets and (in 3.2.12 and 4.1.3) touch_button changed, by ten lines or fewer each, and every other module came back byte-identical — which is also a check on the reconstructor being deterministic. Anything on these pages that turned on a True in a constant table was re-read against the corrected output.

Artifacts

The reverse-engineering scratch (rebuilt ELF, extracted symbol/string tables, carved segments) is kept out of git via .gitignore. Regenerate it from the image with the steps above; this docs/ tree is the durable record.