> ## Documentation Index
> Fetch the complete documentation index at: https://totem-cb8b3887.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Methodology & next steps

> How this firmware was analyzed, the tools used, and how to close the remaining gaps.

## Approach

The firmware findings come from **static analysis** of public artifacts: the v5.0.2 and
v5.0.3 firmware images and the companion Android app. No network service was attacked; the
releases API and S3 objects are served openly. The BLE protocol was additionally checked
end to end with the `totemctl` client (`cmd/totemctl` in this repository) against the
author's own device on firmware 4.1.3 and 5.0.3. That test found the
[transmit-mode](/protocols/ble#transmit-modes) behaviour on macOS, which static analysis
alone had missed. The ESP-NOW mesh was checked the same way with the
[ESP32 emulator](/reference/esp32-emulator) (`cmd/totememu`): it bonded with the author's
Totem on 5.0.3 and exchanged peer status with it, which confirmed the radio settings, the
peer frame layout and the pairing handshake.

### Tools

| Tool                              | Use                                                                |
| --------------------------------- | ------------------------------------------------------------------ |
| `esptool`                         | parse the ESP-IDF image header, segments, and app descriptor       |
| Python + `struct`                 | parse headers, carve segments, read the `esp_app_desc_t`           |
| `capstone` 6.0 (`CS_ARCH_XTENSA`) | disassemble the native Xtensa code                                 |
| `rizin` / `rabin2`                | validate the rebuilt ELF                                           |
| `strings` + `grep`                | recover the qstr symbol table and string constants                 |
| `hermes-dec`                      | decompile the Android app's Hermes bytecode (for the download URL) |

### Steps

<Steps>
  <Step title="Parse the image">
    Confirm ESP32, 7 segments, entry point, and read the app descriptor
    (IDF v5.4, build date, ELF hash).
  </Step>

  <Step title="Rebuild an ELF">
    Map the 7 segments to their load addresses with correct R/W/X perms in an
    `EM_XTENSA` ELF for Ghidra/IDA/rizin.
  </Step>

  <Step title="Identify the runtime">
    Native strings reveal MicroPython v1.25.0 — the app is Python, frozen as bytecode.
  </Step>

  <Step title="Recover the symbol table">
    Extract the qstr pools and rodata strings: the application's method, attribute and
    class names and every log line. The first v5.0.2 pass counted 2,479 identifiers and
    11,857 strings by string scanning. Walking the pools directly gives 4,086 qstrs for
    v5.0.2 (228 core + 972 const + 2,886 frozen) and 4,162 for v5.0.3 (228 + 975 + 2,959).
  </Step>

  <Step title="Reconstruct behavior">
    Map modules, classes, protocol categories, and subsystem behavior from those symbols
    and the `{}`-style (Python `str.format`) log messages.
  </Step>
</Steps>

## Coverage audit

Symbol recovery tells you what exists. It does not tell you what anyone has actually
*read*, and the two are easy to confuse — a module can be named, categorised and given a
confident one-line role on the strength of nothing but its filename.

Every module was therefore audited against `re/v5.0.3/mpy/*.dis` and assigned one of four
ratings, published per module on the [module reference](/reference/modules):

| Rating        | Bar                                                                                      |
| ------------- | ---------------------------------------------------------------------------------------- |
| **DECODED**   | Behaviour documented from the bytecode: named functions, constant *values*, control flow |
| **PARTIAL**   | Some verified internals documented, main logic not                                       |
| **NAME-ONLY** | Role is a hypothesis from the module name and a few strings; never read                  |
| **STOCK**     | Upstream MicroPython or `micropython-lib`; out of scope                                  |

### How the ratings were derived

1. **Import graph.** Every `IMPORT_NAME` opcode in all 94 disassemblies was extracted and
   resolved against the module list, giving each module's importers. This is what
   identified `apa106.py` and `neopixel.py` as dead code, `chat_msg.py`, `f_ota/hotspot.py`
   and `peer_auto_bond.py` as unreferenced, and `f_lib/logger.py` (41 importers) and
   `f_lib/task_mgr.py` (28) as the load-bearing modules.
2. **Distinctive-symbol coverage.** For each module, the identifiers in its `qstr_table`
   that appear in at most two modules across the image — its *distinctive* symbols, which
   excludes shared names like `config` or `Event` — were checked against the full text of
   `docs/`. The fraction of them that appear is a mechanical proxy for decode depth.
   When the audit ran, `f_lib/logger.py`, `f_lib/bitwise.py` and `f_lib/async_helpers.py`
   each scored zero; all three have since been read and are written up on
   [f\_lib](/reference/f-lib).
3. **Manual adjudication.** The proxy is unreliable for small modules (a module with one
   distinctive symbol scores 0% or 100%), so every rating was set by reading what the docs
   actually claim about that module and checking it against the disassembly. The proxy
   ranked the candidates; it did not decide them.

<Warning>
  A "documented" module is not necessarily a *correct* one. This audit found several
  confident claims that the bytecode contradicts: `apa106.py` presented as the LED driver
  (it is unreachable; `leds.py` imports `f_lib/neopixel_v2.py`), a deep-sleep power state
  (the firmware never calls `machine.deepsleep()`), `f_lib/rtc_v2.py` described as
  RTC-memory storage (it is a `machine.RTC` subclass doing calendar maths), and `BleCtrl`
  listed as a class (it is a log-line prefix; the class is `BleController`).

  Each of those came from reading a name or a log string and not the code. That is exactly
  the failure mode the NAME-ONLY rating exists to flag.
</Warning>

## Why not a full source decompile?

The application is **frozen MicroPython bytecode**. Unlike CPython (which has mature
decompilers such as `uncompyle6`/`decompyle3`), MicroPython has **no mature
bytecode-to-source decompiler**. The best available is *bytecode disassembly* (opcodes
plus qstr references), not clean Python source.

However, the frozen format keeps **every identifier as an interned string** (qstr) and
retains all string literals. That symbol table plus the log messages reconstruct each
module's behavior at close to source fidelity — which is how this documentation was
built.

## Closing the remaining gaps

Three items were graded "needs bytecode disassembly" in the
[protocol completeness matrix](/protocols/overview):

1. The complete per-category `cmd_id` enumeration (`TOTEM_MSG_MAP`).
2. The exact binding of each `struct` format to a specific message.
3. The field order of `CHUNK_HDR_FMT`.

The frozen-bytecode disassembly has now been carried out and all three are **recovered**.
The values below are literals present in the binary — every `struct`-format string appears
verbatim in the DROM rodata — and the method that produced them is recorded afterward.

<Note>
  The [`/protocols/overview`](/protocols/overview) completeness matrix now marks these three
  as **Recovered** and reproduces the full decoded map; this page records how they were obtained.
</Note>

### `TOTEM_MSG_MAP` — confirmed

Defined in `espnow_conn_v2.py`, built by a `BUILD_MAP 13` / `STORE_NAME TOTEM_MSG_MAP`
opcode sequence (fully decoded). The map is **flat**, keyed by a 2-byte `bytes` value
`(cat_id, cmd_id)` (byte 0 = `cat_id`, byte 1 = `cmd_id`); **every** value is a payload
`struct`-format string.

<Warning>
  An earlier pass decoded the map's qstr-immediate values with `>>2` and reported
  "handler/type names" (`disconn_animation`, `dev_total_lightsleep_ms`, `dev_info`,
  `device_power`, `disabled`). Those were **decode artifacts and are wrong**. On ESP32
  MicroPython a qstr-immediate is tagged by `(o & 7) == 2` (REPR\_A), and the qstr number is
  `o >> 3`, not `o >> 2`. Under the correct `o >> 3` decode every entry resolves to a
  `struct` format string — the values below.
</Warning>

| `(cat_id, cmd_id)`             | `struct` format      | Size |
| ------------------------------ | -------------------- | ---- |
| `(0, 0)` / `(0, 1)` / `(0, 2)` | `<BBffbbhbbb6B`      | 23 B |
| `(1, 0)`                       | `<BBB7bHbb`          | 14 B |
| `(1, 2)`                       | `<BBiffHii`          | 24 B |
| `(1, 5)`                       | `<BB9B8bhh4B`        | 27 B |
| `(1, 6)`                       | `<BBbbB`             | 5 B  |
| `(1, 7)`                       | `<3Bbb`              | 5 B  |
| `(2, 0)`                       | `<BB6BffbbHbbbhhBBi` | 33 B |
| `(3, 1)` / `(3, 2)`            | `<BBHH4b4BHHBb`      | 20 B |
| `(7, 0)`                       | `<BBffbbbHbbH`       | 19 B |
| `(7, 1)`                       | `<BBHbffb`           | 14 B |

### `struct` format → message bindings — confirmed (for listed)

Each payload format begins with `BB` = the echoed `cat_id`/`cmd_id` header:

| `(cat_id, cmd_id)` | `struct` format | Size |
| ------------------ | --------------- | ---- |
| `(1, 0)`           | `<BBB7bHbb`     | 14 B |
| `(1, 2)`           | `<BBiffHii`     | 24 B |
| `(1, 6)`           | `<BBbbB`        | 5 B  |
| `(1, 7)`           | `<3Bbb`         | 5 B  |
| `(7, 1)`           | `<BBHbffb`      | 14 B |

### Transfer header — format & field order confirmed

`CHUNK_HDR_FMT = <HBBBiHiB`, `CHUNK_HDR_SZ = 16` bytes — packed by `f_ble/chunking.py`'s
`gen_transfer_buff` via `struct.pack_into` at buffer offset 2. The format string, field order,
**and field names** are **confirmed** from the live `pack_into` opcode stream: `gen_transfer_buff`
calls `struct.pack_into('<HBBBiHiB', buff, 2, self.file_id, self.status_id, self.action_id, self.file_type_id, self.byte_pos, self.chunk_no, self.file_size, 0)`
(`LOAD_ATTR` order in the bytecode). The first pass also bound the *name* `CHUNK_HDR_FMT` to
this format, because `f_ble/file_upload.py` imports `CHUNK_HDR_FMT` and `CHUNK_HDR_SZ` from
`chunking`. **That inference was wrong.** v5.0.2's `chunking` never defines those names, so
the import could not have succeeded. v5.0.3 defines them explicitly, as the separate 12-byte
**chunk** header: `CHUNK_HDR_FMT = '<HHiH'`, `CHUNK_HDR_SZ = 12`, `UPLOAD_TO_APP = 3`. See
[chunking](/protocols/message-format#chunking). v5.0.3 also turns the last byte below into
a flags byte.

| `struct` code | Field                                                                         | Type      |
| ------------- | ----------------------------------------------------------------------------- | --------- |
| `H`           | `file_id`                                                                     | uint16 LE |
| `B`           | `status_id`                                                                   | uint8     |
| `B`           | `action_id`                                                                   | uint8     |
| `B`           | `file_type_id`                                                                | uint8     |
| `i`           | `byte_pos`                                                                    | int32 LE  |
| `H`           | `chunk_no`                                                                    | uint16 LE |
| `i`           | `file_size`                                                                   | int32 LE  |
| `B`           | literal 0 in v5.0.2; `pack_flags(is_last_chunk, is_origin_compass)` in v5.0.3 | uint8     |

### How it was done

<Steps>
  <Step title="Locate the frozen data">
    In the image (file offsets): `0xf3d3` points *into* the qstr identifier-string data,
    but mid-pool — the sorted run of identifier strings starts much earlier (around
    `0xe000`) and flows continuously through `0xf3d3` with no sort reset. The concrete
    anchors are the frozen qstr pool header at seg0 `0x5de0c` (`prev=0x3f4213d0`,
    `total_prev_len=1200`, `len=2886`) with its `qstrs[]` pointer array at `0x5de24`. The
    frozen module-name registry begins near `0x27b00`; the DROM segment maps file `0x18` →
    vaddr `0x3f400020`.
  </Step>

  <Step title="Parse the frozen module table">
    Reconstruct the MicroPython v1.25 (`.mpy` version 6.3) `mp_raw_code_t` tree for the two
    comms modules that actually carry these constants: `espnow_conn_v2.py` (home of
    `TOTEM_MSG_MAP` and the per-message `struct`-format registry) and `f_ble/chunking.py`
    (home of `CHUNK_HDR_FMT`).
  </Step>

  <Step title="Disassemble the bytecode">
    Walk the bytecode with a v1.25-matched reader. Reuse the opcode tables from `py/bc0.h`
    / `tools/mpy-tool.py`, but note `mpy-tool.py -d` itself only reads standalone `.mpy`
    blobs; frozen firmware modules (emitted as `proto_fun_*` / `fun_data_*` /
    `const_qstr_table_data_*` C structs, not `.mpy` blobs) require a custom frozen-aware
    reader. Small integers `-16..47` load inline as single-byte opcodes `0x80`–`0xBF`;
    larger ones via `LOAD_CONST_SMALL_INT` (`0x22`, signed var-int) — both embed the value
    in the bytecode. The `TOTEM_MSG_MAP` dict build and the `CHUNK_HDR_FMT` string literal
    are then readable.
  </Step>

  <Step title="Cross-check">
    Validate recovered `(cat_id, cmd_id)` values against the pairs already visible in the
    logs. `(0x01, 0x02)` and `(0x03, 0x01)` validate directly against `TOTEM_MSG_MAP`.
    `(0x06, 0x07)` is a Peer-Sync BLE control frame handled outside `TOTEM_MSG_MAP`, so
    `cat 6` is absent from the map — its absence is expected, not a failed cross-check.
  </Step>
</Steps>

An alternative dynamic approach — running the exact MicroPython build under an ESP32
emulator or on hardware and dumping `TOTEM_MSG_MAP` / `CHUNK_HDR_FMT` from a REPL — would
yield the same values faster if a device or matching build is available.

## Diffing firmware versions

The v5.0.3 image was processed with the same reconstruction and compared to v5.0.2 function
by function:

<Steps>
  <Step title="Carve and find anchors">
    Carve the 7 segments from the image header. Then locate the structures the
    reconstructor needs by their shape rather than by fixed addresses:

    * **qstr pools:** headers `{prev, total_prev_len, alloc, len, hashes*, lengths*,
      qstrs[]}`, validated by the NUL terminator at each declared length.
    * **Type objects:** `mp_type_type` is the only type object whose base points to itself;
      `str`, `bytes`, `tuple`, `float` and `int` point to it and carry their name qstr at +6.
    * **Frozen module table:** the longest run of pointers to `{qstr_table, obj_table,
      proto}` records whose first qstr ends in `.py`.

    On v5.0.2 this reproduces every hand-found address. On v5.0.3 it finds the frozen pool at
    seg0 `0x5f59c` (base 1203, 2,959 qstrs), the const pool at `0x22208` (975) and the module
    table at `0x3f4287b0`–`0x3f428924` (94 modules).
  </Step>

  <Step title="Reconstruct and disassemble">
    Re-serialize every frozen module as a standalone v6.3 `.mpy` and disassemble it with
    MicroPython's own `mpy-tool.py -d`: 96/96 modules for v5.0.2 and 94/94 for v5.0.3, with
    no unknown opcodes, truncated functions or dangling children.
  </Step>

  <Step title="Diff">
    Split each disassembly into functions, drop the byte-offset column, and compare
    added, removed and changed functions plus the qstr and constant tables. **20 modules
    changed, 74 are identical** and 2 were removed. See
    [What changed in 5.0.3](/firmware/changes-5.0.3).

    That page says 21 changed, and both numbers are right: it compares the `.mpy` files
    themselves, and a `.mpy` also carries the source line-number table, which moves
    whenever a file is recompiled. `ble_controller.py` is the one module that differs
    that way and no other — same qstrs, same constant table, same instructions, every
    line-info entry shifted by exactly +2. Twenty-one modules were rebuilt; twenty had
    their code changed.
  </Step>
</Steps>

<Warning>
  **Core-qstr off-by-one in the first v5.0.2 pass.** The reconstructor looked up core qstr
  *g* at `qstrs + 4*(g-1)`. The core pool's `qstrs[0]` is the null qstr, so this read the
  name of qstr *g − 1*, the alphabetical predecessor. Every core name in the old disassembly
  was shifted by one: `keys` stood for `len`, `find` for `format`, `sep` for `set`,
  `values` for `write`, `from_bytes` for `get`, `step`/`split` for `stop`/`start`, and
  `Ellipsis` for `Exception`. Frozen and const-pool names were unaffected, and so were all
  constants, struct formats and control flow. The protocol findings on these pages were
  re-checked against the corrected disassembly.
</Warning>

<Warning>
  **Immediate-object mapping: `True` disassembled as `Ellipsis`.** In `REPR_A` an immediate
  object is `(v << 3) | 6`, and `py/obj.h` assigns `none = 0`, `false = 1`, **`true = 3`** —
  the gap at 2 is deliberate, so that `(v >> 3) & 1` is the boolean. The reconstructor read
  `0x16` (v = 2, unassigned) as `True` and `0x1E` (v = 3) as `Ellipsis`, so **every `True`
  in a constant table came out as `Ellipsis`**, in the rebuilt `.mpy` as well as the
  printed disassembly. Ellipsis is not an immediate at all; it is a ROM object pointer, and
  none occurs in these images.

  `compassing`'s constant table showed it plainly — a family of `(bool, int)` return tuples
  reading `(False, 3)`, `(False, 1)`, `(False, 0)` and `(Ellipsis, 0)` — and
  `f_lib/requests` carried the scheme table `(80, False)`, `(443, Ellipsis)`, where the
  second entry is the HTTPS branch.

  Fixed, and every version regenerated: across all four images only `compassing`,
  `f_lib/requests`, `f_ota/f_assets` and (in 3.2.12 and 4.1.3) `touch_button` changed, by
  ten lines or fewer each, and every other module came back byte-identical — which is also
  a check on the reconstructor being deterministic. Anything on these pages that turned on
  a `True` in a constant table was re-read against the corrected output.
</Warning>

## Artifacts

The reverse-engineering scratch (rebuilt ELF, extracted symbol/string tables, carved
segments) is kept out of git via `.gitignore`. Regenerate it from the image with the
steps above; this `docs/` tree is the durable record.