22 lines
5.8 KiB
Markdown
22 lines
5.8 KiB
Markdown
# On-device person-position experiment
|
||
|
||
This ESP-IDF project combines the DFR1154 camera pin map with Espressif's [pedestrian detection model](https://github.com/espressif/esp-dl/blob/master/models/pedestrian_detect/README.md). It prints timestamped person centers and bounding boxes over USB and broadcasts [gaze frames](../../protocol/gaze.md) from GPIO43 to the two eye boards. Physical delivery and synchronized eye motion are bench-verified.
|
||
|
||
The project contains local Wi-Fi OTA support. Its `partitions.csv` has two 7 MB app slots and its bootloader enables rollback. The one-time USB migration and first OTA update succeeded on 2026-09-30. A later USB flash on 2026-10-01 installed the tracking filter in slot 0; see the [tracking check](../../hardware/tracking-filter-2026-10-01.md) and [OTA procedure](../ota.md). OTA cannot change the partition table itself.
|
||
|
||
## Tracking filter — 2026-10-01
|
||
|
||
After a clearly empty 20-second view yielded 15 apparent person boxes, the camera now rejects top-edge boxes at least 95% of image height. All 15 archived empty-scene boxes fit this geometry; none of 203 matched held-person detections in earlier center/left/right logs did. Once three consistent detections confirm a target, `tracking::Gate` sends the last confirmed position for up to one second through brief model misses or a new unconfirmed candidate. A persistent miss returns `valid=0`. The camera still sends the same version-1 UART frame to both eyes; each eye uses `valid=1` for red/track and `valid=0` for blue/idle. The change compiled, passed its native state-transition test, and was MAC-checked and USB-flashed with verified hashes. The current application SHA-256 is `76d16c8817db9e9568eaf2485a486a0debaa2687fc5a8447954d0cd0d80d468a`.
|
||
|
||
In the same empty view after flashing, the camera sent **0 valid targets in 80 frames** and recorded 15 rejected full-height candidates. Both eyes accepted the same invalid states for all 78 overlapping logged sequences. A close real person might also fill the image height, so test this filter with a visitor at the intended near/middle/far approach positions before accepting it for the porch. [Raw captures and interpretation](../../hardware/tracking-filter-2026-10-01.md).
|
||
|
||
On 2026-09-27, the gaze build was flashed to the camera after its MAC `E8:F6:0A:87:0B:4C` was checked. Following a temporary malformed-frame diagnostic build, the normal camera application (SHA-256 `0acc763624b20a09afbf512e20afad51ffc9de4276102ffa639a0345d39d70a5`) was restored and hash-verified after flashing. Both eyes accepted matching camera coordinates; Joshua confirmed synchronized visible tracking. The [bench log](../../hardware/gaze-link-bench.md) records rejection of malformed/stale commands and recovery. A live USB sample showed 231–232 ms capture-to-result time and `gaze_valid` transitions. The current three-consecutive-frame gate rejects isolated detections, and the camera rejects detections whose boxes span at least 90% of both image dimensions. Other persistent false detections can still pass; this is not finished target selection.
|
||
|
||
The project builds with ESP-IDF v5.5.2 and was hash-verified after flashing to the camera identified by MAC `E8:F6:0A:87:0B:4C`. Its first 20-second live serial sample at the model's default 0.70 score threshold reported `NO_PERSON` at 320 × 240. A subsequent run at threshold 0.35, with better lighting and a person in view, produced stable timestamped bounding boxes at left, center, and right held positions. The corresponding logs and JPEGs are in `../../hardware/camera-captures/`, summarized in `../../hardware/bringup.md`. Typical capture-to-result latency was 233 ms (about four results per second). Espressif reports about 118 ms model time for a 224 × 224 input on an ESP32-S3; the measured figure here includes capture and JPEG decode. Camera pins match DFRobot's [CameraWebServer example](https://github.com/DFRobot/DFR1154_Examples/blob/master/5.3%20CameraWebServer/source%20code/5_3/5_3.ino). Dependencies are pinned to compatible release ranges in `main/idf_component.yml`; `dependencies.lock` records the resolved versions. Diagnostic JPEG output is disabled by default; enable it with `FREDDY_CAMERA_DIAGNOSTIC_JPEG=1` for captures.
|
||
|
||
The current build also drives GPIO47 high at boot to enable the board's infrared illumination, as identified in [DFRobot's DFR1154 guide](https://wiki.dfrobot.com/dfr1154/). The camera must be aimed clear of nearby surfaces: an IR-lit frame captured close to a surface washed out into a purple blur, while a re-aimed frame showed the room. This has not yet established better detection in darkness, and the raw model reports some false positives in an empty room. Diagnostic JPEG output is disabled by default in the gaze build and can be re-enabled with `FREDDY_CAMERA_DIAGNOSTIC_JPEG=1` for captures.
|
||
|
||
When diagnostic JPEG output is enabled, use `capture_diagnostic.py` with PySerial to record synchronized logs and JPEGs. Test continuous walking, dim and brighter lighting, a close person, a distant person, two people, and a person crossing the camera edge before relying on the coordinates for gaze. Improve target persistence and false-positive rejection beyond the preliminary three-frame gate. The DFRobot bundled `Person_Detection_inferencing.zip` is only a 96 × 96 binary classifier (`non_person` versus `person`), so it cannot provide the required position directly.
|
||
|
||
On 2026-09-27 Joshua reported intermittent person lock. A temporary diagnostic image showed the camera aimed mostly at the ceiling/window, with only part of him at the far left edge; the model alternated `PERSON` and `NO_PERSON` and reported left-edge boxes. See [the diagnosis and paired frames](../../hardware/tracking-drop-diagnosis.md). The normal build above was restored and verified; first reframe the camera, then retest before tuning tracking hysteresis.
|