5.8 KiB
On-device person-position experiment
This ESP-IDF project combines the DFR1154 camera pin map with Espressif's pedestrian detection model. It prints timestamped person centers and bounding boxes over USB and broadcasts gaze frames from GPIO43 to the two eye boards. Physical delivery and synchronized eye motion are bench-verified.
The project contains local Wi-Fi OTA support. Its partitions.csv has two 7 MB app slots and its bootloader enables rollback. The one-time USB migration and first OTA update succeeded on 2026-09-30. A later USB flash on 2026-10-01 installed the tracking filter in slot 0; see the tracking check and OTA procedure. OTA cannot change the partition table itself.
Tracking filter — 2026-10-01
After a clearly empty 20-second view yielded 15 apparent person boxes, the camera now rejects top-edge boxes at least 95% of image height. All 15 archived empty-scene boxes fit this geometry; none of 203 matched held-person detections in earlier center/left/right logs did. Once three consistent detections confirm a target, tracking::Gate sends the last confirmed position for up to one second through brief model misses or a new unconfirmed candidate. A persistent miss returns valid=0. The camera still sends the same version-1 UART frame to both eyes; each eye uses valid=1 for red/track and valid=0 for blue/idle. The change compiled, passed its native state-transition test, and was MAC-checked and USB-flashed with verified hashes. The current application SHA-256 is 76d16c8817db9e9568eaf2485a486a0debaa2687fc5a8447954d0cd0d80d468a.
In the same empty view after flashing, the camera sent 0 valid targets in 80 frames and recorded 15 rejected full-height candidates. Both eyes accepted the same invalid states for all 78 overlapping logged sequences. A close real person might also fill the image height, so test this filter with a visitor at the intended near/middle/far approach positions before accepting it for the porch. Raw captures and interpretation.
On 2026-09-27, the gaze build was flashed to the camera after its MAC E8:F6:0A:87:0B:4C was checked. Following a temporary malformed-frame diagnostic build, the normal camera application (SHA-256 0acc763624b20a09afbf512e20afad51ffc9de4276102ffa639a0345d39d70a5) was restored and hash-verified after flashing. Both eyes accepted matching camera coordinates; Joshua confirmed synchronized visible tracking. The bench log records rejection of malformed/stale commands and recovery. A live USB sample showed 231–232 ms capture-to-result time and gaze_valid transitions. The current three-consecutive-frame gate rejects isolated detections, and the camera rejects detections whose boxes span at least 90% of both image dimensions. Other persistent false detections can still pass; this is not finished target selection.
The project builds with ESP-IDF v5.5.2 and was hash-verified after flashing to the camera identified by MAC E8:F6:0A:87:0B:4C. Its first 20-second live serial sample at the model's default 0.70 score threshold reported NO_PERSON at 320 × 240. A subsequent run at threshold 0.35, with better lighting and a person in view, produced stable timestamped bounding boxes at left, center, and right held positions. The corresponding logs and JPEGs are in ../../hardware/camera-captures/, summarized in ../../hardware/bringup.md. Typical capture-to-result latency was 233 ms (about four results per second). Espressif reports about 118 ms model time for a 224 × 224 input on an ESP32-S3; the measured figure here includes capture and JPEG decode. Camera pins match DFRobot's CameraWebServer example. Dependencies are pinned to compatible release ranges in main/idf_component.yml; dependencies.lock records the resolved versions. Diagnostic JPEG output is disabled by default; enable it with FREDDY_CAMERA_DIAGNOSTIC_JPEG=1 for captures.
The current build also drives GPIO47 high at boot to enable the board's infrared illumination, as identified in DFRobot's DFR1154 guide. The camera must be aimed clear of nearby surfaces: an IR-lit frame captured close to a surface washed out into a purple blur, while a re-aimed frame showed the room. This has not yet established better detection in darkness, and the raw model reports some false positives in an empty room. Diagnostic JPEG output is disabled by default in the gaze build and can be re-enabled with FREDDY_CAMERA_DIAGNOSTIC_JPEG=1 for captures.
When diagnostic JPEG output is enabled, use capture_diagnostic.py with PySerial to record synchronized logs and JPEGs. Test continuous walking, dim and brighter lighting, a close person, a distant person, two people, and a person crossing the camera edge before relying on the coordinates for gaze. Improve target persistence and false-positive rejection beyond the preliminary three-frame gate. The DFRobot bundled Person_Detection_inferencing.zip is only a 96 × 96 binary classifier (non_person versus person), so it cannot provide the required position directly.
On 2026-09-27 Joshua reported intermittent person lock. A temporary diagnostic image showed the camera aimed mostly at the ceiling/window, with only part of him at the far left edge; the model alternated PERSON and NO_PERSON and reported left-edge boxes. See the diagnosis and paired frames. The normal build above was restored and verified; first reframe the camera, then retest before tuning tracking hysteresis.