65 lines
14 KiB
Markdown
65 lines
14 KiB
Markdown
# Pi Zero 2 W vision-controller trial
|
||
|
||
Joshua wants to keep both Waveshare ESP32-S3 eye displays and trial his Raspberry Pi Zero 2 W for person detection, target selection, synchronization, and diagnostic logs. He identified an [InnoMaker CAM-OV5647 with an M12 90° diagonal IR-filter lens](https://www.inno-maker.com/product/cam-ov5647/) and showed two Arducam alternatives; his current placement preference is the Arducam day/night camera in Freddy's mouth. This is the preferred **trial architecture**, not yet a measured replacement for the DFRobot camera. The Pi now boots Raspberry Pi OS Lite and is reachable by SSH as `overseer@10.60.1.215`. On 2026-10-01, `rpicam-hello --list-cameras` detected an **IMX708 Camera Module 3** at CSI0; this is not either of the Arducam modules shown earlier. The IMX708 sensor identification does not establish whether the module is the standard or NoIR variant. Physical availability of the Arducam board is unconfirmed. The portable [Pi gaze sender](../firmware/pi_vision/README.md) encodes the existing ESP32 packet and passes local tests, but has not sent a command from the Pi to either eye. Keep the working DFRobot setup available until a Pi walk-by test meets the same gaze-link behavior.
|
||
|
||
## Hardware split
|
||
|
||
```text
|
||
Pi CSI camera → Pi Zero 2 W (detection, one target, tracking state, logs)
|
||
GPIO14 TX, 3.3 V UART, shared ground
|
||
├── eye 1 GPIO44 RX → local eye animation
|
||
└── eye 2 GPIO44 RX → local eye animation
|
||
```
|
||
|
||
The Pi sends the existing [12-byte gaze frame](../protocol/gaze.md) at 115200 baud to both eyes. `valid=1` already means the eyes track and glow red; `valid=0` means idle and blue. Their firmware, Wi-Fi OTA path, iris art, blink, and display wiring do not need changing for the first Pi trial. Disconnect the DFRobot camera's TX from the shared UART line before connecting Pi TX; two transmitter outputs must not be tied together. Join the Pi and both eye grounds, but do not join 3.3 V output rails. The Pi's primary UART is on GPIO14/pin 8; enable UART hardware and disable the serial login console. On Zero 2 W the default primary UART is a mini UART, so consider Raspberry Pi's `disable-bt` overlay if its timing proves unreliable at 115200 baud. [Raspberry Pi UART documentation](https://www.raspberrypi.com/documentation/computers/configuration.html).
|
||
|
||
Both eye boards also have a USB-to-UART Type-C port. A Pi USB OTG host adapter and hub could connect both as separate serial devices and send the same gaze packet to each **without reflashing the eye firmware**. That port's connection disables the eye's 4-pin UART header, so do not use the two paths simultaneously. USB needs a hub, two serial-port writes and stable device identification; the GPIO fan-out is the simpler final arrangement and delivers the same packet to both eyes on one wire. The separate Type-C port labeled `USB` remains available for eye power, diagnostics, and flashing while the 4-pin UART header is used. [Waveshare interface map](https://docs.waveshare.com/ESP32-S3-Touch-LCD-2.1).
|
||
|
||
Pi serial state after Joshua ran `raspi-config` and rebooted on 2026-10-01: `/dev/serial0` points to `/dev/ttyS0`; `enable_uart=1` is set; the kernel command line has only `console=tty1`; GPIO14 is in TXD1 mode. The `overseer` account belongs to `dialout`. The Pi sender now uses Python's built-in `termios`, so it needs no serial package. The physical Pi-to-eye connection and packet delivery still need verification.
|
||
|
||
Camera check on 2026-10-01: `rpicam-hello --list-cameras` initially found the IMX708 and a 640 × 480 `rpicam-still` capture completed. The first frame is saved at [`../diagnostics/pi_imx708_first_frame.jpg`](../diagnostics/pi_imx708_first_frame.jpg), but the lens was blocked. After the UART-config reboot, automatic camera detection failed, even after the ribbon was checked and the Pi rebooted again. `vcgencmd get_throttled` returned `0x0`; the boot device tree had no `imx708` node. Joshua ran `sudo dtoverlay imx708` as a **temporary runtime diagnostic**, after which `rpicam-hello --list-cameras` immediately found the sensor and still capture worked. This points to failed automatic overlay loading at boot, not a disconnected sensor. The overlay will not persist across a reboot; tomorrow's different IR camera may need a different sensor overlay.
|
||
|
||
The latest useful stills are [`../diagnostics/pi_imx708_rotated_view.jpg`](../diagnostics/pi_imx708_rotated_view.jpg) and [`../diagnostics/pi_imx708_person_view.jpg`](../diagnostics/pi_imx708_person_view.jpg). The module currently produces a 180°-inverted frame, corrected by `--rotation 180` in `rpicam-still` or `Transform(hflip=1, vflip=1)` in Picamera2. The second image shows a full standing person in the indoor view. `python3-picamera2` and `python3-opencv` were installed on the Pi for the benchmark.
|
||
|
||
The Pi-to-eye GPIO link was exercised twice with [`smoke_test_link.py`](../firmware/pi_vision/smoke_test_link.py). The second run was visually confirmed by Joshua: both eyes moved left, center, right together while red, then returned to blue idle. This establishes the Pi UART wiring and packet compatibility but does not yet prove camera-driven tracking.
|
||
|
||
## 2026-10-01 IMX708 person-tracking trial
|
||
|
||
- At 320 × 240 with OpenCV's built-in full-body HOG detector, a 15-second empty-room sample processed **94 frames at 6.09 fps** with zero detections. A 20-second stationary-person sample processed **126 frames at 6.16 fps**, detecting a person in 106 frames. Median detector time was about 156 ms. Logs: [`../diagnostics/pi_hog_20261001-200202.jsonl`](../diagnostics/pi_hog_20261001-200202.jsonl) and [`../diagnostics/pi_hog_20261001-200343.jsonl`](../diagnostics/pi_hog_20261001-200343.jsonl).
|
||
- A 25-second moving-person live trial initially detected a person in only 43/157 raw frames. The longest raw gap was 3.34 seconds; a 2-second hold allowed three red-to-blue drops, and Joshua saw flicker. The gaze link kept both eyes synchronized. Log: [`../diagnostics/pi_eye_trial_20261001-201006.jsonl`](../diagnostics/pi_eye_trial_20261001-201006.jsonl).
|
||
- Replaying that log with a 3.5-second hold eliminated its three intermediate red-to-blue drops. A new 25-second live trial processed **158 frames**, detected a person in 92 raw frames, and held valid tracking for 140 frames. Gaze X ranged 109–605/1000, with **zero intermediate red-to-blue drops**. Joshua visually confirmed both eyes stayed red and followed together. Log: [`../diagnostics/pi_eye_trial_20261001-201350.jsonl`](../diagnostics/pi_eye_trial_20261001-201350.jsonl).
|
||
- This establishes camera-to-Pi-to-both-eyes integration indoors. The HOG detector still misses moving people for seconds at a time; during a held miss, eye gaze freezes at the last position, and after departure red can linger for up to 3.5 seconds. IMX708 detection at boot has been intermittent, and multiple-person selection is naive. The porch night result is recorded below. The Arducam IR camera swap requires a new sensor identification and may need its own overlay before repeating the benchmarks.
|
||
|
||
## Porch relocation check — 2026-10-01 night
|
||
|
||
- After a clean shutdown and move, the Pi rejoined Wi-Fi and `/dev/serial0` remained available. On this reboot the IMX708 **was** auto-detected without a manual overlay, so the earlier intermittent boot-detection failure is unresolved rather than permanent. The porch-mounted Pi-to-eye smoke test again made both displays look left, center, and right together in red, then return to blue; Joshua confirmed it visually.
|
||
- The camera still needs 180° software rotation. The unlit upright frame [`../diagnostics/pi_imx708_porch_upright.jpg`](../diagnostics/pi_imx708_porch_upright.jpg) shows the railing/posts but almost no usable detail in the visitor path. With the porch light on, [`../diagnostics/pi_imx708_porch_lit.jpg`](../diagnostics/pi_imx708_porch_lit.jpg) shows Joshua, though the image remains dim and his lower body is cut off.
|
||
- A 12-second **detector-only** lit-porch sample processed 76 frames at 6.05 fps and detected a person in just **2 frames**, while Joshua confirmed he was visible most of the time. Evidence: [`../diagnostics/pi_hog_20261001-202712.jsonl`](../diagnostics/pi_hog_20261001-202712.jsonl). **No-go for using this IMX708/HOG combination as a dependable night visitor trigger in the current porch view.** Do not leave `run_hog_eye.py --drive-eyes` running unattended on this evidence. The Arducam IR camera arriving 2026-10-02 should be evaluated with its actual night image and full-body framing; better illumination alone may not fix the HOG detector's failure on partial bodies.
|
||
|
||
The [InnoMaker listing](https://www.amazon.com/dp/B07G9VLPZH) says this OV5647 module includes both standard and Pi Zero camera ribbons, though the actual cables in Joshua's kit should be checked. Its 39 × 39 mm board and manual-focus M12 lens need a new physical fit check; the existing DFRobot chair pod is not assumed to fit. Raspberry Pi's camera software supports the OV5647 sensor, but the image still needs verification on this third-party board. [Raspberry Pi camera software](https://www.raspberrypi.com/documentation/computers/camera_software.html).
|
||
|
||
The identified lens has an IR filter, so do not assume a separate IR illuminator will provide useful night vision. Test with the real porch lighting. If the night image is inadequate, check whether Joshua owns another NoIR camera or a confirmed compatible no-IR-filter lens before changing hardware. The camera and lighting still need the concealed chair placement and weather protection described in [outdoor deployment](outdoor-deployment.md).
|
||
|
||
## Additional camera candidates identified 2026-10-01
|
||
|
||
Joshua showed an [Arducam 8 MP IMX219 NoIR](https://www.arducam.com/8mp-imx219-noir-for-raspberry-pi-b0395.html) and an [Arducam 5 MP OV5647 day/night camera with switchable IR-cut filter and two IR LED boards](https://www.arducam.com/arducam-for-raspberry-pi-noir-5mp-ov5647-camera-module-motorized-ir-cut-filter-for-daylight-and-night-vision-support-pi-4-zero-pi-3.html). Do not assume these are physically in hand until Joshua confirms. Both list Zero 2 W support and Pi Zero camera cables. The NoIR's 25 × 24 mm board is easier to conceal, but its fixed lens covers about 62° horizontally and needs separate IR illumination. The switchable OV5647 covers about 105° horizontally, offers daytime color and night IR, and has a 36 × 36 mm camera board plus two adjacent IR LED boards. Camera capture rate does not establish Pi person-detection rate; benchmark the detector separately.
|
||
|
||
Joshua's preferred first placement trial for the switchable OV5647 is **inside Freddy's mouth**, not behind the eyes or under the chair armrest. In this character location the two IR emitters may contribute a faint visible red point-glow, but their wavelength and appearance are unverified; do not count on them to produce a broad red aura. A separate dim visible LED effect could supply that look if desired. The mouth cavity must leave lens and both emitters with a clear forward sightline, separate them from reflective teeth/muzzle surfaces, and provide cable/heat clearance. See [mouth camera placement](mouth-camera.md). The earlier chair-armrest DFRobot pod remains a backup and does not fit the Arducam by assumption.
|
||
|
||
## Trial sequence
|
||
|
||
1. Confirm which camera is physically available, check its 15-to-22-pin Pi Zero ribbon, and boot the Pi with Raspberry Pi OS and that camera attached. Verify it appears in `rpicam-hello --list-cameras`. Save a daylight frame and, if using the InnoMaker or switchable OV5647, set manual focus for the visitor path. Check a night frame with the actual porch lighting before choosing an IR solution.
|
||
2. Benchmark a lightweight person detector on the Zero 2 W using the actual camera view. This Pi has a quad-core 1 GHz CPU and 512 MB RAM, so person-detection speed and memory must be measured before assuming it improves on the roughly four-results-per-second ESP32 baseline. Record frames processed per second, detection latency, CPU/memory use, and example frames with and without people. [Pi Zero 2 W specifications](https://www.raspberrypi.com/news/new-raspberry-pi-zero-2-w-2/).
|
||
3. Add one-target continuity and a short miss hold, then translate the chosen bounding-box center into the existing 0–1000 gaze coordinates. Emit exactly one serial stream to both eyes. Log timestamp, detector confidence and box, selected target, gaze validity and coordinates, and transmitted sequence locally with rotation; video need not be recorded continuously.
|
||
4. Disconnect DFRobot TX, connect Pi TX and common ground, and confirm both eyes accept the same frame sequences. Compare their visible gaze direction and red/blue state during approach, departure, an empty scene, and two-person crossings.
|
||
5. Evaluate from the actual concealed chair position in daylight, dusk, and darkness before retiring the DFRobot camera. A Pi-side change cannot by itself synchronize independent eye blink or glow-pulse phase; if those effects need frame-accurate synchronization, extend the gaze protocol with a shared animation cue in a later step.
|
||
|
||
If CPU-only inference cannot meet the measured response target, a Pi AI Camera is a potential later upgrade: Raspberry Pi documents its IMX500 on-camera object detection and says it can be used with Zero 2 W with minor setup changes. This is a fallback, not a purchase recommendation before benchmarking the camera Joshua already owns. [Raspberry Pi AI Camera documentation](https://www.raspberrypi.com/documentation/accessories/ai-camera.html).
|
||
|
||
## Acceptance gate
|
||
|
||
- Both eyes receive the same valid/invalid state and coordinates from the Pi with no dropped or malformed frames in a sustained bench run.
|
||
- A visitor walking across the intended porch path stays selected without rapid switching; departures return both eyes to blue promptly.
|
||
- Empty-path tests do not trigger persistent red eyes. Measure false triggers across day, dusk, and night instead of inferring performance from indoor samples.
|
||
- The Pi meets or improves on the current ~4 Hz camera result rate and keeps end-to-end visible response acceptable in the installed view; otherwise retain the ESP32 camera or try an accelerator.
|