AmViSH 2.0 — Speech-Driven Mobile Robot Prototype

A source-based thesis on connecting speech, a hosted language model, and embedded movement through a small, inspectable action vocabulary.

Abstract

AmViSH 2.0 is a prototype mobile-robot interaction system connecting a laptop speech pipeline to microcontroller-based actuators. The laptop captures mono audio in fixed-duration chunks, applies an RMS-energy comparison, transcribes selected chunks locally with OpenAI Whisper, and sends recognized text to a Groq-hosted chat-completion model. The model is prompted for JSON containing an action label and spoken response. The laptop validates the action against an allowlist, posts it to an ESP32 over local HTTP, and speaks the response with pyttsx3. UDP ping/pong discovery locates the ESP32, which forwards the action over UART to an Arduino for motor and servo control.

Source analysis identifies important limitations: processing is synchronous; audio gating is a basic heuristic; response validation is partial; several prompted actions are unsupported by firmware; movement has no independent timeout; and the HTTP control endpoint is unauthenticated. The checked-in runtime does not implement camera perception, face recognition, obstacle avoidance, persistent memory, or measured evaluation. This thesis documents the actual architecture, implementation, privacy and safety implications, limitations, and a verification roadmap.

Keywords: human–robot interaction, speech recognition, language models, ESP32, Arduino, embedded control.

Purpose and scope

Robotic speech interaction combines audio acquisition, transcription, intent interpretation, action selection, command transport, physical actuation, and response generation. AmViSH 2.0 explores this layered arrangement using laptop software, an ESP32 network bridge, and an Arduino actuator controller. Its contribution is an integrated prototype path and an inspectable boundary between language interpretation and predefined embedded actions.

The model receives recognized text and returns a symbolic action and response. It does not receive raw audio, images, or live sensor readings, and it does not directly generate motor signals. The system does not implement learning, self-awareness, autonomous goal generation, or persistent memory.

System architecture

1. Capture: sounddevice records mono audio at 16 kHz in nominal 3.5-second chunks.
2. Gate: root-mean-square energy is compared with a stored ambient estimate.
3. Transcribe: OpenAI Whisper runs locally and is configured to transcribe English.
4. Interpret: recognized text and up to eight recent process-local messages are sent to a Groq chat-completion model, which is prompted for ACT and SAY.
5. Validate and route: the host parses JSON, checks the action label, posts /ACT to the ESP32, and sends non-empty speech to pyttsx3.
6. Actuate: the ESP32 forwards ACT:LABEL over UART at 9600 baud; the Arduino maps supported labels to motor or servo outputs.

At startup the laptop broadcasts AMVISH_PING over UDP port 4210 and accepts a matching AMVISH_PONG. The responder is not authenticated, and the program does not repeat discovery after startup failure.

The active bridge is firmware/esp32_cam_act_bridge/esp32_cam_act_bridge.ino. The alternate firmware/ESP32_main/ESP32_main.ino supports status and discovery but has no /ACT route, so it cannot replace the documented bridge.

Implementation details

Audio gate and transcription

The first chunk initializes the ambient-energy value and is discarded. Later chunks below half that estimate update it using a smoothing coefficient; chunks above the estimate multiplied by MARGIN are returned. The configured margin is 0.5. This is a simple relative amplitude heuristic, not a validated voice-activity detector or an ambient-plus-margin threshold. It has no frequency analysis, hysteresis, or measured noise testing. Whisper rejects text only if it is shorter than two characters or contains no alphabetic character; no confidence score is checked.

Language-model contract

The API key is loaded from the environment. The API call is synchronous at fixed temperature 0.4. Malformed JSON and action labels outside the allowlist fall back to IDLE. The host does not fully validate the type, length, or content of SAY. An allowlist restricts syntax but cannot ensure a permitted movement is appropriate. The prompt is not a defense against prompt injection.

Sequential execution

The Python runtime processes capture, transcription, hosted inference, HTTP, and speech synthesis one after another. These stages block one another; the system is not a non-blocking real-time loop and has no defined response deadline. HTTP errors are printed and swallowed, and the response status is not treated as a verified actuator acknowledgment.

Embedded controller

The ESP32 accepts unauthenticated HTTP on port 80, extracts an action using basic string searches, then writes a newline-terminated serial message using GPIO 15 as TX. Arduino reads the ESP32 on D2, controls two motor channels and three servos, and dispatches actions when the state changes. Movement continues until a later state transition; there is no independent motor timeout or emergency-stop input. Servo wave routines use blocking delays. Repeating an action already in the active state may not retrigger it.

The wiring workbook lists an ultrasonic sensor and I²C display, but the supplied runtime does not implement them. Workbook ultrasonic TRIG on D2 conflicts with the Arduino's software-serial RX on D2; do not wire that workbook assignment with the current firmware.

Action compatibility

The host and firmware vocabularies are not fully aligned. They should be synchronized and tested together.

Verification plan

The current repository does not provide physical test results. A reproducible evaluation should record source revision, board and driver models, operating system, microphone, room conditions, network layout, model identifier, and date. For each trial, record the expected action, transcript, model output, HTTP response, UART message, observed actuator result, and timestamps. Do not publish secrets or private recordings.

• Audio: test speech, quiet, fan noise, music, and changing background; measure false accepts/rejections and threshold behavior.
• Transcription: use a labeled utterance set at multiple distances and noise levels; report word error rate and inference time.
• Model contract: test malformed JSON, unknown actions, invalid or oversized speech, and provider timeouts.
• Network and firmware: test discovery failures, repeated commands, unknown serial lines, and connection loss.
• Safety: with wheels raised and supervision, measure stop latency and maximum movement duration, including communication failure.

Safety, security, and privacy

This prototype has no safety-rated controller, obstacle detection, authenticated command channel, encrypted local transport, or motor timeout. Initial trials should use raised wheels, a clear area, direct supervision, and a reachable power disconnect. These precautions do not imply certification.

Recognized speech text is sent to a hosted model provider. Operators should be informed and avoid speaking sensitive information. Keep API keys in ignored environment files and Wi-Fi credentials in ignored local headers. Keep the unauthenticated HTTP endpoint on a trusted private network; do not port-forward or expose it publicly.

Limitations

• Audio gating is heuristic and has not been evaluated in noise.
• Processing is sequential; provider and hardware latency are unmeasured.
• Only the action field receives a basic allowlist check; speech output is not fully bounded.
• The local action endpoint is unauthenticated and the bridge parser is minimal.
• Movement is not time-bounded; some model-allowed actions are unsupported.
• No camera, face recognition, obstacle sensing, display output, persistent memory, or empirical performance dataset is present in the analyzed runtime.
• Exact board and power-system models are not fully documented, and dependencies are not pinned.

Accordingly, claims in the earlier thesis about face detection, dynamic temperature, confidence-based filtering, non-blocking operation, robust recovery, and measured stability are not supported by the current source.

Recommendations and conclusion

1. Add an actuator-side movement timeout and independent physical emergency stop before autonomous-motion trials.
2. Synchronize the host, bridge, and Arduino action maps; add explicit acknowledgments.
3. Use robust request parsing and validate and bound every model response field.
4. Calibrate the speech gate or adopt a documented voice-activity detector and test it.
5. Add bounded retries, timeouts, recovery states, structured non-sensitive logs, and pinned dependency versions.
6. Run protocol and hardware-in-loop tests; publish measured results with setup and limitations.

AmViSH 2.0 is an integrated prototype joining local speech capture and recognition, hosted language-model interpretation, network transport, and embedded actuation. Its contribution is an inspectable end-to-end path and concrete system-integration lessons. The current source is narrower than earlier narrative claims: perception is speech-only, processing blocks, validation is partial, some action labels are unsupported, and movement has no bounded stop. The next stage should prioritize safety, acknowledgment, reproducibility, and measurement.

References

1. A. Radford et al., “Robust Speech Recognition via Large-Scale Weak Supervision,” 2022. arXiv:2212.04356.
2. Y. Kim et al., “A Survey on Integration of Large Language Models with Intelligent Robots,” 2024. arXiv:2404.09228.
3. Y. Ma et al., “A Survey on Vision-Language-Action Models for Embodied AI,” 2024. arXiv:2405.14093.