ProcAgent: On-Device AI for Procedural Task Guidance
Researchers have developed ProcAgent, a cutting-edge, on-device assistant designed to help with tasks like assembling furniture and home repairs in environments where privacy and quick response times are crucial. This innovative system runs on a single NVIDIA Jetson AGX Orin and utilizes a unique propose-and-verify approach. It combines continuous low-latency perception with a symbolic task graph, on-demand vision-language verification, and an interaction agent powered by a large language model (LLM). ProcAgent actively monitors how users are progressing and only engages in intricate visual reasoning when it detects confusion or deviations, reducing the mental strain of following instructions. Unlike earlier multimodal assistants that rely on cloud computing, ProcAgent operates entirely on the device, making it perfect for home settings. This study is available on arXiv with ID 2607.24770.
Key facts
- ProcAgent is a fully on-device, agentic, vision-based procedural assistant.
- It runs on a single NVIDIA Jetson AGX Orin.
- It uses a propose-and-verify architecture.
- Components include continuous perception, symbolic task graph, on-demand vision-language verification, and LLM-based interaction agent.
- Designed for privacy-sensitive, latency-critical domestic settings like furniture assembly and home repair.
- Prior assistants rely on cloud inference and fixed always-on perception.
- Published on arXiv with ID 2607.24770.
- Announce type is new.
Entities
Institutions
- arXiv