3D Human Motion Capture for Cooperative Construction Robotics

Project Description

This project develops a robust in‑the‑wild human motion capture system to enable safe and effective human‑robot collaboration in construction environments. By combining visual‑inertial SLAM for ego‑motion estimation with dense geometric and semantic mapping, the system provides essential context for interpreting human actions in dynamic, cluttered, and partially occluded scenes. A deep‑learning‑based pose estimation model fuses visual cues with a learned motion prior to predict full‑body human pose and uncertainty in 3D, even under severe occlusions and camera motion. The project also introduces a control framework for cooperative brick‑laying tasks, pairing human mortar application with robot‑executed pick‑and‑place routines. The recorded real‑world datasets demonstrate the feasibility of accurate pose tracking and lay the groundwork for predictive models that support safer and more autonomous collaboration in construction robotics.

Results

• Recorded a novel real‑world dataset of human motion during brick‑wall construction, capturing occlusions and complex interactions.
• Developed a deep‑learning human pose estimation framework that handles heavy occlusions, disentangles camera and human motion, and models full‑body 3D uncertainty.
• Achieved up to 47% lower reconstruction error, 20% faster runtime, and 76% lower memory usage compared to state‑of‑the‑art methods in unaligned evaluation.
• Implemented a cooperative control framework enabling a robot to perform brick pick‑and‑place actions based on human progress.
• Created a graph‑based model of the brick wall, integrating design and robotic trajectories, enabling repeatable and structured data collection.
• Demonstrated the feasibility of integrating human pose estimation into safety‑critical collaborative tasks in construction robotics.

Follow-up

Future work will focus on leveraging the collected datasets to train predictive models that anticipate human motion, enabling the robot to autonomously trigger task routines without manual intervention. Extending the current probabilistic model to multi‑modal distributions will improve robustness under long‑term occlusions and ambiguous human motion. Beyond single‑worker interactions, the framework can be expanded to support multi‑human collaboration, such as coordinated carrying of large elements or team‑based assembly tasks. Additionally, integrating semantic scene understanding with long‑horizon intent prediction could further enhance safety and task coordination. The system also serves as a foundation for related applications, such as automated disassembly or adaptive on‑site planning, where robust human motion tracking is critical. We have incorporated these aspects into the GNI project “SPAICR: Spatial AI for Cooperative Construction Robotics” and further developed human motion tracking within the ONE Munich Strategy Forum project: Next Generation Human-Centered Robotics.