AVR-AKG: using virtual reality for domain knowledge generation from first-person demonstration

Manipulation tasks in specific environment, such as truss assembly in the space and "Hape" building blocks assembly in the toy room, unfold intentionally under corresponding domains, which contains a lot of domain knowledge. Being able to process contextual knowledge in these activities under corresponding domains over time can help us understand manipulation intentions. However, most researchers use methods of machine learning to make robots understand manipulation scenarios, which is a black model consuming a lot of computing resources. Moreover, the demonstration of assembly tasks in the real world is time-consuming and labor-intensive, and it is not suitable to initialize the assembly scene. To overcome these limitations, we introduced AVR-AKG: an implementing framework for domain knowledge generation that can generate dynamic knowledge graph in real time from assembly demonstrations. A combination of an Assembly Virtual Reality subsystem and an Assembly Knowledge Generation subsystem, in correspondence with assembly tasks and knowledge generation, is used to represent robot manipulation knowledge with Entity-Relation-Entity (E-R-E) and Entity-Attribute-Value (E-A-V) tuples. Using this framework, we propose a case study in which the demonstrator completes an assembly task using "Hape" building blocks, generating knowledge related to the operational context with knowledge graphs during first-person demonstration.