Library of actions: Implementing a generic robot execution framework by using manipulation action semantics


Robotics is a scientific and engineering discipline that is focused on the understanding and use of artificial, embodied capabilities. Contemporary research in this field aims at developing intelligent robotic systems with human-like skills. Basically, to perform an action with a robot, very often the action is parameterized and represented by a robotic-compatible encoding that allows execution. In particular, when a robot has to emulate an observed action sequence, it must first comprehend the inherent characteristic features of the individual actions. Such features need to replicate the semantics of the action with a high degree of invariance between unalike demonstrations of the same action. Simultaneously, the machine needs to be able to execute the action sequence in any appropriate situation.

At present, the most popular approach to achieve this is the programming by demonstration technique. Generally, the robot should be able to conceptually understand the required action sequence and comprehend the general constraints of the individual sub-actions. As of now, several frameworks exist that attempt action generalization and/or action conceptualization.

Nonetheless, a thorough review of published literature regarding the subject matter reveals that much needs to be done to improve on the existing library of actions representations. To this end, University of Göttingen researchers: Mohamad Javad Aein and Professor Florentin Wörgötter in collaboration with Dr. Eren Erdal Aksoy at  Halmstad University introduced a new library of actions, which entailed of a generic framework for executing manipulation actions on robotic systems by combining features that capture action semantics with a framework for execution. Their focus was on manipulation actions because they allow for a rather rigorous ontological structuring. Their work is currently published in The International Journal of Robotics Research.

Essentially, based on prior works from them and others, they engaged in the rigorous structuring of a large set of manipulation actions into a three-layer representation starting from a high, symbolic level via a state machine-like encoding and ending at detailed movement primitives. The research team was able to develop a library of manipulation actions that captured the essence of each action in an abstract way and yet remained compatible with robotic execution. Moreover, they created a generic representation consisting of symbolic and sub-symbolic components.

The authors were able to develop a framework from observing humans, a technique that provided them with a high degree of grounding. Remarkably, the researchers showed that the developed library of actions could be parameterized in a situation-dependent way to execute them either alone or in a sequence. Further, various tests on the presented framework yielded satisfactory execution performance on various atomic and chained manipulation actions.

In summary, the study presented a thorough definition of manipulation actions at symbolic (high) and sub-symbolic (low) levels and linked them through a midlevel finite state machine. Generally, the proposed state machine delivered a mechanism to execute the actions on a robotic arm/hand system. In a statement to Advances in Engineering, Professor Florentin Wörgötter, the lead author is hopeful the proposed library of actions could in the long run turn into a useful (and standardizable) robotics software tool.


Mohamad Javad Aein, Eren Erdal Aksoy, Florentin Wörgötter. Library of actions: Implementing a generic robot execution framework by using manipulation action semantics. The International Journal of Robotics Research 2019, Volume 38(8) page 910–934.

Go To The International Journal of Robotics Research 2019

Check Also