Two-step long short-term memory method for identifying construction activities through positional and attentional cues


Construction of any civil and structural work involves both workers and equipment interacting with each other in a synchronized pattern to accomplish a certain task. Planning, delegating and coordinating various activities in the site among the various work groups is important as it enables the comprehension of jobsite context, which in turn allows the interpretation of worker intentions, the prediction of their movements, and the detection of inappropriate interactions that are counterproductive and may cause disastrous consequences such as struck-by accidents. Therefore, recognizing construction activities and involved working groups is critical to enhancing construction safety and improving productivity. At present, majority of the existing studies use videos that only contain one activity with involved entities and rely solely on the spatial-temporal relationship among entities. However, as mentioned earlier, many workers and machines co-exist and collaborate to accomplish different activities, and not all of them are relevant to the same activity, even though they are spatially close.

There is still a critical knowledge gap, i.e. methods are needed to identify working groups and recognize corresponding activities using images/videos that contain many entities collaborating on various tasks. On this account, researchers from the Lyles School of Civil Engineering at Purdue University: Jiannan Cai (Ph.D. candidate), Yuxi Zhang (Ph.D. student), led by Professor Hubo Cai, proposed a two-step deep learning-based classification approach – working group identification followed by activity recognition, leveraging both positional and attentional cues, to recognize complex interactions and their involved entities from videos that contain different activities with multiple entities. Their work is currently published in the research journal, Automation in Construction.

In their work, spatial and attentional states of individual entities were represented numerically, and the corresponding positional and attentional cues between two entities computed. The researchers then designed the long short-term memory (LSTM) networks so as to: first, classify whether two entities belonging to the same group, and second, recognize the activities they were involved in. In addition, the newly created method was validated using two sets of construction videos.

The authors reported that the proposed framework achieved over 95% accuracy in correctly identifying the working groups and recognizing the activities. In fact, the performance obtained by integrating positional and attentional cues was much higher than that obtained using positional cues alone. Better still, dividing the group activity recognition task into a two-step cascading process yielded better performance than simply conducting a one-step activity recognition.

In summary, the study by Professor Hubo Cai and his research team presented a novel two-step deep learning-based approach that integrates positional and attentional cues to identify construction working groups and recognize corresponding group activities. Also, by identifying working groups before recognizing activities, the group-irrelevant entities are effectively excluded, which leads to improved performance of group activity recognition. In a statement to Advances in Engineering, Professor Hubo Cai highlighted that the presented approach was nearly accurate in that by leveraging both positional and attentional cues, the accuracy was reported to increase from 85% to 95% compared with cases using positional cues alone, hence improved performance.

Two-step long short-term memory method for identifying construction activities through positional and attentional cues - Advances in Engineering

About the author

Jiannan Cai is a Ph.D. candidate at Lyles School of Civil Engineering with a concentration of Construction Engineering at Purdue University. She earned her Bachelor’s and Master’s degrees in Civil Engineering at Tongji University, China. Jiannan joined Purdue in 2016 as a recipient of Andrews Fellowship and is expected to graduate in May, 2020. While in Purdue, she joined the Computational Interdisciplinary Graduate Program (CIGP) with a specialization in Computational Science and Engineering. She was also chosen to attend the cyber carpentry workshop sponsored by NSF on life-cycle data management, cloud computing, and deep learning in 2019. Her research interests lie in automation and robotics in construction, cyber-physical system, and civil integrated management.

Her current researches are related to (1) creating novel methods that couple sensing, computer vision, and machine learning to achieve holistic situational awareness of dynamic and unstructured construction site for automation in construction, and (2) creating effective algorithms to analyze heterogenous data for decision making in infrastructure management. Jiannan’s long-term goal is to integrate innovative technologies, infrastructure/construction domain knowledge, and human factors in support of constructing and managing next-generation civil infrastructure.

About the author

Yuxi Zhang is a Ph.D. student in the Lyles School of Civil Engineering with a concentration of Construction Engineering at Purdue University. Her research interests are in knowledge management of construction engineering, computer vision and network analysis. She is currently conducting research to understand and optimize construction processes in a systematic perspective using advanced computing techniques and multi-sensory systems.

She has developed machine learning algorithms to recognize building symbols from two-dimensional drawings. She also has experience in the implementation of advanced computational technologies in engineering practices to assist state highway agencies to improve their efficiency and efficacy in infrastructure construction.

About the author

Dr. Hubo Cai is an Associate Professor of Civil and Construction Engineering at Purdue University. He is the founding director of the Laboratory of Computer-Integrated Infrastructure Informatics (LCIII) and the co-director of the Discrete Event Simulation (DES) Laboratory. Before joining Purdue University in 2009, Dr. Cai worked at North Carolina Department of Transportation (NCDOT), URS Corporation (now part of AE.COM), and Western Michigan University. Dr. Cai’s research is multi-pronged with research expertise in infrastructure informatics including multimode sensing, advanced computational algorithms and data analytics, and human-machine collaboration. Dr. Cai has a very well-established research record.

He has served as the Principal Investigator or Co-Principal Investigator on 12 research projects, totaling over 4 million, funded by Federal Highway Administration (FHWA), state transportation agencies such as Indiana Department of Transportation (INDOT), Michigan Department of Transportation (MDOT), and North Carolina Department of Transportation (NCDOT), and Nation Science Foundation (NSF). Dr. Cai has over 90 technical publications and has received many awards.

He has been serving as an Associate Editor for ASCE’s Journal of Computing in Civil Engineering (JCCE) since 2003, and he is currently serving on its editorial board. Dr. Cai is also the Chair of ASCE’s Data Sensing and Analysis (DSA) Committee. Dr. Cai holds a B.S. in Construction Management Engineering from Tongji University, and M.S. and Ph.D. degrees in Civil Engineering from North Carolina State University (NCSU).


Jiannan Cai, Yuxi Zhang, Hubo Cai. Two-step long short-term memory method for identifying construction activities through positional and attentional cues. Automation in Construction, volume 106 (2019) 102886.

Go To Automation in Construction

Check Also

Geodynamics of the central-western Mediterranean region: plausible and non-plausible driving forces