Intelligent Microwave MetasurfaceCamera for Smart-Home Sensing Applications

Microwave sensing is the cornerstone of various technological applications ranging from military surveillance to healthcare monitoring. It has the advantage of operating under all-weather and all-day environment which make it a powerful tool for non-invasive examination. However, conventional microwave sensing systems face limitations in their capacity to operate effectively across different scales and frequencies and typically constrained by the need to focus on a single frequency band which restricts their ability to simultaneously achieve wide-angle views and high-resolution imaging. Moreover, these traditional systems often require cooperative targets which limits their utility in real-world and dynamic scenarios where such cooperation is not feasible. In response to these challenges and to address the critical need for more versatile and adaptable EM sensing systems, a new study published in Journal of Advanced Optical Materials by Zhuo Wang, Hongrui Zhang, Hanting Zhao, Shengguo Hu and led by Professor Lianlin Li from the School of Electronics at Peking University together with Professor Tie Jun Cui from Southeast University, the researchers developed a system, a novel intelligent microwave camera, that could overcome the inherent trade-offs between resolution, field of view, and obstacle penetration that plague traditional systems. Using the emerging technology of reprogrammable metasurfaces (RM), they created a multi-task, multi-scale intelligent sensing system capable of adapting to different sensing tasks and environments.

The research team used the 2.4 GHz  module to reconstruct a three-dimensional (3D) human skeleton in real-time. They specifically used this subsystem because it has broad view angle and strong obstacle penetration capability which makes it suitable for monitoring overall human movement within an indoor setting. The researchers trained a skeleton-reconstruction-oriented deep artificial neural network (SR-ANN) using over 80,000 pairs of labeled training samples obtained from a synchronized binocular optical camera, ZED2. The trained network was then used to process the microwave data captured by the 2.4 GHz module. The authors found the system to be able to detect and reconstruct the 3D skeleton of a human subject with an accuracy of less than 3 cm. This level of precision was maintained at a frame rate of 20 Hz, demonstrating the system’s capability for real-time monitoring. The reconstructed skeletons were able to accurately reflect various human postures and movements, such as standing, sitting, and raising a hand, thus validating the system’s effectiveness in coarse-scale human behavior recognition.

The researchers’ experiments then focused on trying to recognize human hand gestures using the higher frequency subsystem of 5.5 GHz  module. They chose the system because it can capture finer details such as hand movements which are crucial for applications in human-machine interfaces. The experiment involved four distinct hand gestures—left-right wave, drawing a circle, back-forth wave, and tighten fist which were performed by participants at different locations. They employed a Long Short-Term Memory-based artificial neural network to classify these gestures based on the time-frequency spectrum of the captured signals. The authors’ findings revealed that the system achieved a recognition accuracy of over 90%, even when the gestures were performed at varying locations within the environment. According to the authors, this high level of accuracy was attributed to the system’s ability to wirelessly lock onto the hand of interest focusing the electromagnetic wavefield and minimizing interference from other body parts or environmental noise. Moreover, the use of the normalized coherent time-frequency processing algorithm further enhanced the system’s robustness because it mitigated the effects of environmental variability.  Afterward, the authors explored the system’s capability to monitor vital signs, specifically respiration and heartbeat using the 9.7 GHz  module. They selected high-frequency band for its sensitivity to subtle physiological signals which are critical for applications in at-home healthcare. Their experiment involved subjects sitting or standing still while the system measured their vital signs using the micro-Doppler effect which captures the minute vibrations of the chest due to breathing and heartbeats. They demonstrated that the 9.7 GHz module accurately detected respiration and heartbeat rates with minimal error compared to a commercial contact bioelectric sensor used as a benchmark. The errors in measurement were within 0.02 Hz for respiration and 0.15 Hz for heartbeat, indicating a high level of precision. The system’s ability to wirelessly lock onto the subject’s chest area and focus the EM wavefield was important in achieving these results because it significantly reduced interference from other parts of the body and environmental factors and they used the variational mode decomposition algorithm to refine the data and isolate the relevant physiological signals from potential noise. In the final experiment, the researchers tested the system’s performance in continuously monitoring a target moving freely within an indoor environment. The experiment combined the capabilities of all three modules (2.4 GHz, 5.5 GHz, and 9.7 GHz) to track overall movements, recognize specific gestures, and monitor vital signs simultaneously. They captured data in real-time and displayed through a custom Python-based software interface which showed the system seamlessly transition between different sensing tasks based on the detected behavior of the subject. For instance, when the subject performed a hand gesture, the system automatically activated the 5.5 GHz module to recognize the gesture, while it switched to the 9.7 GHz module to monitor vital signs when the subject was stationary. These results highlight the new system’s adaptability and robustness in a dynamic setting and the switch between different sensing modes and frequencies allowed the system to provide comprehensive monitoring of the subject’s actions and health status without interruption which demonstrate its potential for practical applications in real world situations.

In conclusion, Professor Lianlin Li and colleagues successfully developed a new multi-task, multi-scale intelligent sensing system capable of adapting to various real-world scenarios which resolved the limitations of traditional EM sensing systems. The application of the reported system is far-reaching. Firstly, the system’s ability to operate across multiple frequencies makes it of high value for healthcare applications where continuous, non-invasive monitoring of vital signs and human behavior is essential. It could indeed revolutionize at-home healthcare and enable real-time tracking of patients’ health status without the need for wearable devices. Moreover, the innovation has significant implications for the Internet of Things because the new system’s capacity to monitor large areas and adapt to different tasks makes it ideal for smart home applications, security systems, and digital twins which can offer a new level of interaction between humans and machines and enable more responsive environments. Furthermore, the authors’ findings open new avenues for the development of intelligent sensing systems in complex environments, such as urban surveillance, disaster response, and autonomous vehicles, where real-time, multi-scale sensing is vital and the ability to switch between different tasks and scales of observation with high precision can ensure that systems operate effectively under all conditions.