Neural networks empowered: a machine learning-enabled, Gyro mmID for enhanced virtual reality and motion tracking applications

Abstract

With the emerging developments in millimeter-wave/5G technologies, the potential for wireless Internet of things devices to achieve widespread sensing, precise localization, and high data-rate communication systems becomes increasingly viable. The surge in interest surrounding virtual reality (VR) and augmented reality (AR) technologies is attributed to the vast array of applications they enable, ranging from surgical training to motion capture and daily interactions in VR spaces. To further elevate the user experience, and real-time and accurate orientation detection of the user, the authors proposes the utilization of a frequency-modulated continuous-wave (FMCW) radar system coupled with an ultra-low-power, sticker-like millimeter-wave identification (mmID). The mmID features four backscattering elements, multiplexed in amplitude, frequency, and spatial domains. This design utilizes the training of a supervised learning classification convolutional neural network, enabling accurate real-time three-axis orientation detection of the user. The proposed orientation detection system exhibits exceptional performance, achieving a noteworthy accuracy of 90.58% over three axes at a distance of 8 m. This high accuracy underscores the precision of the orientation detection system, particularly tailored for medium-range VR/AR applications. The integration of the FMCW-based mmID system with machine learning proves to be a promising advancement, contributing to the seamless and immersive interaction within virtual and augmented environments.

Introduction

Sensing is a fundamental requirement in systems dealing with human–computer and machine–machine interactions. The escalating demand for applications such as the Internet of things and digital twins highlight the pressing need to enhance sensing capabilities for systems that collect precise and accurate data [1]. Recent years have witnessed a surge in interest in millimeter-wave (mmWave) technologies, driven by factors such as the deployment of 5G networks and the widespread adoption of mobile computing, transforming our mobile devices into high-performance computers capable of rapid data processing. In the realm of sensing, mmWave technologies offer distinct advantages, including the potential to reduce system component size, achieve sub-millimeter accuracy in detecting spatial changes, and support very high data rate communications [2, 3]. As a result, mmWave systems emerge as a promising avenue for developing the next generation of ubiquitous sensing devices. Simultaneously, the technical advancements of recent years have facilitated the widespread integration of virtual reality (VR) and augmented reality (AR) into various domains, including healthcare, the automotive industry, and the metaverse [4]. The burgeoning interest in VR and AR underscores the importance of the ability to detect, localize, and determine the orientation of a target for providing an immersive user experience.

Localization, a crucial aspect in VR/AR applications, is commonly performed using radio-frequency identification (RFID) technology, known for its cost-effectiveness and ultra-low-power consumption [5]. The emergence of mmWave readers allows the extension of RFID technology to higher frequencies through the utilization of millimeter-wave identification (mmID) tags. When operating at mmWave frequencies, the path loss is larger due to higher absorption and greater sensitivity to obstacles such as buildings, resulting in greater signal attenuation [6]. However, by leveraging the bandwidth allocated by the Federal Communications Commission and operating at high frequencies, mmIDs-based systems inherently benefit from increased ranging accuracy and compact, wearable form-factors essential for VR/AR applications. Recent studies, employing a frequency-modulated continuous-wave (FMCW) reader [5, 7], have demonstrated highly accurate localization estimations. While location detection remains integral in VR/AR, the equally vital ability to detect the precise object orientation is emphasized, defined by the roll, yaw, and pitch axes of rotation in three-dimensional space. Previous works have explored the use of RFID tags for orientation tracking, demonstrating single-axis orientation detection at lower operational frequencies ranging from 1 to 4 GHz [8–10]. However, these systems are intrinsically limited by large size and limited reading range. Machine learning has shown promise in enhancing accuracy, as demonstrated in [11]. Notably, [12, 13] showcased the ability to detect orientation estimation in multiple axes using multiple tags, albeit limited by low operational frequency, large form factor, and low detection accuracy.

An earlier version of this paper was presented at the 2023 IEEE 20th European Radar Conference (EuRAD 2023) and was published in its proceedings [14]. Within that work, the authors presented a machine learning enhanced mmID system enabling highly accurate, $ \gt 90$%, three-axis orientation prediction at ranges up to 7 m. In this effort, the authors expand on this work by increasing the angular resolution of the system from 10^∘ to 5^∘ over the entire detection region and performing a dynamic evaluation of the system to highlight its robustness. Additionally, a classification convolutional neural network (CNN) is developed and employed to increase the orientation detection accuracy of the system, thereby addressing the limitations of existing systems.

Proposed low cost mmWave system

Architecture of ultra-low-power Gyro mmID tag

The mmID tag, designed for operation at 24.125 GHz, is comprised of two main sections: the RF front end and the baseband circuit. The RF front end design, similar to the approach presented in [11], was selected for a three-axes orientation-detecting mmID. This design features four distributed cross-polarized antenna elements, each incorporating a modulating loading constructed with a super low noise amplifier (LNA) field-effect transistor (FET)(CE3520K3 from CEL) and radial stubs. While the tracking of orientation necessitates only three elements, as demonstrated by [13], a fourth antenna was integrated for enhanced reliability and to facilitate finer amplitude encoding along the roll axis. Figure 1(a) illustrates the tag layout, where elements A-D exhibit modulating frequencies of 49 kHz, 69 kHz, 85 kHz, and 110 kHz, respectively. The chosen modulation frequencies are carefully selected to avoid harmonic interference, ensuring they remain below the third harmonic of each other. Furthermore, the reduction in modulation frequency contributes to the mmID tag’s low power consumption. Employing a cross-polarization configuration for each antenna minimizes interference from received signals to the reader, with each antenna designed to possess a polarization offset of 15^∘ from one another. This configuration enables the encoding of roll angle of rotation information based on the relative received amplitude of each element, while the relative phase difference between two elements remains minimal during rotation in the roll axis.

Figure 1. (a) Proof-of-concept 24 GHz Gyro mmID tag. (b) Diagram of rotational movements for each axis of the mmID. (c) Block diagram of transmitting and receiving chains of the FMCW radar utilized for the interrogation of the mmID tag.

To validate the operational frequency of the mmID, the normalized gain was captured, which is displayed in Fig. 2. Here it can be seen that tag has good agreement with the expected operational frequency. The design of the mmID is implemented on Rogers RO4350B (ϵ_r = 3.66, $\tan\delta$ = 0.0037), with a thickness of 0.51 mm, and antenna elements spaced at 6.5 mm, corresponding to ≈ $\frac{\lambda}{2}$. The baseband circuit generates the modulating signal for each antenna element. In this design, the resistor set voltage-controlled oscillator (VCO) LTC6906 is employed for its ability to control the frequency of the generated signal, as well as its low power consumption, ≈ 14.4 µW. To ensure consistent performance of the mmID, a 1.8 V voltage regulator is employed to provide stable voltage to the VCO. Powering of the system is achieved through the utilization of a 3 V coin cell battery, which contributes to the portability of the system.

Figure 2. Measured normalized gain vs frequency of the cross-polarized mmID.

Proof-of-concept FMCW radar system

The proof-of-concept (PoC) reader for the three-axis orientation detection Gyro tag was the Analog Devices EV-RADAR-MMIC2 Evaluation Board, operating at 24 GHz and utilizes FMCW radar technology. This evaluation board integrates the ADF5901 24 GHz monolithic microwave integrated circuit (MMIC) transmitter, ADF5904 24 GHz MMIC receiver, and the ADF4159 13 GHz phase-locked loop. Figure 1(c) shows the block diagram breakdown of the PoC reader. For both the transmitter and receiver, A-INFO LB-180400-20-C-KF horn antennas with 20 dBi gain were employed in a cross-polarized configuration. To enhance system sensitivity, a 40 dB LNA with a noise figure of 3.2 dB was integrated into the receiving antenna. The radar system was configured with a triangular chirp waveform, featuring a frequency slope of 400 MHz/µs and a chirp period of 5 ms. The complete chirp parameters of the PoC FMCW radar system can be found in Table 1.

Table 1. Chirp parameters of PoC FMCW radar

Signal processing framework

Extraction of Gyro mmID amplitude and phase response

Figure 3 a shows the block diagram of the proposed signal processing scheme. Utilizing the received signal for each corresponding chirp signal sent from the FMCW radar, a range fast Fourier transform (FFT) is applied. Commonly used in scenarios involving FMCW signals, the range FFT is an essential signal processing technique, as it allows for the precise extraction of a targets range and motion information [2]. This method works by transforming time-domain radar signals into the frequency domain through the application of the FFT algorithm. The resulting frequency spectrum provides a detailed representation of spectral characteristics, which can then be used in the identification of the targets distance. Peaks in the frequency spectrum correspond to distinct target distances, enhancing the spatial understanding of radar return signal. A sample spectrum is depicted in Fig. 4. This spectrum enables the extraction of modulating beat frequencies for each element on the tag. The negative and positive modulating peaks of each element can be observed, centered around their respective modulation frequencies.

Figure 3. Signal processing chain to extract amplitude and phase response of the tag and the classification CNN neural network used to predict the orientation angle of the mmID tag.

Figure 4. Range-FFT spectrum of the proof-of-concept mmID at broadside with the response of tag elements A-D highlighted.

After executing the range FFT, a custom peak detection algorithm is used to recognize the modulating beat frequencies associated with each element. Subsequently, the phase differences between elements B-A, C-B, D-C, and A-D can be computed. The significance of phase differences in received signals becomes particularly pronounced at extended reading distances. This is attributed to the diminishing signal strength of each element as the range from the reader increases, which results in a reduction in the dynamic range of the amplitude response, with respect to rotation [15]. Arctangent demodulation is a signal processing technique crucial in communication systems for extracting information encoded in modulated signals. The process involves taking the arctangent of the ratio between the in-phase and quadrature components of the modulated signal [16]. The in-phase and quadrature components represent the amplitude and phase information of the signal, respectively. By applying arctangent demodulation, phase information is unwraped, which leads to the extraction of the modulating signal. This demodulation technique is effective in scenarios where accurate phase information retrieval is essential, as it helps mitigate the impact of phase wrapping that can occur in conventional phase demodulation methods. Arctangent demodulation enhances the fidelity of signal demodulation, contributing to the recovery of information transmitted through frequency modulated radar signals.

Machine learning

In this study, a comprehensive performance evaluation was conducted for two machine learning algorithms, K-nearest neighbors (KNN) and CNN. KNN is a popular machine learning algorithm, commonly employed for classification and regression tasks. Operating on the principle of proximity, KNN makes predictions based on the majority class or average of the K-nearest data points in the feature space [17]. While KNN is versatile and widely applicable, its performance is influenced by the choice of the ‘K’ value, determining the number of neighbors considered.

Classification CNNs are a specialized deep learning architecture designed explicitly for the task of classifying input data into distinct categories. Typically used in image classification scenarios, these networks are adept at learning hierarchical representations of features within the data [17]. Comprised of convolutional layers for feature extraction, pooling layers for spatial down-sampling, and fully connected layers for classification, these networks automatically learn and discern intricate patterns, which are crucial for accurate classification. The convolutional layers utilize filters to convolve over input data, capturing local and global features, while pooling layers enhance computational efficiency by reducing spatial dimensions. The final fully connected layers employ learned features to make predictions and assign input data to predefined classes. This training process involves optimizing network parameters through backpropagation and gradient descent, ensuring the model generalizes well to unseen data. To determine the optimal hyper-parameters for the classification CNN, a random-search based optimization technique was applied [18]. The designed network is made up of a two hidden layers, each containing 10 neurons, with a rectified linear activation and softmax activation functions applied to layers 1 and 2, respectively.

Each model utilizes the phase difference from neighboring antennas, i.e. elements B-A, C-B, D-C, and A-D, along with the amplitude response from each element with respect to orientation as inputs. To standardize the dataset, each feature was scaled using min-max normalization. Additional information on the dataset used for these two models are discussed in more detail below in “Experimental validation of proposed system” section.

Experimental validation of proposed system

Measurement setup

The experimental setup, illustrated in Fig. 5, was designed to enable complete evaluation of the tag’s behavior across different orientations. To enable rotation along all three axes, the tag was affixed to a three-axis gimbal holder, with individual control exerted on each axis facilitated by planetary geared stepper motors, allowing for precise angular steps of 5^∘. The motors traversed a range of ±90^∘, resulting in 50,653 diverse orientations for the tag. This carefully chosen range ensures complete coverage within the angular space conducive to tag detection.

Figure 5. Experimental setup of the system at a distance of 10 m.

For the visualization and extraction of the transmitted and received signals, the Tektronix DPO 7354 Oscilloscope was employed. Operating at a sampling rate of 500 kHz, the oscilloscope recorded 19 up and down ramps for each angular configuration, totaling 38 observations and yielding 1,924,814 unique samples. Furthermore, a synchronized clock is used to ensure precise alignment between the radar and the oscilloscope, which is crucial for timing purposes to acquire the relative phase differentials between elements accurately. The extracted signals were then exported to MATLAB to allow for post-processing. The experimental parameters where specifically chosen to provide an accurate and robust understanding of the tag’s performance across diverse orientations.

Analysis of machine learning models

Experiments were conducted at 11 distinct distances (0.5 m, 1 m, 2 m, 3 m, 4 m, 5 m, 6 m, 7 m, 8 m, 9 m, and 10 m), forming a comprehensive dataset of 21,172,954 samples by aggregating data from each experiment. Given the dataset’s substantial size, an 80/20 train-test split was employed, with 80% of the data utilized for model training and the remaining 20% for assessing its performance.

The results, presented in Table 2, reveal that while the KNN demonstrated comparable accuracy to the CNN at distances up to 3 m, its accuracy significantly declined at longer distances. Conversely, the CNN achieved high accuracy ($ \gt 90$%) at distances up to 8 m. While the KNN demonstrated acceptable results in [14], this study faced challenges in replicating similar accuracy. The elevation of angular resolution from 10^∘ to 5^∘ resulted in increased similarity in amplitude and phase for each of the four antennas during tag rotation. Consequently, data for each angular configuration grouped closely together, which hinders the algorithm’s ability to distinguish different orientation configurations. Tailored for advanced feature extraction, the CNN excelled in capturing intricate patterns and spatial relationships in the phase difference and amplitude response of four antennas, contributing to their excellent performance in this study.

Table 2. Comparison of accuracy using K-nearest neighbors and classification CNN models

In Fig. 6, the confusion matrices for all three axes at a distance of 5 m are illustrated using the classification CNN. Each matrix is a composite representation of rotations, where one axis is held constant at a specific angle, and the remaining two axes rotate within ± 90^∘. For instance, the yaw confusion matrix at 90^∘ encompasses all rotations of pitch and roll while maintaining the yaw axis at 90^∘. Here, it can be seen that the model was able to achieve high accuracy along each axis. Additionally, this measurement was able to achieve a true positive rate of 93%, highlighting the models ability to correctly predict the orientation of the mmID. Notably, a discernible pattern emerges as false estimations manifest consistently when the angle exceeds ± 80^∘ across all three axes. This consistent trend is observable in other experiments and is likely attributed to the tag exceeding the beamwidth of the transmitting and receiving antennas, leading to the diminished accuracy.

Figure 6. Confusion matrices at 5 m: (a) roll axis fixed with yaw and pitch axes rotating, (b) yaw axis fixed with roll and pitch axes rotating, (c) pitch axis fixed with roll and yaw axes rotating.

Dynamic system evaluation

To evaluate the system’s robustness, a dynamic evaluation was conducted involving a sequence of experiments where the mmID traversed a predetermined path, as seen in Fig. 7. With the tag placed at a range of 5 m, seven experiments were executed, encompassing a range of rotation speeds from 0.25× to 1.75×, equivalent to 3.5–23.5^∘/sec, relative to the tag’s rotation speed used during the training of both machine learning models. The detailed results of these experiments are outlined in Table 3. An analysis of the results indicate that, although the accuracy of the CNN experiences a decline with increasing speed, it consistently maintains a high accuracy rate exceeding 92%. Additionally, the CNN model consistently outperforms the KNN model, consistent with the previously presented findings. These experiments not only highlights the system’s adaptability to dynamic conditions but also emphasizes the CNN model’s effectiveness in sustaining accurate orientation tracking, even under varying rotational dynamics.

Figure 7. Programmed path of the mmID for system evaluation.

Table 3. Comparison of accuracy using K-nearest neighbors and classification CNN for varying rotation speeds

Conclusion

In summary, the authors introduced a highly scalable, ultra-low power 24 GHz mmID tag for predicting its three-axis orientation at extended ranges by employing a classification CNN machine learning algorithm. The proposed system exhibited an accuracy exceeding 90% even at a range of 8 m from the PoC reader. An evaluation of the system was performed at various rotation speeds, in which the system was able to maintain high accuracy ($ \gt 90$%) throughout each experiment. Further evaluation of the proposed system will be explored including the dynamic orientation tracking of the mmID integrated with a moving target. In the context of VR and AR applications demanding low latency and devices with highly accurate orientation tracking, this cost-effective mmID tag emerges as a promising solution. Moreover, the system presents a unique opportunity for drone swarms applications, with the integration of multiple tags. With capabilities extending to immersive experiences and the mapping of AR objects into user interactions, this mmID tag stands out as a viable candidate for future systems. Additionally, the proposed PoC mmID sensor holds potential for advancing wireless motion capture systems. By integrating multiple mmID tags to provide accurate location and orientation information, the system offers a cost-effective alternative to traditional optical camera configurations, contributing to increased accessibility and efficiency in motion capture technology.