Research on Application of Electromagnetic Environment Data Warehouse Based on Big Data

Introduction

With the continuous improvement in the informatization of electromagnetic environment applications and the long-term operation of related businesses, various departments have accumulated vast amounts of electromagnetic environment data (Chen & Zhao, 2020). This data has played an important role in radio management, spectrum control, electronic warfare, and other areas (Cheng et al., 2019; Ding, 2015; Ha & Jia, 2015). However, there are also many challenges in terms of comprehensive storage and application of the data (Group, 2017):

• 1.

  Difficulty in storing massive raw data: Electromagnetic environment sensing devices have powerful data collection capabilities, with fast data output frequencies and diverse data types. Over long periods, these devices accumulate large amounts of data, and existing information systems are unable to meet the storage demands.

• 2.

  Large differences in data elements: Currently, China has been building electromagnetic environment sensing devices for different business applications. Since these devices are produced by different manufacturers and follow different data transmission protocols, the resulting electromagnetic environment data varies in source and structure.

• 3.

  Data silos: The existing electromagnetic environment data is non-integrated, with massive amounts of data stored independently in each business system. This leads to inefficiencies in data sharing and interoperability, and there is a lack of integrated data applications.

• 4.

  Low data value density: Short-term data is insufficient to reflect the state of the electromagnetic environment, and effective analysis can only be achieved using long-term, large-scale data. Therefore, developing a standardized solution with fast storage and highly efficient processing for electromagnetic environment data has become an urgent need to enhance the value of these data applications.

As a subject-oriented, integrated, stable, and time-variant collection of data used to support management decisions, a data warehouse enables the integration of multiple heterogeneous data sources. It can reorganize data according to themes to meet online analytical processing (OLAP) needs, supporting data mining and management decision-making (He et al., 2022). A data warehouse based on Oracle and its related components was constructed to address the storage and processing requirements of radio data, as detailed in relevant studies (Hu, 2017). This data warehouse design was well-suited for scenarios characterized by low data volume, structured data storage, and stable business needs. A solution based on Oracle database and data warehouse technologies had been proposed, specifically tailored for the efficient storage and processing of radio monitoring data (Imran et al., 2021). This solution integrates functionalities, such as clustering analysis, unknown signal prediction, and pattern mining, utilizing intelligent data mining techniques to improve the comprehensive coverage of signals. Furthermore, by incorporating OLAP analysis within a Browser/Server model, the system enables more intuitive and visual data presentation, offering robust support for decision-making. However, with the increasing variety and volume of monitoring data, relying on relational databases, such as Oracle, can result in excessive database load. Additionally, the table structure design may become suboptimal, posing challenges in effectively supporting diverse data retrieval and processing demands.