Apache IoTDB
Apache IoTDB is a column-oriented open-source, time-series database management system written in Java. It has both edge and cloud versions, provides an optimized columnar file format for efficient time-series data storage, and TSDB with high ingestion rate, low latency queries and data analysis support. It is specially optimized for time-series oriented operations like aggregations query, downsampling and sub-sequence similarity search. The name IoTDB comes from Internet of Things Database, which means it was designed as an IoT-native TSDB that resolves the pain points of the typical IoT scenarios, including massive data generation, high frequency sampling, out-of-order data, specific analytics requirements, high costs of storage and operation & maintenance, low computational power of IoT devices.
History
Apache IoTDB is a project initiated by Prof. Jianmin Wang's team in the School of Software at Tsinghua University. In 2011, the team chose to use open source NoSQL technology instead of Oracle for a project with mass machine data management, and noticed the insufficiency of NoSQL in the industrial internet of things scenarios. The team started to develop a data management system and formally proposed TsFile, an optimized columnar compact file storage format for time series data, in March 2016. The source code was then opened on GitHub.In June 2016, based on TsFile, the team began to develop IoTDB, an IIoT database supporting real-time reading & writing and analysis.
In November 2018, the project IoTDB entered incubator at the Apache Software Foundation.
On September 16, 2020, the ASF officially issued a resolution to promote Apache IoTDB to the global Top-Level Project following a public discussion vote by the community and a show of hands vote by the board.
Architecture
The complete storage system of Apache IoTDB follows a client-server architecture, including IoTDB engine and several components as IoTDB suite. IoTDB suite can provide a series of functions in the real situation such as data collection, data writing, data storage, data query, data visualization and data analysis. This allows data collected by the sensor to constantly persist in server, where the data can then be used for native query or shipped to other open-source platforms for data analysis. In particular, IoTDB provides a mode called "Edge-Cloud Cooperation", which can synchronize data collected at every user-configured interval from one IoTDB instance to another using Sync Tool.Users can use JDBC to write time series data to local/remote IoTDB. This time series data may represent system state data, message queue data, time series data from applications, or other time series data in the database. The data can be directly written to TsFile locally or on Hadoop Distributed File System.
TsFile is a column storage file format developed for accessing, compressing and storing time series data in Apache IoTDB. Its structure is based on LSM-Tree, which reduces the computational resources and optimizes the performance of Apache IoTDB.
TsFile could be written to the HDFS, thereby implementing data processing tasks such as abnormality detection and machine learning on the Hadoop or Spark data processing platform.
For the data written to HDFS or local TsFile, users can use TsFile-Hadoop-Connector or TsFile-Spark-Connector to allow Hadoop or Spark to process data. The results of the analysis can be written back to TsFile in the same way. Also, IoTDB and TsFile provide client tools to meet the various needs of users in writing and viewing data in SQL form, script form and graphical form.