Table model data deletion #13878

jt2594838 · 2024-10-23T01:46:51Z

Upgrade the format of the Modification File to V2 (with the suffix ".mods2") to support table model data deletion;
Refactor memory structures to support table model data deletion;
During start-up, V1 Modification Files will be upgraded to V2 asynchronously.
Deletions to different Modification Files are performed in parallel.
Further abstractions are added to the mod file management to make future extensions easier.

Grammar definition:

Examples:

Assumed table schema in examples:
vehicle1(deviceType STRING ID, deviceId STRING ID, s0 INT32 MEASUREMENT, s1 INT64 MEASUREMENT)

Supported examples:

Unsupported examples:

Evaluation

Notice: compaction is disabled in all tests to avoid files being removed.

Write performance

We evaluate the time consumption of a single deletion over different numbers of files involved.
The experiment is performed by writing x TsFiles on a 1C1D instance
and then performing a deletion on all of them 5 times,
recording the average time consumption.

Below are the results

"Before" refers to the master branch (using the tree model),
while "after" is this branch (using the table model).

The results show that the time consumption is linear to the number of files, which is expected.
The parallel deletion optimization reduces the time consumption to around 1/3.
It is noticeable that both lines go up suddenly after a threshold,
which is about 2400 for "after" and 3800 for "before".
It is most possible that the cache of the CPU or the disk is overflowed and the IO efficiency is thus reduced.

Read performance

We also experiment on the effects of deletions on queries.
In this test, we write 100 TsFiles, each only with 100 points of 1 time series.
Then, after writing x deletions, each like
DELETE FROM test.table1 WHERE deviceId = 'd0',
we perfom 10 queries like
SELECT * FROM test.table1
and record the average query latency.

The following picture gives the results:

Similarly, "Before" refers to the master branch (using the tree model),
while "after" is this branch (using the table model).

As expected, the more deletions are written the slower queries will be,
and the query latency is about linear to the number of deletions.
Compared with "Before", the "After" line may reduce the query latency by about 40%,
due to the more compact binary format.

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/LoadTsFileAnalyzer.java

# Conflicts: # pom.xml

…ble_data_deletion

…ble_data_deletion # Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/modification/IDPredicate.java

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/LoadTsFileManager.java

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/InsertionCrossSpaceCompactionTask.java

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/CompactionUtils.java

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/mem/MTreeBelowSGMemoryImpl.java

* Update IoTDBTableIT.java * Update DataNodeInternalRPCServiceImpl.java

JackieTien97 · 2024-11-11T07:49:19Z

...ache/iotdb/db/pipe/extractor/dataregion/realtime/listener/PipeInsertionDataNodeListener.java

+    // TODO: implement
+    return null;


throw Exception may be better? @Caideyipi @SteveYurongSu can plan when to support this.

...re/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/node/WALNode.java

shuwenwei · 2024-11-13T02:04:51Z

...he/iotdb/db/storageengine/dataregion/compaction/execute/utils/MultiTsFileDeviceIterator.java

    if (tsFileResource.getStartTime(device) < timeLowerBoundForCurrentDevice) {
      ttlDeletion =
-          new Deletion(
+          new TreeDeletionEntry(


Should distinguish between tree models and table models for current device.

shuwenwei · 2024-11-13T02:53:39Z

.../apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/log/TsFileIdentifier.java

@@ -194,8 +195,9 @@ public File getFileFromDataDirsIfAnyAdjuvantFileExists() {
      File file = FSFactoryProducer.getFSFactory().getFile(dataDir, partialFileString);
      if (file.exists()
          || new File(file.getAbsolutePath() + TsFileResource.RESOURCE_SUFFIX).exists()
-          || new File(file.getAbsolutePath() + ModificationFile.FILE_SUFFIX).exists()
-          || new File(file.getAbsolutePath() + ModificationFile.COMPACTION_FILE_SUFFIX).exists()) {
+          || ModificationFileV1.getNormalMods(file).exists()


Also check the compaction mods file v1?

shuwenwei · 2024-11-13T03:09:11Z

...ache/iotdb/db/storageengine/dataregion/compaction/execute/recover/CompactionRecoverTask.java

-    if (!checkAndDeleteFile(file)) {
+    File resourceFile =
+        getFileFromDataDirs(tsFileIdentifier.getFilePath() + TsFileResource.RESOURCE_SUFFIX);
+    if (!checkAndDeleteFile(resourceFile)) {
      success = false;
    }



Also check old version mods files and compaction mods files?

shuwenwei · 2024-11-13T03:13:30Z

...e/iotdb/db/storageengine/dataregion/compaction/tablemodel/CompactionWithAllNullRowsTest.java

@@ -263,9 +263,9 @@ public void testCompactionWithAllDeletion() throws IOException, IllegalPathExcep
      writer.endFile();
    }
    resource1
-        .getModFile()
-        .write(new Deletion(new MeasurementPath(deviceID, ""), Long.MAX_VALUE, Long.MAX_VALUE));


May use TableDeletionEntry because the device is table model.

* drop column * Update IoTDBTableIT.java * Update IoTDBTableIT.java

…ble_data_deletion

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/TsFileProcessor.java

* adaptation * Fix * Update DeleteDevice.java * Update AnalyzeUtils.java * Update IoTDBDeviceIT.java

shuwenwei · 2024-11-15T03:13:18Z

...e/src/main/java/org/apache/iotdb/db/tools/validate/TsFileOverlapValidationAndRepairTool.java

@@ -129,10 +129,9 @@ private static void moveSeqResourceToUnsequenceDir(TsFileResource resource) thro
    moveFile(
        new File(tsfile.getAbsolutePath() + TsFileResource.RESOURCE_SUFFIX),
        new File(targetFile.getAbsolutePath() + TsFileResource.RESOURCE_SUFFIX));
-    if (resource.modFileExists()) {
+    if (resource.anyModFileExists()) {


If only shared mods file exists, it will also enter the following move method.

shuwenwei · 2024-11-15T03:38:32Z

...b/db/storageengine/dataregion/compaction/execute/performer/impl/FastCompactionPerformer.java

@@ -143,18 +143,17 @@ public void perform() throws Exception {
            x -> x.definitelyNotContains(device) || !x.isDeviceAlive(device, ttl));
        sortedSourceFiles.sort(Comparator.comparingLong(x -> x.getStartTime(device)));
        if (ttl != Long.MAX_VALUE) {
-          Deletion ttlDeletion =
-              new Deletion(
+          TreeDeletionEntry ttlDeletion =


Should distinguish between tree models and table models for current device.

sonarcloud · 2024-11-15T03:38:38Z

Quality Gate failed

Failed conditions
D Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

shuwenwei · 2024-11-15T03:51:25Z

...a/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/CompactionUtils.java

@@ -189,20 +200,17 @@ public static void addFilesToFileMetrics(TsFileResource resource) {
            resource.getTsFile().length(),
            resource.isSeq(),
            resource.getTsFile().getName());
-    if (resource.modFileExists()) {
+    if (resource.anyModFileExists()) {


For the shared mods file, is this redundant addition here?

shuwenwei · 2024-11-15T04:30:56Z

...ache/iotdb/db/storageengine/dataregion/compaction/execute/task/InnerSpaceCompactionTask.java

+    TsFileResource firstSource = filesView.sourceFilesInLog.get(0);
+    TsFileResource firstTarget = filesView.targetFilesInPerformer.get(0);
+    ModFileManagement modFileManagement = firstSource.getModFileManagement();
+    ModificationFile modificationFile = modFileManagement.allocateFor(firstTarget);


If a shared mods file has been created here and the resource file has not yet been serialized, can it be found during restart recovery?

I think if the resource file is not serialized, then the compaction is not completed, and the target file will be removed eventually.
So, it does not matter whether the shared mod file can be found in the recovery.

If the shared mod file can not be found in the recovery and the compaction task will rollback in the system recovery(kill -9 before the target resource been serialized), at this point, the compaction recover task may not be able to locate this shared mods file.

shuwenwei · 2024-11-15T06:21:40Z

...tanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/TsFileResource.java

+  private volatile ModificationFile exclusiveModFile;
+
+  private volatile ModificationFile sharedModFile;
+  private long shardModFileOffset;


shard -> shared

shuwenwei · 2024-11-15T06:25:12Z

...anode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/snapshot/SnapshotTaker.java

@@ -206,10 +206,10 @@ private boolean createSnapshot(List<TsFileResource> resources, String snapshotId
        createHardLink(
            new File(snapshotTsFile.getAbsolutePath() + TsFileResource.RESOURCE_SUFFIX),
            new File(tsFile.getAbsolutePath() + TsFileResource.RESOURCE_SUFFIX));
-        if (resource.getModFile().exists()) {
+        if (resource.exclusiveModFileExists()) {


Also copy the shared mods files if it does not exist in snapshot dir.

jt2594838 and others added 30 commits October 12, 2024 11:27

temp save

7dc39f1

temp save

469f3b6

Merge branch 'master' into table_data_deletion

f115dc6

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/LoadTsFileAnalyzer.java

Change modification format

480b4b1

Merge branch 'master' into table_data_deletion

6104265

# Conflicts: # pom.xml

update tsfile version

45e33a8

fix deviceId match

87baed0

Merge branch 'table_data_deletion' of github.com:apache/iotdb into ta…

2ec1aa4

…ble_data_deletion

fix identitySinOperatorTest

9bb80e3

refactor interface hierachy

1dacc0b

refactor package structure

acb1a47

spotless

088009b

support table deletion

05ba9c8

Merge branch 'table_data_deletion' of github.com:apache/iotdb into ta…

ba59a3f

…ble_data_deletion # Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/modification/IDPredicate.java

Merge branch 'master' into table_data_deletion

2ffd6b4

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/LoadTsFileManager.java

fix test

cb83826

remove v1 mod file

2dc417f

fix read empty mod

1a234ba

add table deletion IT

ffc3eba

Merge branch 'master' into table_data_deletion

655c84f

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/InsertionCrossSpaceCompactionTask.java

Fix nullability check in buildTsBlock().

ba01a6f

fix partialPath type in TreeDeletionEntry

bec444a

add predicate ut

d64b0a0

allow multiple mods in a plan node

0dd0c71

Merge branch 'master' into table_data_deletion

2798a66

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/CompactionUtils.java

implment deleteDataForDropTable & fix IT

bcf8ccb

fix log level

120acb3

Merge branch 'master' into table_data_deletion

d0bfa97

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/mem/MTreeBelowSGMemoryImpl.java

ignore one test

7d5c3a6

Added table IT (#13978)

9b511dc

* Update IoTDBTableIT.java * Update DataNodeInternalRPCServiceImpl.java

jt2594838 added 5 commits November 8, 2024 14:30

ignore perf test

884ec96

use buffered stream to read mods

00a44b5

fix ut

09ba21b

spotless

002d385

Merge branch 'master' into table_data_deletion

5faf169

jt2594838 marked this pull request as ready for review November 11, 2024 02:04

JackieTien97 requested changes Nov 11, 2024

View reviewed changes

jt2594838 added 3 commits November 11, 2024 18:24

add mod file exists marker and log condition

aaeb322

fix tests

4645ddc

fix test

6f185de

shuwenwei reviewed Nov 13, 2024

View reviewed changes

Caideyipi and others added 8 commits November 13, 2024 14:55

Drop column adaptation (#14073)

41e821c

* drop column * Update IoTDBTableIT.java * Update IoTDBTableIT.java

adaptation (#14077)

a87f0e7

fix comments

17f0a7f

Merge branch 'table_data_deletion' of github.com:apache/iotdb into ta…

d0e6e34

…ble_data_deletion

Merge branch 'master' into table_data_deletion

1fe88ff

# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/TsFileProcessor.java

Fixed the adaptation of delete device (#14081)

67c986b

* adaptation * Fix * Update DeleteDevice.java * Update AnalyzeUtils.java * Update IoTDBDeviceIT.java

Merge branch 'master' into table_data_deletion

a31ff27

update maxTime in TVList after deletion

fcf3f8b

shuwenwei reviewed Nov 15, 2024

View reviewed changes

fix comment

1a54935

shuwenwei reviewed Nov 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Table model data deletion #13878

Table model data deletion #13878

jt2594838 commented Oct 23, 2024 •

edited

Loading

JackieTien97 Nov 11, 2024

shuwenwei Nov 13, 2024

jt2594838 Nov 13, 2024

shuwenwei Nov 13, 2024

jt2594838 Nov 13, 2024

shuwenwei Nov 13, 2024

jt2594838 Nov 13, 2024

shuwenwei Nov 13, 2024

jt2594838 Nov 13, 2024

shuwenwei Nov 15, 2024

jt2594838 Nov 15, 2024

shuwenwei Nov 15, 2024

sonarcloud bot commented Nov 15, 2024

shuwenwei Nov 15, 2024

jt2594838 Nov 15, 2024

shuwenwei Nov 15, 2024

jt2594838 Nov 15, 2024

shuwenwei Nov 15, 2024 •

edited

Loading

shuwenwei Nov 15, 2024

shuwenwei Nov 15, 2024

Table model data deletion #13878

Are you sure you want to change the base?

Table model data deletion #13878

Conversation

jt2594838 commented Oct 23, 2024 • edited Loading

Grammar definition:

Examples:

Supported examples:

Unsupported examples:

Evaluation

Write performance

Read performance

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Nov 15, 2024

Quality Gate failed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shuwenwei Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jt2594838 commented Oct 23, 2024 •

edited

Loading

shuwenwei Nov 15, 2024 •

edited

Loading