OpenSearch Index: Failed to migrate into Cold Storage for Application Pipeline

[YikaiHu]

Describe the bug

Failed to migrate into Cold Storage for Application Pipeline.
Due to the absence of the "@timestamp" alias within the created App Index template, Index State Manager is unable to locate the correct timestamp.

Expected Behavior

Migrate the index to cold storage successfully.

Current Behavior

There will be an error message:

Failed to migrate index xxxx to cold storage because of invalid timestamp_field [@timestamp]

Reproduction Steps

Enable cold migeration.

Possible Solution

No response

Additional Information/Context

No response

Solution Version

v2.0.0

AWS Region. e.g., us-east-1

No response

Other information

No response

[YikaiHu]

经过排查，目前Service Pipeline没有出现该问题，迁移正常

目前我们的App Pipeline 的index template 如下:

GET _index_template/spring-dev-01-template

{
  "index_templates" : [
    {
      "name" : "spring-dev-01-template",
      "index_template" : {
        "index_patterns" : [
          "spring-dev-01-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "codec" : "best_compression",
              "refresh_interval" : "1s",
              "number_of_shards" : "1",
              "number_of_replicas" : "1",
              "plugins" : {
                "index_state_management" : {
                  "rollover_alias" : "spring-dev-01"
                }
              }
            }
          },
          "mappings" : {
            "properties" : {
              "level" : {
                "type" : "keyword"
              },
              "logger" : {
                "type" : "text"
              },
              "time" : {
                "type" : "date"
              },
              "thread" : {
                "type" : "text"
              },
              "message" : {
                "type" : "text"
              }
            }
          }
        },
        "composed_of" : [ ]
      }
    }
  ]
}

我们的Service Pipeline

GET _index_template/centralclwebacl-dev-01-waf-template

可以看到我们有 ，从而是这个导致了问题

              "@timestamp" : {
                "path" : "timestamp",
                "type" : "alias"
              },

{
  "index_templates" : [
    {
      "name" : "centralclwebacl-dev-01-waf-template",
      "index_template" : {
        "index_patterns" : [
          "centralclwebacl-dev-01-waf-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "codec" : "best_compression",
              "mapping" : {
                "total_fields" : {
                  "limit" : "1000"
                },
                "ignore_malformed" : "false"
              },
              "refresh_interval" : "1s",
              "number_of_shards" : "1",
              "number_of_replicas" : "1",
              "plugins" : {
                "index_state_management" : {
                  "rollover_alias" : "centralclwebacl-dev-01-waf"
                }
              }
            }
          },
          "mappings" : {
            "properties" : {
              "terminatingRuleId" : {
                "type" : "keyword"
              },
              "terminatingRuleType" : {
                "type" : "keyword"
              },
              "ruleGroupList" : {
                "properties" : {
                  "terminatingRule" : {
                    "properties" : {
                      "action" : {
                        "type" : "keyword"
                      },
                      "ruleId" : {
                        "type" : "keyword"
                      }
                    }
                  },
                  "ruleGroupId" : {
                    "type" : "keyword"
                  }
                }
              },
              "httpSourceId" : {
                "type" : "keyword"
              },
              "labels" : {
                "properties" : {
                  "name" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "ignore_above" : 256,
                        "type" : "keyword"
                      }
                    }
                  }
                }
              },
              "webaclId" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                }
              },
              "@timestamp" : {
                "path" : "timestamp",
                "type" : "alias"
              },
              "webaclName" : {
                "type" : "keyword"
              },
              "action" : {
                "type" : "keyword"
              },
              "httpRequest" : {
                "properties" : {
                  "args" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "ignore_above" : 256,
                        "type" : "keyword"
                      }
                    }
                  },
                  "country" : {
                    "type" : "keyword"
                  },
                  "headers" : {
                    "properties" : {
                      "name" : {
                        "type" : "keyword"
                      },
                      "value" : {
                        "type" : "text",
                        "fields" : {
                          "keyword" : {
                            "ignore_above" : 256,
                            "type" : "keyword"
                          }
                        }
                      }
                    }
                  },
                  "httpVersion" : {
                    "type" : "keyword"
                  },
                  "requestId" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "ignore_above" : 256,
                        "type" : "keyword"
                      }
                    }
                  },
                  "clientIp" : {
                    "type" : "ip"
                  },
                  "httpMethod" : {
                    "type" : "keyword"
                  },
                  "uri" : {
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "ignore_above" : 256,
                        "type" : "keyword"
                      }
                    }
                  }
                }
              },
              "httpSourceName" : {
                "type" : "keyword"
              },
              "formatVersion" : {
                "type" : "keyword"
              },
              "timestamp" : {
                "format" : "epoch_millis",
                "type" : "date"
              }
            }
          },
          "aliases" : { }
        },
        "composed_of" : [ ]
      }
    }
  ]
}

[JoeShi]

This is not a bug. If the log has timestamp field configured, it should be work as expected. The root cause is that the log do not have timestamp field