[BUG] RestStatus.CONFLICT when updating the state index with two steps concurrently

[yuye-aws]

What is the bug?

Users may not be able to provision the text-to-visualization-claude.json sample template.

How can one reproduce the bug?

Provision the flow framework for text to visualization agents. Please remember to

POST /_plugins/_flow_framework/workflow?provision=true
{
  "name": "Text to visualization agents",
  "description": "This template is to create all Agents required for text to visualization",
  "use_case": "REGISTER_AGENTS",
  "version": {
    "template": "1.0.0",
    "compatibility": [
      "2.18.0",
      "3.0.0"
    ]
  },
  "workflows": {
    "provision": {
      "user_params": {},
      "nodes": [
        {
          "id": "create_claude_connector",
          "type": "create_connector",
          "previous_node_inputs": {},
          "user_inputs": {
            "credential": {
              "access_key": "...",
              "secret_key": "..."
            },
            "parameters": {
              "endpoint": "bedrock-runtime.us-east-1.amazonaws.com",
              "content_type": "application/json",
              "auth": "Sig_V4",
              "max_tokens_to_sample": "8000",
              "service_name": "bedrock",
              "temperature": "0.0000",
              "response_filter": "$.content[0].text",
              "region": "us-east-1",
              "anthropic_version": "bedrock-2023-05-31"
            },
            "version": "1",
            "name": "Claude haiku runtime Connector",
            "protocol": "aws_sigv4",
            "description": "The connector to BedRock service for claude model",
            "actions": [
              {
                "action_type": "predict",
                "method": "POST",
                "url": "https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-3-haiku-20240307-v1:0/invoke",
                "headers": {
                  "content-type": "application/json",
                  "x-amz-content-sha256": "required"
                },
                "request_body": "{\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"${parameters.prompt}\"}]}],\"anthropic_version\":\"${parameters.anthropic_version}\",\"max_tokens\":${parameters.max_tokens_to_sample}}"
              }
            ]
          }
        },
        {
          "id": "register_claude_model",
          "type": "register_remote_model",
          "previous_node_inputs": {
            "create_claude_connector": "connector_id"
          },
          "user_inputs": {
            "name": "claude-haiku",
            "description": "Claude model",
            "deploy": true
          }
        },
        {
          "id": "create_t2vega_tool",
          "type": "create_tool",
          "previous_node_inputs": {
            "register_claude_model": "model_id"
          },
          "user_inputs": {
            "parameters": {
              "prompt": "You're an expert at creating vega-lite visualization. No matter what the user asks, you should reply with a valid vega-lite specification in json.\nYour task is to generate Vega-Lite specification in json based on the given sample data, the schema of the data, the PPL query to get the data and the user's input.\nLet's start from dimension and metric/date. Now I have a question, I already transfer it to PPL and query my Opensearch cluster. \nThen I get data. For the PPL, it will do aggregation like \"stats AVG(field_1) as avg, COUNT(field_2) by field_3, field_4, field_5\". \nIn this aggregation, the metric is [avg, COUNT(field_2)] , and then we judge the type of field_3,4,5. If only field_5 is type related to date, the dimension is [field_3, field_4], and date is [field_5]\nFor example, stats SUM(bytes) by span(timestamp, 1w), machine.os, response, then SUM(bytes) is metric and span(timestamp, 1w) is date, while machine.os, response are dimensions.\nNotice: Some fields like 'span()....' will be the date, but not metric and dimension. \nAnd one field will only count once in dimension count. You should always pick field name from schema\nTo summarize,\nA dimension is a categorical variable that is used to group, segment, or categorize data. It is typically a qualitative attribute that provides context for metrics and is used to slice and dice data to see how different categories perform in relation to each other.\nThe dimension is not date related fields. The dimension and date are very closed. The only difference is date is related to datetime, while dimension is not.\nA metric is a quantitative measure used to quantify or calculate some aspect of the data. Metrics are numerical and typically represent aggregated values like sums, averages, counts, or other statistical calculations.\n\nIf a ppl doesn't have aggregation using 'stats', then each field in output is dimension.\nOtherwise, if a ppl use aggregation using 'stats' but doesn't group by using 'by', then each field in output is metric.\n\nThen for each given PPL, you could give the metric and dimension and date. One field will in only one of the metric, dimension or date.\n\nThen according to the metric number and dimension number of PPL result, you should first format the entrance code by metric_number, dimension_number, and date_number. For example, if metric_number = 1, dimension_number = 2, date_number=1, then the entrance code is  121.\nI define several use case categories here according to the entrance code.\nFor each category, I will define the entrance condition (number of metric and dimension)\nI will also give some defined attribute of generated vega-lite. Please refer to it to generate vega-lite.\n\nType 1:\nEntrance code: <1, 1, 0>\nDefined Attributes:\n      {\n      \"title\": \"<title>\",\n      \"description\": \"<description>\",\n      \"mark\": \"bar\",\n      \"encoding\": {\n        \"x\": {\n          \"field\": \"<metric name>\",\n          \"type\": \"quantitative\"\n        },\n        \"y\": {\n          \"field\": \"<dimension name>\",\n          \"type\": \"nominal\"\n        }\n      },\n    }\n\nType 2:\nEntrance code: <1, 2, 0>\nDefined Attributes:\n{\n      \"mark\": \"bar\",\n      \"encoding\": {\n        \"x\": {\n          \"field\": \"<metric 1>\",\n          \"type\": \"quantitative\"\n        },\n        \"y\": {\n          \"field\": \"<dimension 1>\",\n          \"type\": \"nominal\"\n        },\n        \"color\": {\n          \"field\": \"<dimension 2>\",\n          \"type\": \"nominal\"\n        }\n      }\n    }\n\n\nType 3\nEntrance code: <3, 1, 0>\nDefined Attributes:\n{\n    \"mark\": \"point\",\n    \"encoding\": {\n        \"x\": {\n            \"field\": \"<metric 1>\",\n            \"type\": \"quantitative\"\n        },\n        \"y\": {\n            \"field\": \"<metric 2>\",\n            \"type\": \"quantitative\"\n        },\n        \"size\": {\n            \"field\": \"<metric 3>\",\n            \"type\": \"quantitative\"\n        },\n        \"color\": {\n            \"field\": \"<dimension 1>\",\n            \"type\": \"nominal\"\n        }\n    }\n}\n\nType 4\nEntrance code: <2, 1, 0>\nDefined Attributes:\n{\n    \"mark\": \"point\",\n    \"encoding\": {\n        \"x\": {\n            \"field\": \"<mtric 1>\",\n            \"type\": \"quantitative\"\n        },\n        \"y\": {\n            \"field\": \"<metric 2>\",\n            \"type\": \"quantitative\"\n        },\n        \"color\": {\n            \"field\": \"<dimension 1>\",\n            \"type\": \"nominal\"\n        }\n    }\n}\n\nType 5:\nEntrance code: <2, 1, 1>\nDefined Attributes:\n{\n      \"layer\": [\n        {\n          \"mark\": \"bar\",\n          \"encoding\": {\n            \"x\": {\n              \"field\": \"<date 1>\",\n              \"type\": \"temporal\"\n            },\n            \"y\": {\n              \"field\": \"<metric 1>\",\n              \"type\": \"quantitative\",\n              \"axis\": {\n                \"title\": \"<metric 1 name>\"\n              }\n            },\n            \"color\": {\n              \"field\": \"<dimension 1>\",\n              \"type\": \"nominal\"\n            }\n          }\n        },\n        {\n          \"mark\": {\n            \"type\": \"line\",\n            \"color\": \"red\"\n          },\n          \"encoding\": {\n            \"x\": {\n              \"field\": \"<date 1>\",\n              \"type\": \"temporal\"\n            },\n            \"y\": {\n              \"field\": \"<metric 2>\",\n              \"type\": \"quantitative\",\n              \"axis\": {\n                \"title\": \"<metric 2 name>\",\n                \"orient\": \"right\"\n              }\n            },\n            \"color\": {\n              \"field\": \"<dimension 1>\",\n              \"type\": \"nominal\"\n            }\n          }\n        }\n      ],\n      \"resolve\": {\n        \"scale\": {\n          \"y\": \"independent\"\n        }\n      }\n    }\n\nType 6:\nEntrance code: <2, 0, 1>\nDefined Attributes:\n{\n      \"title\": \"<title>\",\n      \"description\": \"<description>\",\n      \"layer\": [\n        {\n          \"mark\": \"area\",\n          \"encoding\": {\n            \"x\": {\n              \"field\": \"<date 1>\",\n              \"type\": \"temporal\"\n            },\n            \"y\": {\n              \"field\": \"<metric 1>\",\n              \"type\": \"quantitative\",\n              \"axis\": {\n                \"title\": \"<metric 1 name>\"\n              }\n            }\n          }\n        },\n        {\n          \"mark\": {\n            \"type\": \"line\",\n            \"color\": \"black\"\n          },\n          \"encoding\": {\n            \"x\": {\n              \"field\": \"date\",\n              \"type\": \"temporal\"\n            },\n            \"y\": {\n              \"field\": \"metric 2\",\n              \"type\": \"quantitative\",\n              \"axis\": {\n                \"title\": \"<metric 2 name>\",\n                \"orient\": \"right\"\n              }\n            }\n          }\n        }\n      ],\n      \"resolve\": {\n        \"scale\": {\n          \"y\": \"independent\"\n        }\n      }\n    }\n    \nType 7:\nEntrance code: <1, 0, 1>\nDefined Attributes:\n{\n      \"title\": \"<title>\",\n      \"description\": \"<description>\",\n      \"mark\": \"line\",\n      \"encoding\": {\n        \"x\": {\n          \"field\": \"<date 1>\",\n          \"type\": \"temporal\",\n          \"axis\": {\n            \"title\": \"<date name>\"\n          }\n        },\n        \"y\": {\n          \"field\": \"<metric 1>\",\n          \"type\": \"quantitative\",\n          \"axis\": {\n            \"title\": \"<metric name>\"\n          }\n        }\n      }\n    }\n\nType 8:\nEntrance code: <1, 1, 1>\nDefined Attributes:\n{\n      \"title\": \"<title>\",\n      \"description\": \"<description>\",\n      \"mark\": \"line\",\n      \"encoding\": {\n        \"x\": {\n          \"field\": \"<date 1>\",\n          \"type\": \"temporal\",\n          \"axis\": {\n            \"title\": \"<date name>\"\n          }\n        },\n        \"y\": {\n          \"field\": \"<metric 1>\",\n          \"type\": \"quantitative\",\n          \"axis\": {\n            \"title\": \"<metric name>\"\n          }\n        },\n        \"color\": {\n          \"field\": \"<dimension 1>\",\n          \"type\": \"nominal\",\n          \"legend\": {\n            \"title\": \"<dimension name>\"\n          }\n        }\n      }\n    }\n\nType 9:\nEntrance code: <1, 2, 1>\nDefined Attributes:\n{\n      \"title\": \"<title>\",\n      \"description\": \"<description>\",\n      \"mark\": \"line\",\n      \"encoding\": {\n        \"x\": {\n          \"field\": \"<date 1>\",\n          \"type\": \"temporal\",\n          \"axis\": {\n            \"title\": \"<date name>\"\n          }\n        },\n        \"y\": {\n          \"field\": \"<metric 1>\",\n          \"type\": \"quantitative\",\n          \"axis\": {\n            \"title\": \"<metric 1>\"\n          }\n        },\n        \"color\": {\n          \"field\": \"<dimension 1>\",\n          \"type\": \"nominal\",\n          \"legend\": {\n            \"title\": \"<dimension 1>\"\n          }\n        },\n        \"facet\": {\n          \"field\": \"<dimension 2>\",\n          \"type\": \"nominal\",\n          \"columns\": 2\n        }\n      }\n    }\n\nType 10:\nEntrance code: all other code\nAll others type.\nUse a table to show the result\n\n\nBesides, here are some requirements:\n1. Do not contain the key called 'data' in vega-lite specification.\n2. If mark.type = point and shape.field is a field of the data, the definition of the shape should be inside the root \"encoding\" object, NOT in the \"mark\" object, for example, {\"encoding\": {\"shape\": {\"field\": \"field_name\"}}}\n3. Please also generate title and description\n\nThe sample data in json format:\n${parameters.sampleData}\n\nThis is the schema of the data:\n${parameters.dataSchema}\n\nThe user used this PPL query to get the data: ${parameters.ppl}\n\nThe user's question is: ${parameters.input_question}\n\nNotice: Some fields like 'span()....' will be the date, but not metric and dimension. \nAnd one field will only count once in dimension count.  You should always pick field name from schema.\n And when you code is <2, 1, 0>, it belongs type 4.\n  And when you code is <1, 2, 0>, it belongs type 9.\n\n\nNow please reply a valid vega-lite specification in json based on above instructions.\nPlease return the number of dimension, metric and date. Then choose the type. \nPlease also return the type.\nFinally return the vega-lite specification according to the type.\nPlease make sure all the key in the schema matches the word I given. \nYour answer format should be:\nNumber of metrics:[list the metric name here, Don't use duplicate name]  <number of metrics {a}>  \nNumber of dimensions:[list the dimension name here]  <number of dimension {b}> \nNumber of dates:[list the date name here]  <number of dates {c}> \nThen format the entrance code by: <Number of metrics, Number of dimensions, Number of dates>\nType and its entrance code: <type number>: <its entrance code>\nThen apply the vega-lite requirements of the type.\n<vega-lite> {here is the vega-lite json} </vega-lite>\n\nAnd don't use 'transformer' in your vega-lite and wrap your vega-lite json in <vega-lite> </vega-lite> tags\n"
            },
            "name": "Text2Vega",
            "type": "MLModelTool"
          }
        },
        {
          "id": "create_instruction_based_t2vega_tool",
          "type": "create_tool",
          "previous_node_inputs": {
            "register_claude_model": "model_id"
          },
          "user_inputs": {
            "parameters": {
              "prompt": "You're an expert at creating vega-lite visualization. No matter what the user asks, you should reply with a valid vega-lite specification in json.\nYour task is to generate Vega-Lite specification in json based on the given sample data, the schema of the data, the PPL query to get the data and the user's input.\nNow I will give you some examples about how to create vega-lite\nSimple description:\nA bar chart encodes quantitative values as the extent of rectangular bars.\nresult vega-lite\n{'mark': 'bar', 'encoding': {'x': {'field': 'X', 'type': 'nominal'}, 'y': {'field': 'Y', 'type': 'quantitative'}}}\nSimple description:\nA bar chart showing the US population distribution of age groups in 2000.\nresult vega-lite\n{'mark': 'bar', 'encoding': {'x': {'aggregate': 'sum', 'field': 'X'}, 'y': {'field': 'Y'}}}\nSimple description:\nA bar chart that sorts the y-values by the x-values\nresult vega-lite\n{'mark': 'bar', 'encoding': {'x': {'aggregate': 'sum', 'field': 'X'}, 'y': {'field': 'Y', 'type': 'ordinal', 'sort': '-x'}}}\nSimple description:\nA bar chart with bars grouped by field X, and colored by field C\nresult vega-lite\n{'mark': 'bar', 'encoding': {'x': {'field': 'X'}, 'y': {'field': 'Y', 'type': 'quantitative'}, 'color': {'field': 'C'}, 'xOffset': {'field': 'C'}}}\nSimple description:\nA vertical bar chart with multiple bars for each X colored by field C, stacked on each other\nresult vega-lite\n{'mark': 'bar', 'encoding': {'x': {'timeUnit': '...', 'field': 'X', 'type': 'ordinal'}, 'y': {'aggregate': 'count', 'type': 'quantitative'}, 'color': {'field': 'C', 'type': 'nominal'}}}\nSimple description:\nA horizontal bar chart with multiple bars for each X colored by field C, stacked next to each other\nresult vega-lite\n{'mark': 'bar', 'encoding': {'x': {'aggregate': 'sum', 'field': 'X'}, 'y': {'field': 'Y'}, 'color': {'field': 'C'}}}\nSimple description:\nA stacked bar chart, where all stacks are normalized to sum to 100%\nresult vega-lite\n{'mark': 'bar', 'encoding': {'x': {'field': 'X'}, 'y': {'aggregate': 'sum', 'field': 'Y', 'stack': 'normalize'}, 'color': {'field': 'C'}}}\nSimple description:\nA bar chart with overlayed bars by group and transparency\nresult vega-lite\n{'mark': 'bar', 'encoding': {'x': {'field': 'X', 'type': 'ordinal'}, 'y': {'aggregate': 'sum', 'field': 'Y', 'stack': None}, 'color': {'field': 'C'}, 'opacity': {'value': 0.7}}}\nSimple description:\nA histogram is like a bar chart, after binning one field and aggregating the other\nresult vega-lite\n{'mark': 'bar', 'encoding': {'x': {'bin': True, 'field': 'X'}, 'y': {'aggregate': 'count'}}}\nSimple description:\nA pie chart encodes proportional differences among a set of numeric values as the angular extent and area of a circular slice.\nresult vega-lite\n{'mark': 'arc', 'encoding': {'theta': {'field': 'T', 'type': 'quantitative'}, 'color': {'field': 'C', 'type': 'nominal'}}}\nSimple description:\nHeatmap with binned quantitative variables on both axes\nresult vega-lite\n{'mark': 'rect', 'encoding': {'x': {'bin': {'maxbins': 60}, 'field': 'X', 'type': 'quantitative'}, 'y': {'bin': {'maxbins': 40}, 'field': 'Y', 'type': 'quantitative'}, 'color': {'aggregate': 'count', 'type': 'quantitative'}}}\nSimple description:\nA scatterplot shows the relationship between two quantitative variables X and Y\nresult vega-lite\n{'mark': 'point', 'encoding': {'x': {'field': 'X', 'type': 'quantitative'}, 'y': {'field': 'Y', 'type': 'quantitative'}}}\nSimple description:\nA scatterplot with data points from different groups having a different color and shape\nresult vega-lite\n{'mark': 'point', 'encoding': {'x': {'field': 'X', 'type': 'quantitative'}, 'y': {'field': 'Y', 'type': 'quantitative'}, 'color': {'field': 'C', 'type': 'nominal'}, 'shape': {'field': 'C', 'type': 'nominal'}}}\nSimple description:\nA scatter plot where the marker size is proportional to a quantitative field\nresult vega-lite\n{'mark': 'point', 'encoding': {'x': {'field': 'X', 'type': 'quantitative'}, 'y': {'field': 'Y', 'type': 'quantitative'}, 'size': {'field': 'S', 'type': 'quantitative'}}}\nSimple description:\nShow a quantitative variable over time, for different groups\nresult vega-lite\n{'mark': 'line', 'encoding': {'x': {'field': 'X', 'type': 'temporal'}, 'y': {'field': 'Y', 'type': 'quantitative'}, 'color': {'field': 'C', 'type': 'nominal'}}}\nSimple description:\nHeatmap with ordinal or nominal variables on both axes\nresult vega-lite\n{'mark': 'rect', 'encoding': {'y': {'field': 'Y', 'type': 'nominal'}, 'x': {'field': 'X', 'type': 'ordinal'}, 'color': {'aggregate': 'mean', 'field': 'C'}}}\nSimple description:\nMultiple line charts arranged next to each other horizontally\nresult vega-lite\n{'mark': 'line', 'encoding': {'x': {'field': 'X', 'type': 'temporal'}, 'y': {'field': 'Y', 'type': 'quantitative'}, 'color': {'field': 'C', 'type': 'nominal'}, 'column': {'field': 'F'}}}\nSimple description:\nMultiple line charts arranged next to each other vertically\nresult vega-lite\n{'mark': 'bar', 'encoding': {'x': {'field': 'X'}, 'y': {'aggregate': 'sum', 'field': 'Y'}, 'row': {'field': 'F'}}}\nSimple description:\nA line chart layed over a stacked bar chart, with independent y axes to accomodate different scales\nresult vega-lite\n{'layer': [{'mark': 'bar', 'encoding': {'x': {'field': 'X', 'type': 'ordinal'}, 'y': {'field': 'Y1', 'type': 'quantitative'}, 'color': {'field': 'C', 'type': 'nominal'}}}, {'mark': 'line', 'encoding': {'x': {'field': 'X', 'type': 'temporal'}, 'y': {'field': 'Y2', 'type': 'quantitative'}, 'color': {'field': 'C', 'type': 'nominal'}}}], 'resolve': {'scale': {'y': 'independent'}}}\nSimple description:\nA line chart with highlighting two regions of time with rectangles\nresult vega-lite\n{'layer': [{'mark': 'rect', 'data': {'values': [{'start': '...', 'end': '...', 'event': '...'}, {'start': '...', 'end': '...', 'event': '...'}]}, 'encoding': {'x': {'field': 'start', 'type': 'temporal'}, 'x2': {'field': 'end', 'type': 'temporal'}, 'color': {'field': 'event', 'type': 'nominal'}}}, {'mark': 'line', 'encoding': {'x': {'field': 'X', 'type': 'temporal'}, 'y': {'field': 'Y', 'type': 'quantitative'}, 'color': {'value': '#333'}}}]}\nSimple description:\nPlacing a horizontal dashed rule at a specific y value, on top of a line chart\nresult vega-lite\n{'layer': [{'mark': 'line', 'encoding': {'x': {'field': 'X', 'type': 'temporal'}, 'y': {'field': 'Y', 'type': 'quantitative'}, 'color': {'field': 'C', 'type': 'nominal'}}}, {'mark': {'type': 'rule', 'strokeDash': [2, 2], 'size': 2}, 'encoding': {'y': {'datum': '...', 'type': 'quantitative'}}}]}\nSimple description:\nPlacing a vertical dashed rule at a specific x value, on top of a line chart\nresult vega-lite\n{'layer': [{'mark': 'line', 'encoding': {'x': {'field': 'X', 'type': 'temporal'}, 'y': {'field': 'Y', 'type': 'quantitative'}, 'color': {'field': 'C', 'type': 'nominal'}}}, {'mark': {'type': 'rule', 'strokeDash': [2, 2], 'size': 2}, 'encoding': {'x': {'datum': {'year': '...', 'month': '...', 'date': '...', 'hours': '...', 'minutes': '...'}, 'type': 'temporal'}}}]}\nBesides, here are some requirements:\n1. Do not contain the key called 'data' in vega-lite specification.\n2. If mark.type = point and shape.field is a field of the data, the definition of the shape should be inside the root \"encoding\" object, NOT in the \"mark\" object, for example, {\"encoding\": {\"shape\": {\"field\": \"field_name\"}}}\n3. Please also generate title and description\nThe sample data in json format:\n${parameters.sampleData}\nThis is the schema of the data:\n${parameters.dataSchema}\nThe user used this PPL query to get the data: ${parameters.ppl}\nThe user's input question is: ${parameters.input_question}\nThe user's instruction on the visualization is: ${parameters.input_instruction}\nNow please reply a valid vega-lite specification in json based on above instructions.\nPlease only contain vega-lite in your response.\nFor each x, y, don't use list. \nFor all key 'encoding', use key 'layer' to include it, like {\"layer\": [{\"encoding\": ...}, ...]}\n"
            },
            "name": "Text2Vega",
            "type": "MLModelTool"
          }
        },
        {
          "id": "t2vega_agent",
          "type": "register_agent",
          "previous_node_inputs": {
            "create_t2vega_tool": "tools"
          },
          "user_inputs": {
            "parameters": {},
            "type": "flow",
            "name": "t2vega agent",
            "description": "this is the t2vega agent that has a set of rules to generate the visualizations"
          }
        },
        {
          "id": "t2vega_instruction_based_agent",
          "type": "register_agent",
          "previous_node_inputs": {
            "create_instruction_based_t2vega_tool": "tools"
          },
          "user_inputs": {
            "parameters": {},
            "type": "flow",
            "name": "t2vega instruction based agent",
            "description": "this is the t2vega agent that supports instructions"
          }
        }
      ]
    }
  }
}

You may meet with error. (This error may not always happen)

{
  "workflow_id": "0efEcJMBBQOJw60yNqxL",
  "error": "Failed to update workflow state for 0efEcJMBBQOJw60yNqxL on step t2vega_instruction_based_agent to add resource agent_id 1-fEcJMBBQOJw60yNqzf during step t2vega_instruction_based_agent, restStatus: CONFLICT",
  "state": "FAILED",
  "resources_created": [
    {
      "workflow_step_name": "create_connector",
      "workflow_step_id": "create_claude_connector",
      "resource_type": "connector_id",
      "resource_id": "0ufEcJMBBQOJw60yNqyC"
    },
    {
      "workflow_step_name": "register_remote_model",
      "workflow_step_id": "register_claude_model",
      "resource_type": "model_id",
      "resource_id": "1OfEcJMBBQOJw60yNqy7"
    },
    {
      "workflow_step_name": "deploy_model",
      "workflow_step_id": "register_claude_model",
      "resource_type": "model_id",
      "resource_id": "1OfEcJMBBQOJw60yNqy7"
    },
    {
      "workflow_step_name": "register_agent",
      "workflow_step_id": "t2vega_agent",
      "resource_type": "agent_id",
      "resource_id": "1ufEcJMBBQOJw60yNqzf"
    }
  ]
}

What is the expected behavior?

The flow framework provision API should work.

What is your host/environment?

Operating system, version.

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[yuye-aws]

There is actually no conflict in the flow framework template, so I doubt whether there is some bug in the flow framework. The error seems to come from the JAVA file src/main/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandler.java. Can some maintainer take a look?

[dblock]

[Catch All Triage - 1, 2, 3]

[dbwiddis]

There is actually no conflict in the flow framework template, so I doubt whether there is some bug in the flow framework. The error seems to come from the JAVA file src/main/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandler.java. Can some maintainer take a look?

Flow Framework does not have any APIs that return CONFLICT so it's coming from an external call. Here's the line of code where that error message is produced:

flow-framework/src/main/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandler.java

Line 867 in 0284554

"Failed to update workflow state for {} on step {} to {} resource {} {}",

That method is called in two places; this call is a response to a GET so wouldn't produce that exception:

flow-framework/src/main/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandler.java

Line 830 in 0284554

}, ex -> handleStateUpdateException(workflowId, resource, operation, 0, listener, ex)));

And here where we're trying to update the state index document but failing after 3 retries:

flow-framework/src/main/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandler.java

Lines 823 to 829 in 0284554

	client.update(
	updateRequest,
	ActionListener.wrap(
	r -> handleStateUpdateSuccess(workflowId, resource, operation, listener),
	e -> handleStateUpdateException(workflowId, resource, operation, retries, listener, e)
	)
	);

The fact that you don't always get the error implies a probable race condition. In this case it looks like we may be trying to simultaneously update the state document from multiple different workflow steps at the same time.

Here's a look at the steps being executed: (indented steps depend on the one above them)

create_claude_connector
└── register_claude_model
    ├── create_t2vega_tool
    │   └── t2vega_agent
    └── create_instruction_based_t2vega_tool
        └── t2vega_instruction_based_agent

Given the t2vega_agent successfully deployed it looks like the failure was trying to also simultaneously create the t2vega_instruction_based_agent.

We are using IfSeqNo and primaryTerm to make sure we are updating the correct version of the document, so the "retries" aren't helping as we continue to fail on the same attempt to update the old document.

In theory, we could mitigate this in Flow Framework with one or more of the following:

• use some sort of locking mechanism/semaphore on the state index updates
• retry (manually) on conflict, fetching a new version of the document. We'd have to decide how many times to retry to mitigate these simultaneous updates
• add some sort of "jitter" to the execution time of these update calls to minimize the chance of simultaneous updates

In the meantime, you can adjust the template to work around the race condition using one of the following

• add an "edge" between t2vega_agent and t2vega_instruction_based_agent to enforce a before/after execution even though they're not logically dependent on each other
• add a no-op step before one of the two agent calls with a delay parameter set

I'll leave this issue open as a feature request to improve our handling of simultaneous updates.

[dbwiddis]

Bug introduced here: prior version with painless script would use a new source document to update on retries:

• Update workflow state without using painless script  #894

Possible fix would follow a simplified version of the DDB Client changes in this PR:

• [Feature/multi_tenancy] Implement retryOnConflict on UpdateDataObject ml-commons#2941