Model API QA: Errors met when testing model training and replacing loaded model

[ancalita]

Problem description:

1. Testing /model/train endpoint with the example in the docs leads to a YAML validation error (400 Bad Request error:

"The request body does not contain valid YAML. Error: Failed to validate YAML. Please make sure the file is correct and all mandatory parameters are specified. Here are the errors found during validation:\n  in Line 1:\n      Value 'b'- intent: greet\\\\n examples: |\\\\n - hey\\\\n - hello\\\\n - hi'' is not a list. Value path: '/nlu'"

2. Testing PUT /model endpoint with this request body (existing file on disk) leads to 400 Bad Request:

{
"model_file": "/models/20211110-091805-impulsive-birth.tar.gz"
}

It seems not to be able to identify the path given, this is the response error message:
"Agent with name '/models/20211110-091805-impulsive-birth.tar.gz' could not be loaded."
The debug log shows this user warning:
UserWarning: No valid configuration given to load agent. Agent loaded with no model!

[indam23]

Found the same - @tayfun I see you're assigned so maybe you already saw this, but it expects literal yaml, so a documentation issue - this request works:

curl --request POST \
  --url http://localhost:5005/model/train \
  --header 'Content-Type: text/yaml' \
  --data 'version: '\''3.0'\''
config:
  store_entities_as_slots: true
session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true
intents:
- greet:
    use_entities: true
- goodbye:
    use_entities: true
- affirm:
    use_entities: true
- deny:
    use_entities: true
- mood_great:
    use_entities: true
- mood_unhappy:
    use_entities: true
- bot_challenge:
    use_entities: true
entities:
- name
slots:
  name:
    type: rasa.shared.core.slots.AnySlot
    initial_value: null
    influence_conversation: false
    mappings:
    - type: from_entity
      entity: name
  yes:
    type: rasa.shared.core.slots.AnySlot
    initial_value: null
    influence_conversation: false
    mappings:
    - type: from_intent
      intent: affirm
      value: yes
    - type: from_intent
      not_intent: affirm
      value: no
responses:
  utter_greet:
  - text: Hey {name}! How are you?
  utter_cheer_up:
  - text: '\''Here is something to cheer you up:'\''
    image: https://i.imgur.com/nGF1K8f.jpg
  utter_did_that_help:
  - text: Did that help you?
  utter_happy:
  - text: Great, carry on!
  utter_goodbye:
  - text: Bye
  utter_iamabot:
  - text: I am a bot, powered by Rasa.
actions: []
forms: {}
e2e_actions: []
nlu:
- intent: greet
  examples: |
    - hey [Molly](name)
    - hello [Bob](name)
    - hi [Chris](name)
    - hello there [Jim](name)
    - good morning [Fred](name)
    - good evening [Sally](name)
    - moin
    - hey there
    - let'\''s go
    - hey dude
    - goodmorning
    - goodevening
    - good afternoon
    - hi
- intent: goodbye
  examples: |
    - cu
    - good by
    - cee you later
    - good night
    - bye
    - goodbye
    - have a nice day
    - see you around
    - bye bye
    - see you later
stories:
- story: happy path
  steps:
  - intent: greet
  - action: utter_greet
  - intent: mood_great
  - action: utter_happy
rules:
- rule: Say goodbye anytime the user says goodbye
  steps:
  - intent: goodbye
  - action: utter_goodbye
- rule: Say '\''I am a bot'\'' anytime the user challenges
  steps:
  - intent: bot_challenge
  - action: utter_iamabot
'

[indam23]

@ancalita I think the model replacement 400s because of the absolute path - was the model really at the root of the machine?

If I follow the doc and use the full path to an existing model, instead of a 400 I get a 204, but the loaded model's name doesn't change:

 $ curl --request PUT \
  --url http://localhost:5005/model \
  --header 'Content-Type: application/json' \
  --data '{
  "model_file": "/Users/melinda/rasa_bots/3.0rc2_QA/models/20211115-213850-covering-bosun.tar.gz"
}' -vv
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 5005 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5005 (#0)
> PUT /model HTTP/1.1
> Host: localhost:5005
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 101
> 
* upload completely sent off: 101 out of 101 bytes
< HTTP/1.1 204 No Content
< Access-Control-Expose-Headers: filename
< Access-Control-Allow-Credentials: true
< connection: keep-alive
< 
* Connection #0 to host localhost left intact
* Closing connection 0

status remains the same as before:

$ curl --request GET \
  --url 'http://localhost:5005/status'
  
{
  "model_file": "20211115-220201-witty-agent.tar.gz",
  "model_id": "b0d7f6d363ff47cba922c33dbe38fb1b",
  "num_active_training_jobs": 0
}

there's only a log about UnexpectTED in the logs to show it read the model at all

2021-11-15 22:12:23 WARNING  rasa.shared.utils.common  - The UnexpecTED Intent Policy is currently experimental and might change or be removed in the future 🔬 Please share your feedback on it in the forum (https://forum.rasa.com) to help us make this feature ready for production.

[indam23]

Also: After unloading the model with a DELETE, doing the PUT just reactivates the previously loaded model.

[ancalita]

@ancalita I think the model replacement 400s because of the absolute path - was the model really at the root of the machine?

@melindaloubser1 Oh that could have been the problem, not using absolute path, because indeed the models were not really at the root directory. But probably this could be made clearer in the docs (to use absolute paths) for those naive folks like myself copy-pasting examples 😅

[tayfun]

Yes, yes you are both right @melindaloubser1 @ancalita . I need to update the documentation (and remove obsolete code which was handling JSON since only YAML is supported now). I'm struggling with redocly, specifically only using yaml in documentation has not been easy.

[indam23]

^^ I've had a fight with the API spec and YAML example snippets too, really cannot figure out what the magic is there besides that specifying the ref to the snippet separately for both schema & example sort-of works

[tayfun]

@melindaloubser1 You're right. I had too much confidence in OpenAPI and redocly I think. It seems XML and YAML are not supported directly, see Redocly/redoc#492 (comment)

[indam23]

@tayfun Could you get the model replacement to work? Since the docs change the path instructions only, seems the endpoint itself may not be behaving as it ought?