"anthropic.claude-3-5-sonnet-20241022-v2:0 with on-demand throughput isn’t supported."

[ldavis9000aws]

What happened?

1. Ran git pull on the repo
2. Ran "docker-compose pull"
3. Restarted Librechat, navigated to the "AWS Bedrock" endpoint, then choose anthropic.claude-3-5-sonnet-20241022-v2:0.
4. When I start a chat, I receive this error:

LibreChat | 2024-10-28 23:01:29 error: [api/server/controllers/agents/client.js #sendCompletion] Unhandled error type Invocation of model ID anthropic.claude-3-5-sonnet-20241022-v2:0 with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model.
LibreChat | 2024-10-28 23:01:29 error: [handleAbortError] AI response error; aborting request: Invocation of model ID anthropic.claude-3-5-sonnet-20241022-v2:0 with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model.

If I switch to the "anthropic.claude-3-5-sonnet-20240620-v1:0" model, it works fine.

Steps to Reproduce

1. Ran a git pull
2. Ran "docker-compose pull"
3. Restarted Librechat, navigated to the "AWS Bedrock" endpoint, then choose anthropic.claude-3-5-sonnet-20241022-v2:0.
4. Start a chat with the new anthropic.claude-3-5-sonnet-20241022-v2:0 model

What browsers are you seeing the problem on?

No response

Relevant log output

No response

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Moved to troubleshooting since this depends on your configured AWS account/region.

[ldavis9000aws]

It looks like the new Claude 3.5 v2 Sonnet Model in Amazon Bedrock is only available in us-west-2 at the moment. It also required me to duplicate access requests to all of the other models I use via Librechat in us-east-1 for us-west-2 (a region I never use) just to get access to the new model. For now, I guess specifying us-west-2 as the region in the .env is the workaround until it's available in us-east-1. However, if the user were able to specify their AWS creds and region in the endpoint via the LibreChat UI (#4572), we wouldn't have to worry about updating the default region defined in the .env file.

Thanks for pointing me towards the model access issue that was causing the error.

[stephaneminisini]

To be able to use the latest Sonnet 3.5 from the us-east-1 region, the model inference profile id is 'us.anthropic.claude-3-5-sonnet-20241022-v2:0', it will route the inference to the region where the model is hosted.

[tai-acall]

Thanks a lot, you save my day.

[josephroffey]

I was hitting the same issue with an EU instance profile and I had to do the same thing 'eu.anthropic.claude-3-5-sonnet-20240620-v1:0' so your comment pointed me in the right direction, thanks!