Error using custom imported Llama3.1 model on Bedrock

[mgaionWalit]

• Lanchain V 0.3.2
• Lanchain AWS v 0.2.2

We are using a fine-tuned version of Llama 3.1-instruct, uploaded to Bedrock. Since we are using an ARN model ID (which does not contain any information about the specific Foundation Model used), we encountered an issue.

In the code chat_models/bedrock.py at line 349, there is an if statement evaluating the model string to choose between Llama2 and Llama3 for prompt conversion.

In our case, we need to use convert_messages_to_prompt_llama3, but the logic falls into the else statement, which uses convert_messages_to_prompt_llama.

Is there any solution to ensure the correct conversion function is used?

Thank you!

[3coins]

@mgaionWalit
The best solution here would be to and a new attribute to supply the base model used to fine tune the custom model.

[mgaionWalit]

Thanks for the reply.

There is a temporary workaround?

[3coins]

@mgaionWalit
Don't think there is a way to handle this without a code change. I don't have any custom model setup to verify, but does the Bedrock API provide any info regarding the model that can help us infer the base provider/model?

[3coins]

@mgaionWalit
The converse API now supports imported models for Meta and Mistral models, can you verify if this works as expected with the ChatBedrockConverse, by passing in a provider value.

[mgaionWalit]

Hi @3coins

thanks for you support.

I've updated langchain-aws to the latest available version (0.2.4) and tried to init the custom model with ChatBedrockConverse :

bedrock_runtime = boto3.client(
    service_name="bedrock-runtime"
)
model = ChatBedrockConverse(
    client=bedrock_runtime,
    model_id='arn:aws:bedrock:us-east-1:XXXXXX:imported-model/XXXXXX', # here I set the arn of the imported model
    provider='meta',
    temperature=0.15,
    max_tokens=100
)

but I still get an error when trying to start a chat with stream that says:
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the ConverseStream operation: This action doesn't support the model that you provided. Try again with a supported text or chat model

Am I doing something wrong? Do you have some suggestions on how to make it work?

[patilvishal0597]

Per conversation, we actually looked at Converse support for custom import models.

I was able to work on the issue; Since it was created, Bedrock has added Converse support for custom import models and it is expected the user adds a correct chat template. For the model mentioned in the issue Meta-Llama-3.1-8B-Instruct, the custom import supports Converse. The list of supported models and how to set up the template is documented in the AWS Bedrock User Guide, Converse API code samples for custom model import.

Here is the list of steps I did to make it work:

1. Get the model from HF
2. Ensure the chat template in the tokenizer_config.json file is appropriate, per the documentation, while uploading the model to S3.
3. Import the model to Bedrock via Imported Models
4. Used this following code:

from langchain_aws import ChatBedrockConverse
from langchain_aws import ChatBedrock

def main():
   # Initializing Bedrock chat converse for imported models
   llm = ChatBedrockConverse(
       model_id='arn:aws:bedrock:us-east-1:xxxxxxx:imported-model/xxxxxxxx', # here I set the arn of the imported model
       region_name="us-east-1",
       provider='meta',
       temperature=0.15,
       max_tokens=100
   )

   # Invoke the llm
   response = llm.invoke("What is Amazon Web Services?)
   print(response.content)

if __name__ == '__main__':
    main()

and it will yield an output:

Amazon Web Services (AWS) is a comprehensive cloud computing platform provided by Amazon that offers a wide range of services for computing, storage, databases, analytics, machine learning, and more. It allows businesses and individuals to store, process, and manage data, as well as run applications and services on the internet.

AWS was launched in 2006 and has since become one of the leading cloud computing platforms in the world, with a vast array of services and features that cater to various needs and industries