[Question]: How to integrate Google Vertex AI's Claude API with LibreChat using service account authentication?

[svnkxmza]

Hello,

I've been trying to integrate the Claude API provided by Google Vertex AI into LibreChat.

My current approach:

1. I've attempted to use the custom_endpoint feature in LibreChat.
2. However, I'm facing difficulties with the authentication process.

Key Issue:
The custom_endpoint configuration seems to only offer an API key option, but Google Vertex AI requires service account authentication using a JSON key file (auth.json).

Questions:

1. Is it possible to use Google service account authentication (auth.json) with LibreChat's custom_endpoint feature?
2. If yes, how can I properly configure this in LibreChat?
3. Does anyone have a working example or guide for integrating Google Vertex AI's Claude API with LibreChat?

Any insights, examples, or guidance would be greatly appreciated. Thank you in advance for your help!

[lidonius1122]

You can use the documentation on librechat.ai where it explains how to do it.

https://www.librechat.ai/docs/configuration/pre_configured_ai/google

You need to place the JSON file in the /api/data/ folder.

[spdustin]

@svnkxmza

Are you loading litellm in your docker-compose.override.yml?

services:
  api:
    volumes: # binds librechat.yml
    - type: bind
      source: ./librechat.yaml
      target: /app/librechat.yaml

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    volumes: # I placed my litellm configs in a separate directory
      - ./extras/litellm-config.yaml:/app/config.yaml # litellm config
      - ./extras/vertex.json:/app/vertex.json # service account
    command: [ "--config", "/app/config.yaml", "--port", "8000", "--num_workers", "8" ]
    environment:
      GOOGLE_APPLICATION_CREDENTIALS: /app/vertex.json # same local bind path as volume

# ./extras/litellm-config.yaml
model_list:
  - model_name: claude-3-opus-vertex
    litellm_params:
      model: vertex_ai/claude-3-opus@20240229
      vertex_ai_project: "<your project id>"
      vertex_ai_location: "<your region with model>"
  - model_name: claude-3-5-sonnet-vertex
    litellm_params:
      model: vertex_ai/claude-3-5-sonnet@20240620
      vertex_ai_project: "<your project id>"
      vertex_ai_location: "<your region with model>" 
general_settings:
  master_key: sk_live_SetToRandomValueabcdefgh8675309 # ignored
litellm_settings:
  drop_params: true # needed

# librechat.yml
# ...snipped to this entry under endpoints>custom
    - name: Claude
      apiKey: sk_live_SetToRandomValueabcdefgh8675309 # ignored
      baseURL: "http://litellm:8000/v1"
      # A "default" model to start new users with. The "fetch" will pull the rest of the available models from LiteLLM
      # More or less this is "irrelevant", you can pick any model. Just pick one you have defined in LiteLLM.
      models:
        default: ["claude-3-5-sonnet-vertex","claude-3-opus-vertex"]
        fetch: false
      titleConvo: true
      summarize: false
      forcePrompt: false
      modelDisplayLabel: "Claude"

Hope that helps

[svnkxmza]

@svnkxmza

Are you loading litellm in your docker-compose.override.yml?

services:
  api:
    volumes: # binds librechat.yml
    - type: bind
      source: ./librechat.yaml
      target: /app/librechat.yaml

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    volumes: # I placed my litellm configs in a separate directory
      - ./extras/litellm-config.yaml:/app/config.yaml # litellm config
      - ./extras/vertex.json:/app/vertex.json # service account
    command: [ "--config", "/app/config.yaml", "--port", "8000", "--num_workers", "8" ]
    environment:
      GOOGLE_APPLICATION_CREDENTIALS: /app/vertex.json # same local bind path as volume

# ./extras/litellm-config.yaml
model_list:
  - model_name: claude-3-opus-vertex
    litellm_params:
      model: vertex_ai/claude-3-opus@20240229
      vertex_ai_project: "<your project id>"
      vertex_ai_location: "<your region with model>"
  - model_name: claude-3-5-sonnet-vertex
    litellm_params:
      model: vertex_ai/claude-3-5-sonnet@20240620
      vertex_ai_project: "<your project id>"
      vertex_ai_location: "<your region with model>" 
general_settings:
  master_key: sk_live_SetToRandomValueabcdefgh8675309 # ignored
litellm_settings:
  drop_params: true # needed

# librechat.yml
# ...snipped to this entry under endpoints>custom
    - name: Claude
      apiKey: sk_live_SetToRandomValueabcdefgh8675309 # ignored
      baseURL: "http://litellm:8000/v1"
      # A "default" model to start new users with. The "fetch" will pull the rest of the available models from LiteLLM
      # More or less this is "irrelevant", you can pick any model. Just pick one you have defined in LiteLLM.
      models:
        default: ["claude-3-5-sonnet-vertex","claude-3-opus-vertex"]
        fetch: false
      titleConvo: true
      summarize: false
      forcePrompt: false
      modelDisplayLabel: "Claude"

Hope that helps

Thank you for your detailed guidance! It has given me a clearer direction on how to proceed. However, I'm still encountering an issue, and I would appreciate further assistance.

Here are my debug logs:

{
  baseURL: "https://us-east5-aiplatform.googleapis.com/v1",
    modelOptions.model: "claude-3-5-sonnet-vertex",
    modelOptions.temperature: 1,
    modelOptions.top_p: 1,
    modelOptions.presence_penalty: 0,
    modelOptions.frequency_penalty: 0,
    modelOptions.stop: undefined,
    modelOptions.max_tokens: undefined,
    modelOptions.user: "667034da36bda12282943a89",
    modelOptions.stream: true,
    // 3 message(s)
    modelOptions.messages: [{"role":"user","content":"123"},{"role":"assistant","content":"An error occurred while processing your request. Pl>
}
2024-06-24T15:29:35.547Z debug: Making request to https://us-east5-aiplatform.googleapis.com/v1/chat/completions
2024-06-24T15:29:36.498Z warn: [OpenAIClient.chatCompletion][stream] API error
2024-06-24T15:29:36.499Z error: [handleAbortError] AI response error; aborting request: 404 status code (no body)
2024-06-24T15:29:36.506Z debug: [AskController] Request closed

And here is my current librechat.yaml configuration:

custom:
  - name: Claude
    apiKey: sk_live_SetToRandomValueabcdefgh8675309 # ignored
    baseURL: "https://us-east5-aiplatform.googleapis.com/v1/projects/xxxx-xxxxx-xxxxxxx-c5/locations/us-east5/publishers/anthropic/"
    # A "default" model to start new users with. The "fetch" will pull the rest of the available models from LiteLLM
    # More or less this is "irrelevant", you can pick any model. Just pick one you have defined in LiteLLM.
    models:
      default: ["claude-3-5-sonnet-vertex","claude-3-opus-vertex"]
      fetch: false
    titleConvo: true
    summarize: false
    forcePrompt: false
    modelDisplayLabel: "Claude"

Thank you for providing the guidance on setting up . This has been incredibly helpful as I had no clear direction before. Despite this, I’ve been stuck for the last three hours trying to resolve an issue with the setup. I have experience setting up Azure's GPT using guidelines, but I couldn't find a similar guide for this integration, which left me without clear instructions.litellmbaseURL

Could you please assist me further with the correct setup for the and any additional configurations needed?baseURL

Thanks again for your help!

[spdustin]

You do need the litellm config passed into the container, did you do that part? :)

[R-Prady]

im trying to do the same thing. is litellm necessary? is there a way to do without litellm?

[mfmezger]

Are you loading litellm in your docker-compose.override.yml?

services:
  api:
    volumes: # binds librechat.yml
    - type: bind
      source: ./librechat.yaml
      target: /app/librechat.yaml

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    volumes: # I placed my litellm configs in a separate directory
      - ./extras/litellm-config.yaml:/app/config.yaml # litellm config
      - ./extras/vertex.json:/app/vertex.json # service account
    command: [ "--config", "/app/config.yaml", "--port", "8000", "--num_workers", "8" ]
    environment:
      GOOGLE_APPLICATION_CREDENTIALS: /app/vertex.json # same local bind path as volume

# ./extras/litellm-config.yaml
model_list:
  - model_name: claude-3-opus-vertex
    litellm_params:
      model: vertex_ai/claude-3-opus@20240229
      vertex_ai_project: "<your project id>"
      vertex_ai_location: "<your region with model>"
  - model_name: claude-3-5-sonnet-vertex
    litellm_params:
      model: vertex_ai/claude-3-5-sonnet@20240620
      vertex_ai_project: "<your project id>"
      vertex_ai_location: "<your region with model>" 
general_settings:
  master_key: sk_live_SetToRandomValueabcdefgh8675309 # ignored
litellm_settings:
  drop_params: true # needed

# librechat.yml
# ...snipped to this entry under endpoints>custom
    - name: Claude
      apiKey: sk_live_SetToRandomValueabcdefgh8675309 # ignored
      baseURL: "http://litellm:8000/v1"
      # A "default" model to start new users with. The "fetch" will pull the rest of the available models from LiteLLM
      # More or less this is "irrelevant", you can pick any model. Just pick one you have defined in LiteLLM.
      models:
        default: ["claude-3-5-sonnet-vertex","claude-3-opus-vertex"]
        fetch: false
      titleConvo: true
      summarize: false
      forcePrompt: false
      modelDisplayLabel: "Claude"

this works for me. thank you! :).