Providing file URL inputs to VertexAI Edge provider

[dan-massey]

Hi there, I'm getting started with assistant-ui and I'm trying to make calls to Vertex AI via an Edge Provider. I'd like to include both a system prompt and file inputs that are defined using gs:// URLs.

Vertex AI accepts file inputs as gs:// urls in their client libraries, and the @ai-sdk docs also mention passing URLs as data beign supported.

1. Is it possible to provide file URLs in the request to getEdgeRuntimeResponse, and is it possible to provide them as part of the system prompt?

Looking at the source, seems like the system message schema only accepts content of type TextContentPartSchema and the user accepts TextContentPartSchema, ImageContentPartSchema or Unstable_AudioContentPart.

I think I'm missing something?

2. Is this the correct way to provide a system prompt?

  return getEdgeRuntimeResponse({
    options: {
      model: vertex("gemini-1.5-pro"),
      system: "this is my system prompt",
    },
    requestData,
    abortSignal: request.signal,
  });

[Yonom]

The AI SDK doesn't support passing files as part of the system prompt, but only as a user message. Does Vertex support this on their end?

[dan-massey]

Ah, thanks for taking a look @Yonom.

The Vertex AI nodejs client doesn't prevent including file parts in the system prompt or in system contents, but it the server does throw an error if you try to include a file in either approach:

[VertexAI.ClientError]: got status: 400 Bad Request. {"error":{"code":400,"message":"Please ensure that only text part is used to specify the system_instruction.","status":"INVALID_ARGUMENT"}}

[VertexAI.ClientError]: got status: 400 Bad Request. {"error":{"code":400,"message":"Unable to submit request because a request can only contain one System role content with one text part. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini","status":"INVALID_ARGUMENT"}}

However, it does work to supply the file as in the "assistant" role's part. For example, this works:

import { VertexAI, type GenerateContentRequest } from "@google-cloud/vertexai";

// Initialize Vertex with your Cloud project and location
const vertex_ai = new VertexAI({
  project: "<PROJECT_NAME>",
  location: "us-central1",
});
const model = "gemini-1.5-flash-002";
const generativeModel = vertex_ai.preview.getGenerativeModel({
  model: model,
  generationConfig: {
    maxOutputTokens: 8192,
    temperature: 1,
    topP: 0.95,
  },
  safetySettings: [],
  systemInstruction: {
    role: "system",
    parts: [{
        text: `You are a document classification assistant. Given a document, your task is to classify the document type from provided document types. You must reply concisely with the document type.`,
      }],
  },
});

  const req: GenerateContentRequest = {
    contents: [
      {
        role: "assistant",
        parts: [{ text: "Here is a supplied document." }, {
            fileData: {
              mimeType: "application/pdf",
              fileUri: `gs://cloud-samples-data/generative-ai/pdf/w9.pdf`,
            },
          }],
      },
      { role: "user", parts: [{
        text: `What\'s the document type? Reply from the following options: Invoice, Bank Statement, Paystub, Form 1040, Form W-9, Form 1099-R.`,
      }] },
    ],
  };

  const streamingResp = await generativeModel.generateContentStream(req);

[Yonom]

Thank you, this should help me with the implementation