Skip to content

Latest commit

 

History

History
600 lines (452 loc) · 36 KB

usage.md

File metadata and controls

600 lines (452 loc) · 36 KB
Raw

LLM Interface Usage Documentation

Table of Contents

LLMInterface

To use the LLMInterface.* functions, first import llm-interface. You can do this using either the CommonJS require syntax:

const { LLMInterface } = require('llm-interface');

getAllModelNames()

Retrieves a sorted list of all model names available in the configuration.

const modelNames = LLMInterface.getAllModelNames();
console.log(modelNames);

getEmbeddingsModelAlias(interfaceName, alias)

Retrieves an embeddings model name for a specific interfaceName alias.

const model = LLMInterface.getEmbeddingsModelAlias('openai', 'default');
console.log(model);

getInterfaceConfigValue(interfaceName, key)

Retrieves a specific configuration value for a given model.

  • interfaceName (String): The name of the model.
  • key (String): The configuration key to retrieve.
const apiKey = LLMInterface.getInterfaceConfigValue('openai', 'apiKey');
console.log(apiKey);

getModelAlias(interfaceName, alias)

Retrieves a model name for a specific interfaceName alias.

const model = LLMInterface.getModelAlias('openai', 'default');
console.log(model);

setApiKey(interfaceNames, apiKey)

Sets the API key for one or multiple interfaces.

  • interfaceNames (String|Object): The name of the interface or an object mapping interface names to API keys.
  • apiKey (String): The API key.
LLMInterface.setApiKey('openai', 'your-api-key');
// or
LLMInterface.setApiKey({ openai: 'your-api-key', cohere: 'another-api-key' });

setEmbeddingsModelAlias(interfaceName, alias, name)

Sets an alias for a model within a specific interface.

  • interfaceName (String): The name of the interface.
  • alias (String): The alias to set.
  • name (String): The model name.
LLMInterface.setEmbeddingsModelAlias(
  'openai',
  'default',
  'text-embedding-3-large',
);

setModelAlias(interfaceName, alias, name)

Sets an alias for a model within a specific interface.

  • interfaceName (String): The name of the interface.
  • alias (String): The alias to set.
  • name (String): The model name.
LLMInterface.setModelAlias('openai', 'default', 'gpt-3.5-turbo');

configureCache(cacheConfig = {})

Configures the cache system for the session. LLMInterface supports three caching mechanisms: simple-cache, flat-cache, and cache-manager. To use flat-cache or cache-manager, you need to install the corresponding packages.

  • cacheConfig (Object): Configuration options for the cache.
LLMInterface.configureCache({ cache: 'simple-cache', path: './cache' });

flushCache()

Clears the active cache for the session. Ensure you run LLMInterface.configureCache() beforehand.

LLMInterface.flushCache();

sendMessage(interfaceName, message, options = {}, interfaceOptions = {})

Sends a message to a specified interface and returns the response. The specified interface must already have its API key set, or it must be passed using the array format.

  • interfaceName (String|Array): The name of the LLM interface or an array containing the name of the LLM interface and the API key.
  • message (String|Object): The message to send.
  • options (Object|number, optional): Model specific options or parameters. If a number, it represents the cache timeout in seconds.
  • interfaceOptions (Object, optional): Interface-specific options.
// use this after you've set your API key
try {
  const response = await LLMInterface.sendMessage('openai', 'Hello, world!', {
    max_tokens: 100,
  });
  console.log(response.results);
} catch (error) {
  console.error(error.message);
}
// or use this to set your API key in the same command
try {
  const response = await LLMInterface.sendMessage(
    ['openai', 'your-api-key'],
    'Hello, world!',
    { max_tokens: 100 },
  );
  console.log(response.results);
} catch (error) {
  console.error(error.message);
}

streamMessage(interfaceName, message, options = {})

Streams a message to a specified interface and returns the response stream. You can also stream responses using LLMInterface.sendMessage, just pass options.stream=true.

  • interfaceName (String): The name of the LLM interface.
  • message (String|Object): The message to send.
  • options (Object|number, optional): Model specific options or parameters. If a number, it represents the cache timeout in seconds.
try {
  const stream = await LLMInterface.streamMessage('openai', 'Hello, world!', {
    max_tokens: 100,
  });
  const result = await processStream(stream.data);
} catch (error) {
  console.error(error.message);
}

processStream(stream) is not part of LLMInterface. It is defined in thestreaming mode example.

embeddings(interfaceName, embeddingString, options = {}, interfaceOptions = {}, defaultProvider = 'voyage')

Generates embeddings using a specified LLM interface.

  • interfaceName (String): The name of the LLM interface to use.
  • embeddingString (String): The string to generate embeddings for.
  • options (Object|number, optional): Model specific options or parameters. If a number, it represents the cache timeout in seconds.
  • interfaceOptions (Object, optional): Options specific to the LLM interface.
  • defaultProvider (String, optional): The default provider to use if the specified interface doesn't support embeddings. Defaults to 'voyage'.
try {
  const embeddings = await LLMInterface.embedding('openai', 'Text to embed', {
    max_tokens: 100,
  });
  console.log(embeddings);
} catch (error) {
  console.error(error.message);
}

chat.completions.create(interfaceName, message, options = {}, interfaceOptions = {})

Alias for LLMInterface.sendMessage (For those OpenAI fans :)).

const response = await LLMInterface.chat.completions.create(
  'openai',
  'Hello, world!',
  { max_tokens: 100 },
);
console.log(response.results);

Supported Interface Names

The following are the interfaceNames for each supported LLM provider (in alphabetical order):

Interface Name Provider Name .sendMessage .streamMessage .embeddings
ai21 ai21 AI21 Studio ✓ ✓ ✓
ailayer AiLAYER ✓
aimlapi aimlapi AIMLAPI ✓ ✓ ✓
anthropic anthropic Anthropic ✓ ✓
anyscale anyscale Anyscale ✓ ✓
cloudflareai cloudflareai Cloudflare AI ✓ ✓
cohere cohere Cohere ✓ ✓ ✓
corcel corcel Corcel ✓ ✓
deepinfra deepinfra DeepInfra ✓ ✓ ✓
deepseek deepseek DeepSeek ✓ ✓
fireworksai Fireworks AI ✓ ✓ ✓
forefront forefront Forefront AI ✓
friendliai FriendliAI ✓ ✓
gemini Google Gemini ✓ ✓ ✓
gooseai gooseai GooseAI ✓ ✓
groq Groq ✓ ✓
huggingface Hugging Face Inference ✓ ✓
hyperbeeai HyperBee AI ✓ ✓
lamini lamini Lamini ✓ ✓
llamacpp LLaMA.CPP ✓ ✓ ✓
mistralai mistralai Mistral AI ✓ ✓ ✓
monsterapi monsterapi Monster API ✓ ✓
neetsai neetsai Neets.ai ✓ ✓
novitaai Novita AI ✓ ✓
nvidia NVIDIA AI ✓ ✓
octoai OctoAI ✓
ollama Ollama ✓ ✓ ✓
openai OpenAI ✓ ✓ ✓
perplexity perplexity Perplexity AI ✓ ✓
rekaai rekaai Reka AI ✓ ✓
replicate replicate Replicate ✓ ✓
shuttleai shuttleai Shuttle AI ✓ ✓
thebai TheB.ai ✓ ✓
togetherai togetherai Together AI ✓ ✓ ✓
voyage Voyage AI ✓
watsonxai Watsonx AI ✓ ✓
writer writer Writer ✓ ✓
zhipuai Zhipu AI ✓ ✓

This is regularly updated! :)

LLMInterfaceSendMessage

To use the LLMInterfaceSendMessage function, first import LLMInterfaceSendMessage. You can do this using either the CommonJS require syntax:

const { LLMInterfaceSendMessage } = require('llm-interface');

LLMInterfaceSendMessage(interfaceName, apiKey, message, options = {}, interfaceOptions = {})

Sends a message using the specified LLM interface.

  • interfaceName (String): The name of the LLM interface.
  • apiKey (String): The API key.
  • message (String|Object): The message to send.
  • options (Object, optional): Additional options for the message.
  • interfaceOptions (Object, optional): Interface-specific options.
try {
  const response = await LLMInterfaceSendMessage(
    'openai',
    'your-api-key',
    'Hello, world!',
    { max_tokens: 100 },
  );
  console.log(response.results);
} catch (error) {
  console.error(error.message);
}

This is a legacy function and will be depreciated.

LLMInterfaceStreamMessage

To use the LLMInterfaceStreamMessage function, first import LLMInterfaceStreamMessage. You can do this using either the CommonJS require syntax:

const { LLMInterfaceStreamMessage } = require('llm-interface');

LLMInterfaceStreamMessage(interfaceName, apiKey, message, options = {})

Streams a message using the specified LLM interface.

  • interfaceName (String): The name of the LLM interface.
  • apiKey (String): The API key.
  • message (String|Object): The message to send.
  • options (Object, optional): Additional options for the message.
try {
  const stream = await LLMInterfaceStreamMessage(
    'openai',
    'your-api-key',
    'Hello, world!',
    { max_tokens: 100 },
  );
  const result = await processStream(stream.data);
} catch (error) {
  console.error(error.message);
}

processStream(stream) is defined in the streaming mode example.

This is a legacy function and will be depreciated.

Message Object

The message object is a critical component when interacting with the various LLM APIs through the LLM Interface npm module. It contains the data that will be sent to the LLM for processing and allows for complex conversations. Below is a detailed explanation of the structure of a valid message object."

Structure of a Message Object

A valid message object typically includes the following properties:

  • model: A string specifying the model to use for the request (optional).
  • messages: An array of message objects that form the conversation history.

Different LLMs may have their own message object rules. For example, both Anthropic and Gemini always expect the initial message to have the user role. Please be aware of this and structure your message objects accordingly.

LLMInterface will attempt to auto-correct invalid objects where possible.

Options Object

The options object is an optional component that lets you send LLM provider specific parameters. While parameter names are fairly consistent, they can vary slightly, so it is important to pay attention.

However, max_token is a special value, and is automatically normalized and is set with a default value of 1024.

Structure of an Options Object

A valid options object can contain any number of LLM provider specific parameters, however it always contains the default options.max_tokens value of 150:

  • max_tokens (default: 150)

Two other common values of interest are:

  • stream (default: false)
  • response_format (default: null)

If options.stream is true, then a LLMInterface.sendMessage() or LLMInterfaceSendMessage() call becomes a LLMInterface.streamMessage() call.

If options.response_format is set to "jsonobject", along with including a JSON schema in the prompt, many LLM providers will return a valid JSON object. _Not all providers support this feature.

const options = {
  max_tokens: 1024,
  temperature: 0.3, // Lower values are more deterministic, Higher are more creative
};

Interface Options Object

The interfaceOptions is an optional component when interacting with the various LLM APIs through the LLM Interface npm module. It contains interface-specific configuration.

Structure of an Interface Options Object

A valid interfaceOptions object can contain any of the following properties:

  • retryAttempts (default: 1)
  • retryMultiplier (default: 0.3)
  • cacheTimeoutSeconds (default: false)
  • attemptJsonRepair (default: false)
  • includeOriginalResponse (default: false)
const interfaceOptions = {
  retryAttempts: 3,
  retryMultiplier: 0.5, // llm-interface uses progressive delays, Lower values are faster
  cacheTimeoutSeconds: 60,
  attemptJsonRepair: true,
  includeOriginalResponse: true,
};

Interface object examples are provided in the /examples folder:

Caching

Caching is a crucial feature that can significantly enhance your application's performance by reducing the number of requests made to the LLM APIs. The LLM Interface npm module supports various caching mechanisms, each tailored to different use cases and configuration options.

Cache Initialization

Before utilizing a cache, it needs to be initialized. The exception to this rule is the Simple Cache. If you do not initialize a cache but request a cached response, the Simple Cache will automatically initialize before processing the request.

Simple Cache

Simple Cache uses the default cache engine provided by the LLM Interface npm module. It is suitable for basic caching needs without additional dependencies.

Example Initialization (Optional)

Here's how to configure the Simple Cache:

LLMInterface.configureCache({ cache: 'simple-cache' });

Flat Cache

Flat Cache is a simple and efficient in-memory cache that uses a file-based storage. It is ideal for lightweight caching needs and is easy to set up.

Installation

Before using the Flat Cache, install the necessary package:

npm install flat-cache

Example Initialization

Here's how to configure the Flat Cache:

LLMInterface.configureCache({ cache: 'flat-cache' });

Cache Manager

Cache Manager is a well-known package that supports many backends for caching. It allows you to use various storage systems for caching, such as in-memory, Redis, SQLite, and file system-based caches. This flexibility makes it a robust choice for different caching needs.

Installation

Before using Cache Manager, install the necessary packages, include the packages for your store:

npm install cache-manager@4.0.0 cache-manager-fs-hash

Example Initialization

Here's how to configure the Cache Manager with a file system-based store (using cache-manager-fs-hash):

const fsStore = require('cache-manager-fs-hash');

LLMInterface.configureCache({
  cache: 'cache-manager',
  config: {
    store: fsStore,
    options: {
      path: '../../cache', // Path to the directory where cache files will be stored
      ttl: 60 * 60, // Time to live in seconds (1 hour)
      subdirs: true, // Create subdirectories to reduce the number of files per directory
      zip: false, // Compress files to save space
    },
  },
});

Advanced Backends

Cache Manager also supports advanced backends like Redis, Memcached, and MongoDB. Here are examples of how to configure each:

  • Redis
const redisStore = require('cache-manager-redis-store');

LLMInterface.configureCache({
  cache: 'cache-manager',
  config: {
    store: redisStore,
    options: {
      host: 'localhost', // Redis server host
      port: 6379, // Redis server port
      ttl: 60 * 60, // Time to live in seconds (1 hour)
    },
  },
});
  • Memcached
const memcachedStore = require('cache-manager-memcached-store');

LLMInterface.configureCache({
  cache: 'cache-manager',
  config: {
    store: memcachedStore,
    options: {
      servers: '127.0.0.1:11211', // Memcached server address
      ttl: 60 * 60, // Time to live in seconds (1 hour)
    },
  },
});
  • MongoDB
const mongoStore = require('cache-manager-mongodb');

LLMInterface.configureCache({
  cache: 'cache-manager',
  config: {
    store: mongoStore,
    options: {
      uri: 'mongodb://localhost:27017/cache', // MongoDB connection URI
      collection: 'cacheCollection', // MongoDB collection name
      ttl: 60 * 60, // Time to live in seconds (1 hour)
    },
  },
});

Memory Cache

Memory Cache stores responses in memory for quick retrieval during subsequent requests within the specified time-to-live (TTL).

Example Initialization

LLMInterface.configureCache({ cache: 'memory-cache' });

Using the Cache

Once the cache is configured, you can use the cache by passing a cacheTimeoutSeconds value through options.

try {
  const response = await LLMInterface.sendMessage('openai', 'Hello, world!', {
    cacheTimeoutSeconds: 3600,
  });
  console.log(response.results);
} catch (error) {
  console.error(error.message);
}
// or
try {
  const response = await LLMInterface.sendMessage(
    'openai',
    'Hello, world!',
    3600,
  );
  console.log(response.results);
} catch (error) {
  console.error(error.message);
}

Caching examples are provided in the /examples folder: