Skip to content

Commit

Permalink
refactor(component): rename Instill Format to Instill Type
Browse files Browse the repository at this point in the history
  • Loading branch information
@donch1989
donch1989 committed Jan 2, 2025
1 parent d6466bf commit e19d20d
Show file tree
Hide file tree
Showing 177 changed files with 5,501 additions and 10,477 deletions.
169 changes: 76 additions & 93 deletions pkg/component/CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,42 +54,39 @@ operator/hello/v0
├──assets
│ └──hello.svg
├──config
│ ├──definition.json
│ └──tasks.json
│ ├──definition.yaml
│ └──tasks.yaml
├──main.go
├──main_test.go
└──README.mdx
```

### Add the configuration files

Create a `config` directory and add the files `definition.json`, `tasks.json`,
and `setup.json` (optional). Together, these files define the behavior of the
Create a `config` directory and add the files `definition.yaml`, `tasks.yaml`,
and `setup.yaml` (optional). Together, these files define the behavior of the
component.

#### `definition.json`
#### `definition.yaml`

The `definition.json` file describes the high-level information of the
The `definition.yaml` file describes the high-level information of the
component.

```json
{
"id": "hello",
"uid": "e05d3d71-779c-45f8-904d-e90a050ca3b2",
"title": "Hello",
"type": "COMPONENT_TYPE_OPERATOR",
"description": "'Hello, world' operator used as a template for adding components",
"spec": {},
"availableTasks": [
"TASK_GREET"
],
"documentationUrl": "https://www.instill.tech/docs/component/operator/hello",
"icon": "assets/hello.svg",
"version": "0.1.0",
"sourceUrl": "https://github.com/instill-ai/pipeline-backend/pkg/component/blob/main/operator/hello/v0",
"releaseStage": "RELEASE_STAGE_ALPHA",
"public": true
}
```yaml
id: hello
uid: e05d3d71-779c-45f8-904d-e90a050ca3b2
title: Hello
type: COMPONENT_TYPE_OPERATOR
description: "'Hello, world' operator used as a template for adding components"
spec: {}
availableTasks:
- TASK_GREET
documentationUrl: https://www.instill.tech/docs/component/operator/hello
icon: assets/hello.svg
version: 0.1.0
sourceUrl: https://github.com/instill-ai/pipeline-backend/pkg/component/blob/main/operator/hello/v0
releaseStage: RELEASE_STAGE_ALPHA
public: true
```
This file defines the component properties:
Expand All @@ -106,7 +103,7 @@ This file defines the component properties:
- **`availableTasks`** defines the tasks the component can perform.
- When a component is created in a pipeline, one of the tasks has to be
selected, i.e., a configured component can only execute one task.
- Task configurations are defined in `tasks.json`.
- Task configurations are defined in `tasks.yaml`.
- **`documentationUrl`** points to the official documentation of the component.
- **`icon`** is the local path to the icon that will be displayed in the console
when creating the component. If left blank, a placeholder icon will be shown.
Expand All @@ -124,57 +121,42 @@ This file defines the component properties:
- **`public`** indicates whether the component is visible to the public.


#### `tasks.json`
#### `tasks.yaml`

The `tasks.json` file describes the task details of the component. The key
The `tasks.yaml` file describes the task details of the component. The key
should be in the format `TASK_NAME`.

```json
{
"TASK_GREET": {
"shortDescription": "Greet someone / something",
"title": "Greet",
"input": {
"description": "Input",
"uiOrder": 0,
"properties": {
"target": {
"uiOrder": 0,
"description": "The target of the greeting",
"acceptFormats": [
"string"
],
"title": "Greeting target",
"format": "string"
}
},
"required": [
"target"
],
"title": "Input",
"type": "object"
},
"output": {
"description": "The greeting sentence",
"uiOrder": 0,
"properties": {
"greeting": {
"description": "A greeting sentence addressed to the target",
"uiOrder": 0,
"required": [],
"title": "Greeting",
"type": "string",
"format": "string"
}
},
"required": [
"greeting"
],
"title": "Output",
"type": "object"
}
}
}
```yaml
TASK_GREET:
shortDescription: Greet someone / something
title: Greet
input:
description: Input
uiOrder: 0
properties:
target:
uiOrder: 0
description: The target of the greeting
type: string
title: Greeting target
required:
- target
title: Input
type: object
output:
description: The greeting sentence
uiOrder: 0
properties:
greeting:
description: A greeting sentence addressed to the target
uiOrder: 0
required: []
title: Greeting
type: string
required:
- greeting
title: Output
type: object
```

This file defines the input and output schema of each task:
Expand All @@ -187,13 +169,13 @@ This file defines the input and output schema of each task:
provide a description of the task in the component. If
**`shortDescription`** does not exist, it will be the same as
**`description`**.
- **`input`** is a JSON Schema that describes the input of the task.
- **`output`** is a JSON Schema that describes the output of the task.
- **`input`** is a schema that describes the input of the task.
- **`output`** is a schema that describes the output of the task.

**Properties within `input` and `output` Objects**

- **`required`** indicates whether the property is required.
- **`format`**: describes the format of this field, which could be `string`,
- **`type`**: describes the format of this field, which could be `string`,
`number`, `boolean`, `file`, `document`, `image`, `video`, `audio`, `array`,
or `object`.
- **`title`** is used by the console to provide the title of the property in the component.
Expand All @@ -207,37 +189,38 @@ This file defines the input and output schema of each task:

**Properties within `input` Objects**

- **`acceptFormats`** is an array indicating the data types of acceptable
input fields. It should be an array of [**Instill
Format**](https://www.instill.tech/docs/vdp/instill-format).
- Currently, we do not support the `time` type. When the input is a `date` or `datetime`, it should be represented as a string. The `date` or `datetime` will be automatically parsed in UTC timezone by the YAML parser. Please ensure this point is noted in the documentation, specifically for the `start-to-read-date` in the Slack component.
- **`type`** indicates the data type of the output field, which should be one
of `string`, `number`, `boolean`, `file`, `document`, `image`, `video`,
`audio`, `array`, or `object`. Please refer to [**Instill
Format**](https://www.instill.tech/docs/vdp/instill-format) for more details.

- **`instillSecret`** indicates the data must reference the secrets and cannot
be used in plaintext.

**Properties within `output` Objects**

- **`format`** indicates the data type of the output field, which should be one
- **`type`** indicates the data type of the output field, which should be one
of `string`, `number`, `boolean`, `file`, `document`, `image`, `video`,
`audio`, `array`, or `object`. Please refer to [**Instill
Format**](https://www.instill.tech/docs/vdp/instill-format) for more details.

See the [example recipe](#example-recipe) to understand how these fields map to
the recipe of a pipeline when configured to use this operator.

#### `setup.json`
#### `setup.yaml`

For components that need to set up some configuration before execution
(typically, components that connect with 3rd party applications or services that
need to set up a connection), `setup.json` can be used to describe these
configurations. The format is the same as the `input` objects in `tasks.json`.
need to set up a connection), `setup.yaml` can be used to describe these
configurations. The format is the same as the `input` objects in `tasks.yaml`.

The setup of a component can be defined within the recipe as key-value fields,
or as a reference to a **Connection** (see the
[**Integrations**](https://www.instill.tech/docs/vdp/integration) doc for more
information). Certain components support OAuth 2.0 integrations. If you want
your component to support this sort of connection:

- In `setup.json`, add the OAuth information under the `instillOAuthConfig`
- In `setup.yaml`, add the OAuth information under the `instillOAuthConfig`
property.
- `authUrl` contains the address where the authorization code can be
requested.
Expand Down Expand Up @@ -289,10 +272,10 @@ const (
)
var (
//go:embed config/definition.json
definitionJSON []byte
//go:embed config/tasks.json
tasksJSON []byte
//go:embed config/definition.yaml
definitionYAML []byte
//go:embed config/tasks.yaml
tasksYAML []byte
once sync.Once
comp *component
Expand All @@ -307,7 +290,7 @@ type component struct {
func Init(bc base.Component) *component {
once.Do(func() {
comp = &component{Component: bc}
err := comp.LoadDefinition(definitionJSON, nil, tasksJSON, nil)
err := comp.LoadDefinition(definitionYAML, nil, tasksYAML, nil)
if err != nil {
panic(err)
}
Expand Down Expand Up @@ -619,8 +602,8 @@ output:
Documentation helps user to integrate the component in their pipelines. A good
component definition will have clear names for their fields, which will also
contain useful descriptions. The information described in `definition.json` and
`tasks.json` is enough to understand how a component should be used. `compogen`
contain useful descriptions. The information described in `definition.yaml` and
`tasks.yaml` is enough to understand how a component should be used. `compogen`
is a tool that parses the component configuration and builds a `README.mdx` file
document displaying its information in a human-readable way. To generate the
document, just add the following line on top of `operator/hello/v0/main.go`:
Expand Down Expand Up @@ -673,7 +656,7 @@ Semantic Versioning guidelines.
It is recommended to start a component at `v0.1.0`. A major version 0 is
intended for rapid development.

The `releaseStage` property in `definition.json` indicates the stability of a
The `releaseStage` property in `definition.yaml` indicates the stability of a
component.

- A component skeleton (with only the minimal configuration files and a dummy
Expand Down
4 changes: 2 additions & 2 deletions pkg/component/ai/anthropic/v0/.compogen/bottom.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@ variable:
repository-name:
title: Repository Name
description: Name of the repository i.e. instill-core
format: string
type: string
repository-owner:
title: Repository Owner
description: Name of the repository owner i.e. instill-ai
format: string
type: string
output:
result:
title: Result
Expand Down
22 changes: 11 additions & 11 deletions pkg/component/ai/anthropic/v0/README.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ ${connection.<my-connection-id>}`.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Format | Note |
| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| API Key | `api-key` | string | Fill in your Anthropic API key. To find your keys, visit the Anthropic console page. |

Expand All @@ -54,14 +54,14 @@ Anthropic's text generation models (often called generative pre-trained transfor

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Input | Field ID | Format | Description |
| Input | Field ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_TEXT_GENERATION_CHAT` |
| Model Name (required) | `model-name` | string | The Anthropic model to be used. <br/><details><summary><strong>Enum values</strong></summary><ul><li>`claude-3-5-sonnet-latest`</li><li>`claude-3-5-sonnet-20241022`</li><li>`claude-3-5-sonnet-20240620`</li><li>`claude-3-opus-20240229`</li><li>`claude-3-sonnet-20240229`</li><li>`claude-3-haiku-20240307`</li></ul></details> |
| Prompt (required) | `prompt` | string | The prompt text. |
| System Message | `system-message` | string | The system message helps set the behavior of the assistant. For example, you can modify the personality of the assistant or provide specific instructions about how it should behave throughout the conversation. By default, the model’s behavior is set using a generic message as "You are a helpful assistant.". |
| Prompt Images | `prompt-images` | array[string] | The prompt images (Note: The prompt images will be injected in the order they are provided to the 'prompt' message. Anthropic doesn't support sending images via image-url, use this field instead). |
| [Chat History](#text-generation-chat-chat-history) | `chat-history` | array[object] | Incorporate external chat history, specifically previous messages within the conversation. Please note that System Message will be ignored and will not have any effect when this field is populated. Each message should adhere to the format: : \{"role": "The message role, i.e. 'system', 'user' or 'assistant'", "content": "message content"\}. |
| [Chat History](#text-generation-chat-chat-history) | `chat-history` | array[object] | Incorporate external chat history, specifically previous messages within the conversation. Please note that System Message will be ignored and will not have any effect when this field is populated. Each message should adhere to the format: \{"role": "The message role, i.e. 'system', 'user' or 'assistant'", "content": "message content"\}. |
| Seed | `seed` | integer | The seed (Note: Not supported by Anthropic Models). |
| Temperature | `temperature` | number | The temperature for sampling. |
| Top K | `top-k` | integer | Top k for sampling. |
Expand All @@ -74,11 +74,11 @@ Anthropic's text generation models (often called generative pre-trained transfor

<h4 id="text-generation-chat-chat-history">Chat History</h4>

Incorporate external chat history, specifically previous messages within the conversation. Please note that System Message will be ignored and will not have any effect when this field is populated. Each message should adhere to the format: : \{"role": "The message role, i.e. 'system', 'user' or 'assistant'", "content": "message content"\}.
Incorporate external chat history, specifically previous messages within the conversation. Please note that System Message will be ignored and will not have any effect when this field is populated. Each message should adhere to the format: \{"role": "The message role, i.e. 'system', 'user' or 'assistant'", "content": "message content"\}.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Format | Note |
| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Content](#text-generation-chat-content) | `content` | array | The message content. |
| Role | `role` | string | The message role, i.e. 'system', 'user' or 'assistant'. |
Expand All @@ -89,7 +89,7 @@ The message content.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Format | Note |
| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| [Image URL](#text-generation-chat-image-url) | `image-url` | object | The image URL. |
| Text | `text` | string | The text content. |
Expand All @@ -101,7 +101,7 @@ The image URL.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Format | Note |
| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| URL | `url` | string | Either a URL of the image or the base64 encoded image data. |
</div>
Expand All @@ -111,7 +111,7 @@ The image URL.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Output | Field ID | Format | Description |
| Output | Field ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Text | `text` | string | Model Output. |
| [Usage](#text-generation-chat-usage) (optional) | `usage` | object | Usage tokens in Anthropic. |
Expand All @@ -124,7 +124,7 @@ The image URL.

<div class="markdown-col-no-wrap" data-col-1 data-col-2>

| Field | Field ID | Format | Note |
| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Input Tokens | `input-tokens` | number | The input tokens used by Anthropic. |
| Output Tokens | `output-tokens` | number | The output tokens used by Anthropic. |
Expand Down Expand Up @@ -169,11 +169,11 @@ variable:
repository-name:
title: Repository Name
description: Name of the repository i.e. instill-core
format: string
type: string
repository-owner:
title: Repository Owner
description: Name of the repository owner i.e. instill-ai
format: string
type: string
output:
result:
title: Result
Expand Down
19 changes: 0 additions & 19 deletions pkg/component/ai/anthropic/v0/config/setup.json
View file

This file was deleted.

Loading

0 comments on commit e19d20d

Please sign in to comment.