Skip to content

Commit

Permalink
Update python SDK reference
Browse files Browse the repository at this point in the history
  • Loading branch information
jverre committed Sep 2, 2024
1 parent e51e513 commit 96581fe
Show file tree
Hide file tree
Showing 14 changed files with 132 additions and 53 deletions.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Comet
=====
Opik
====

.. autoclass:: opik.Comet
.. autoclass:: opik.Opik
:members:
:inherited-members:

This file was deleted.

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,8 +1,25 @@
metrics
=======

Opik includes a number of pre-built metrics to help you evaluate your LLM application.

Each metric can be called as a standalone function using the `score` method::

from opik.evaluation.metrics import Hallucination

metric = Hallucination()

metric.score(
input="What is the capital of France?",
output="The capital of France is Paris. It is famous for its iconic Eiffel Tower and rich cultural heritage.",
context=["France is a country in Western Europe. Its capital is Paris, which is known for landmarks like the Eiffel Tower."],
)

Or as part of an evaluation run using the `evaluate` function.

You can learn more about each metric in the following sections:

.. toctree::
:hidden:
:maxdepth: 4
:titlesonly:

Expand Down
51 changes: 29 additions & 22 deletions apps/opik-documentation/python-sdk-docs/source/index.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
opik
==============
Opik
====

=============
Main features
Expand All @@ -9,11 +9,11 @@ The Comet Opik platform is a suite of tools that allow you to evaluate the outpu

In includes the following features:

- `Tracing <...>`_: Ability to log LLM calls and traces to the Comet platform.
- `LLM evaluation metrics <...>`_: A set of functions that evaluate the output of an LLM, these are both heuristic metrics and LLM as a Judge.
- `Evaluation <...>`_: Ability to log test datasets in Comet and evaluate using some of our LLM evaluation metrics.
- `Tracing <https://www.comet.com/docs/opik/tracing/log_traces>`_: Ability to log LLM calls and traces to the Opik platform.
- `LLM evaluation metrics <https://www.comet.com/docs/opik/evaluation/metrics/heuristic_metrics>`_: A set of functions that evaluate the output of an LLM, these are both heuristic metrics and LLM as a Judge.
- `Evaluation <https://www.comet.com/docs/opik//evaluation/evaluate_your_llm>`_: Ability to log test datasets in Opik and evaluate using some of our LLM evaluation metrics.

For a more detailed overview of the platform, you can refer to the `Comet Opik documentation <...>`_.
For a more detailed overview of the platform, you can refer to the `Comet Opik documentation <https://www.comet.com/docs/opik>`_.

============
Installation
Expand All @@ -24,7 +24,7 @@ To get start with the package, you can install it using pip::
pip install opik

By default, all traces, datasets and experiments will be logged to the Comet Cloud platform. If you
would like to self-host the platform, you can refer to our `self-serve documentation <...>`_.
would like to self-host the platform, you can refer to our `self-serve documentation <https://www.comet.com/docs/opik/self-host/self_hosting_opik>`_.

=============
Using the SDK
Expand All @@ -49,7 +49,7 @@ To log your first trace, you can use the `track` decorator::

**Note:** The `track` decorator supports nested functions, if you track multiple functions, each functionc call will be associated with the parent trace.

**Integrations**: If you are using LangChain or OpenAI, Comet Opik as `built-in integrations <...>`_ for these libraries.
**Integrations**: If you are using LangChain or OpenAI, Comet Opik as `built-in integrations <https://www.comet.com/docs/opik/tracing/integrations/langchain>`_ for these libraries.

----------------------------
Using LLM evaluation metrics
Expand Down Expand Up @@ -78,7 +78,7 @@ Running evaluations

Evaluations are run using the `evaluate` function, this function takes a dataset, a task and a list of metrics and returns a dictionary of scores::

from opik import Comet, track
from opik import Opik, track
from opik.evaluation import evaluate
from opik.evaluation.metrics import EqualsMetric, HallucinationMetric
from opik.integrations.openai import track_openai
Expand All @@ -100,8 +100,8 @@ Evaluations are run using the `evaluate` function, this function takes a dataset
return ["..."]

# Fetch the dataset
comet = Comet()
dataset = comet.get_dataset(name="your-dataset-name")
client = Opik()
dataset = client.get_dataset(name="your-dataset-name")

# Define the metrics
equals_metric = EqualsMetric()
Expand All @@ -121,39 +121,46 @@ Evaluations are run using the `evaluate` function, this function takes a dataset
metrics=[equals_metric, hallucination_metric],
)

=========
Reference
=========

You can learn more about the `opik` python SDK in the following sections:

.. toctree::
:hidden:

Comet
:maxdepth: 1
Opik
track
comet_context/index
opik_context/index

.. toctree::
:caption: Evaluation
:hidden:
:maxdepth: 4
:maxdepth: 1

evaluation/Dataset
evaluation/DatasetItem
evaluation/evaluate
evaluation/metrics/index

.. toctree::
:caption: Testing
:maxdepth: 1

testing/llm_unit

.. toctree::
:caption: Integrations
:hidden:
:maxdepth: 4
:maxdepth: 1

integrations/openai/index
integrations/langchain/index

.. toctree::
:caption: Objects
:hidden:
:maxdepth: 4
:maxdepth: 1

Objects/Trace.rst
Objects/Span.rst
Objects/FeedbackScoreDict.rst
Objects/UsageDict.rst
Objects/UsageDict.rst

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
OpikTracer
==========

.. autoclass:: opik.integrations.langchain.OpikTracer
:members:
Original file line number Diff line number Diff line change
@@ -1,9 +1,34 @@
langchain
=========

Opik integrates with Langchain to allow you to log your Langchain calls to the Opik platform, simply wrap the Langchain client with `OpikTracer` to start logging::

from langchain.chains import LLMChain
from langchain_openai import OpenAI
from langchain.prompts import PromptTemplate
from opik.integrations.langchain import OpikTracer

# Initialize the tracer
opik_tracer = OpikTracer()

# Create the LLM Chain using LangChain
llm = OpenAI(temperature=0)

prompt_template = PromptTemplate(
input_variables=["input"],
template="Translate the following text to French: {input}"
)

llm_chain = LLMChain(llm=llm, prompt=prompt_template)

# Generate the translations
translation = llm_chain.run("Hello, how are you?", callbacks=[opik_tracer])
print(translation)

You can learn more about the `OpikTracer` decorator in the following section:

.. toctree::
:hidden:
:maxdepth: 4
:titlesonly:

CometTracer
OpikTracer
Original file line number Diff line number Diff line change
@@ -1,9 +1,22 @@
openai
=======

Opik integrates with OpenAI to allow you to log your OpenAI calls to the Opik platform, simply wrap the OpenAI client with `track_openai` to start logging::

from opik.integrations.openai import openai_wrapper
from openai import OpenAI

openai_client = OpenAI()
openai_client = openai_wrapper(openai_client)

response = openai_client.Completion.create(
prompt="Hello, world!",
)

You can learn more about the `track_openai` decorator in the following section:

.. toctree::
:hidden:
:maxdepth: 4
:titlesonly:

track_openai
track_openai
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
get_current_span
================

.. autofunction:: opik.opik_context.get_current_span
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
get_current_trace
=================

.. autofunction:: opik.opik_context.get_current_trace
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
opik_context
============

The opik context module provides a way to access the current span and trace from within a tracked function::

from opik import opik_context, track

@track
def my_function():
span = opik_context.get_current_span()
trace = opik_context.get_current_trace()

You can learn more about each function in the following sections:

.. toctree::
:maxdepth: 4
:titlesonly:

get_current_span
get_current_trace
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
llm_unit
========

.. autoclass:: opik.llm_unit
:members:
:inherited-members:

0 comments on commit 96581fe

Please sign in to comment.