Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline from config file #220

Merged
merged 67 commits into from
Dec 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
e9712a9
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 15, 2024
b52c45e
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 16, 2024
84c1780
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 17, 2024
47d4782
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 21, 2024
bc7a2f9
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 22, 2024
a945284
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 22, 2024
4e13c23
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 23, 2024
5367bed
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 24, 2024
21d1223
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 25, 2024
3329cd7
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 25, 2024
d8f6364
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Oct 28, 2024
4cec2f3
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Nov 4, 2024
4445b49
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Nov 5, 2024
939b18c
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python
stellasia Nov 18, 2024
6437fe7
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python…
stellasia Nov 25, 2024
e92d7bb
SimpleKGPipeline config parser
stellasia Nov 27, 2024
221c734
Factorize common parmeters
stellasia Nov 27, 2024
9813743
Fix
stellasia Nov 27, 2024
fb5939a
Fix again (ruff)
stellasia Nov 27, 2024
45dd67e
Add builder, docstrings
stellasia Nov 28, 2024
53aba3a
Adds example
stellasia Nov 28, 2024
ba56f44
Ruff
stellasia Nov 28, 2024
cc85fb0
Add headers
stellasia Nov 28, 2024
62ae4e1
Another header
stellasia Nov 28, 2024
4b80926
Remove old file - more detailed example
stellasia Nov 28, 2024
4d8eddb
Fix JSON
stellasia Nov 28, 2024
536c9ff
WIP
stellasia Nov 30, 2024
668fbc2
Adds more param resolvers
stellasia Dec 1, 2024
1d884a1
A bit of mypy
stellasia Dec 1, 2024
e2eb5d0
Add root types to allow instantiation from python object directly
stellasia Dec 3, 2024
390be0c
Document + mypy
stellasia Dec 3, 2024
adfc908
Add embedder config
stellasia Dec 3, 2024
ecda174
Implement SimpleKGBuilder with this setup
stellasia Dec 3, 2024
ab46edb
Add YAML config example
stellasia Dec 4, 2024
9dc8bec
Example config files for custom pipeline
stellasia Dec 4, 2024
d551fb4
Update SimpleKGPipeline
stellasia Dec 4, 2024
b070f6b
Fix UT
stellasia Dec 4, 2024
d192667
Fix tests
stellasia Dec 4, 2024
137cd76
Simplify and mypy
stellasia Dec 4, 2024
260478a
ruff
stellasia Dec 4, 2024
35133dd
Add missing dep
stellasia Dec 4, 2024
b5e6471
Missing import for '|' annotations
stellasia Dec 4, 2024
30b878d
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python…
stellasia Dec 4, 2024
a7d37e4
Restructure files + increase tests coverage
stellasia Dec 5, 2024
99af0de
mypy
stellasia Dec 5, 2024
3632557
Refactor examples
stellasia Dec 5, 2024
3f8fbf4
Test runner, clean simple kg builder test (remove duplicates)
stellasia Dec 5, 2024
3f6a9e1
E2E tests
stellasia Dec 5, 2024
30543dc
Use fsspec in config reader
stellasia Dec 5, 2024
ccb46f6
Changelog
stellasia Dec 5, 2024
aedd9a3
Fix test
stellasia Dec 5, 2024
3122eda
Use cast to remove a type ignore comment
stellasia Dec 6, 2024
ce21353
Also use cast here to remove type ignore
stellasia Dec 6, 2024
4e04d04
Close instantiated drivers
stellasia Dec 6, 2024
1e1ce2a
ruff
stellasia Dec 6, 2024
8834351
Adding loggers
stellasia Dec 6, 2024
82b4598
Make close function async
stellasia Dec 6, 2024
ede69f1
Fix tests
stellasia Dec 6, 2024
6613451
fix UT
stellasia Dec 6, 2024
aeea211
Write doc about SimpleKGPipeline and config files
stellasia Dec 6, 2024
83f5133
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python…
stellasia Dec 6, 2024
76434ec
Update api.rst
stellasia Dec 6, 2024
2bebbfe
Merge branch 'main' of https://github.com/neo4j/neo4j-graphrag-python…
stellasia Dec 11, 2024
54a55ae
Recreate lock file after merge
stellasia Dec 11, 2024
8572b42
Merge branch 'main' into feature/config-files
stellasia Dec 12, 2024
94483d8
Merge branch 'feature/config-files' of https://github.com/stellasia/n…
stellasia Dec 12, 2024
663df47
Add more comments to explain Config/Type models
stellasia Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
## Added
- Integrated json-repair package to handle and repair invalid JSON generated by LLMs.
- Introduced InvalidJSONError exception for handling cases where JSON repair fails.
- Ability to create a Pipeline or SimpleKGPipeline from a config file. See [the example](examples/build_graph/from_config_files/simple_kg_pipeline_from_config_file.py).

## Changed
- Updated LLM prompts to include stricter instructions for generating valid JSON.
Expand Down
45 changes: 37 additions & 8 deletions docs/source/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,18 @@ API Documentation
Components
**********

KGWriter
========
DataLoader
==========

.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.KGWriter
:members: run
.. autoclass:: neo4j_graphrag.experimental.components.pdf_loader.DataLoader
:members: run, get_document_metadata

Neo4jWriter
===========

.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.Neo4jWriter
:members: run
PdfLoader
=========

.. autoclass:: neo4j_graphrag.experimental.components.pdf_loader.PdfLoader
:members: run, load_file

TextSplitter
============
Expand Down Expand Up @@ -85,6 +86,17 @@ LLMEntityRelationExtractor
.. autoclass:: neo4j_graphrag.experimental.components.entity_relation_extractor.LLMEntityRelationExtractor
:members: run

KGWriter
========

.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.KGWriter
:members: run

Neo4jWriter
===========

.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.Neo4jWriter
:members: run

SinglePropertyExactMatchResolver
================================
Expand Down Expand Up @@ -112,6 +124,23 @@ SimpleKGPipeline
:members: run_async


************
Config files
************


SimpleKGPipelineConfig
======================

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.template_pipeline.simple_kg_builder.SimpleKGPipelineConfig


PipelineRunner
==============

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.runner.PipelineRunner


.. _retrievers-section:

**********
Expand Down
Binary file modified docs/source/images/kg_builder_pipeline.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
59 changes: 59 additions & 0 deletions docs/source/types.rst
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,62 @@ SchemaConfig
============

.. autoclass:: neo4j_graphrag.experimental.components.schema.SchemaConfig

LexicalGraphConfig
===================

.. autoclass:: neo4j_graphrag.experimental.components.types.LexicalGraphConfig


Neo4jDriverType
===============

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.Neo4jDriverType


Neo4jDriverConfig
=================

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.Neo4jDriverConfig


LLMType
=======

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.LLMType


LLMConfig
=========

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.LLMConfig


EmbedderType
============

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.EmbedderType


EmbedderConfig
==============

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.EmbedderConfig


ComponentType
=============

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.ComponentType


ComponentConfig
===============

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.ComponentConfig


ParamFromEnvConfig
==================

.. autoclass:: neo4j_graphrag.experimental.pipeline.config.param_resolver.ParamFromEnvConfig
Loading
Loading