Pipeline from config file (#220)

* SimpleKGPipeline config parser * Factorize common parmeters * Fix * Fix again (ruff) * Add builder, docstrings * Adds example * Ruff * Add headers * Another header * Remove old file - more detailed example * Fix JSON * WIP * Adds more param resolvers * A bit of mypy * Add root types to allow instantiation from python object directly * Document + mypy * Add embedder config * Implement SimpleKGBuilder with this setup * Add YAML config example * Example config files for custom pipeline * Update SimpleKGPipeline * Fix UT * Fix tests * Simplify and mypy * ruff * Add missing dep * Missing import for '|' annotations * Restructure files + increase tests coverage * mypy * Refactor examples * Test runner, clean simple kg builder test (remove duplicates) * E2E tests * Use fsspec in config reader * Changelog * Fix test * Use cast to remove a type ignore comment * Also use cast here to remove type ignore * Close instantiated drivers * ruff * Adding loggers * Make close function async * Fix tests * fix UT * Write doc about SimpleKGPipeline and config files * Update api.rst * Recreate lock file after merge * Add more comments to explain Config/Type models
neo4j · Dec 12, 2024 · ff6862e · ff6862e
1 parent b2ba9a3
commit ff6862e
Show file tree

Hide file tree

Showing 56 changed files with 3,721 additions and 551 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,7 @@
 ## Added
 - Integrated json-repair package to handle and repair invalid JSON generated by LLMs.
 - Introduced InvalidJSONError exception for handling cases where JSON repair fails.
+- Ability to create a Pipeline or SimpleKGPipeline from a config file. See [the example](examples/build_graph/from_config_files/simple_kg_pipeline_from_config_file.py).
 
 ## Changed
 - Updated LLM prompts to include stricter instructions for generating valid JSON.

diff --git a/docs/source/api.rst b/docs/source/api.rst
@@ -9,17 +9,18 @@ API Documentation
 Components
 **********
 
-KGWriter
-========
+DataLoader
+==========
 
-.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.KGWriter
-    :members: run
+.. autoclass:: neo4j_graphrag.experimental.components.pdf_loader.DataLoader
+    :members: run, get_document_metadata
 
-Neo4jWriter
-===========
 
-.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.Neo4jWriter
-    :members: run
+PdfLoader
+=========
+
+.. autoclass:: neo4j_graphrag.experimental.components.pdf_loader.PdfLoader
+    :members: run, load_file
 
 TextSplitter
 ============
@@ -85,6 +86,17 @@ LLMEntityRelationExtractor
 .. autoclass:: neo4j_graphrag.experimental.components.entity_relation_extractor.LLMEntityRelationExtractor
     :members: run
 
+KGWriter
+========
+
+.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.KGWriter
+    :members: run
+
+Neo4jWriter
+===========
+
+.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.Neo4jWriter
+    :members: run
 
 SinglePropertyExactMatchResolver
 ================================
@@ -112,6 +124,23 @@ SimpleKGPipeline
     :members: run_async
 
 
+************
+Config files
+************
+
+
+SimpleKGPipelineConfig
+======================
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.template_pipeline.simple_kg_builder.SimpleKGPipelineConfig
+
+
+PipelineRunner
+==============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.runner.PipelineRunner
+
+
 .. _retrievers-section:
 
 **********

diff --git a/docs/source/images/kg_builder_pipeline.png b/docs/source/images/kg_builder_pipeline.png
diff --git a/docs/source/types.rst b/docs/source/types.rst
@@ -82,3 +82,62 @@ SchemaConfig
 ============
 
 .. autoclass:: neo4j_graphrag.experimental.components.schema.SchemaConfig
+
+LexicalGraphConfig
+===================
+
+.. autoclass:: neo4j_graphrag.experimental.components.types.LexicalGraphConfig
+
+
+Neo4jDriverType
+===============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.Neo4jDriverType
+
+
+Neo4jDriverConfig
+=================
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.Neo4jDriverConfig
+
+
+LLMType
+=======
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.LLMType
+
+
+LLMConfig
+=========
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.LLMConfig
+
+
+EmbedderType
+============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.EmbedderType
+
+
+EmbedderConfig
+==============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.EmbedderConfig
+
+
+ComponentType
+=============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.ComponentType
+
+
+ComponentConfig
+===============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.ComponentConfig
+
+
+ParamFromEnvConfig
+==================
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.param_resolver.ParamFromEnvConfig