[DATASOURCE]: Persistent data handler update #478

amadolid · 2024-06-22T07:57:04Z

CHANGES:

!!! IMPORTANT !!!

As much as possible, Anchor classes should only communicate with Anchor classes too
Architype classes will act as view of Anchor on jaclang perspective
JacFeature implementations should be the bridge between Architype and Anchors

!!! --------- !!!

ObjectAnchor is now the description of the architype in datasource perspective.
datasource uses python's shelve.Shelf if specified. Backed up by memory.
Anchors now have synced and unsynced state.
Synced Anchors are either retrieve from the datasource or newly created from jaclang
Unsynced Anchors are either from reference id or from lazy loaded anchors from existing Synced Anchors
Saving Anchors will unsync non self reference to avoid recurssion
Access validation now support blacklist and whitelist approach
Access can be associated with read or write permission and also with root or node

!!! TO FOLLOW !!!

common operation to manipulate access

!!! --------- !!!

jaclang/cli/cli.py

jaclang/core/architype.py

jaclang/core/memory.py

jaclang/core/architype.py

jaclang/plugin/tests/test_jaseci.py

jaclang/core/architype.py

jaclang/cli/cli.py

jaclang/core/architype.py

jaclang/core/memory.py

jaclang/core/architype.py

jaclang/core/memory.py

ypkang

I am okay with merging this. @marsninja if you sign off, we can 🚢 this.

jaclang/cli/cli.py

jaclang/core/context.py

jaclang/cli/cli.py

jaclang/compiler/constant.py

jaclang/plugin/default.py

jaclang/vendor/orjson/orjson.cpython-312-x86_64-linux-gnu.so

jaclang/cli/cli.py

jaclang/compiler/constant.py

CHANGES: - check if current root is super root - check if it's owned by the current root - check if target's root given access to current root - check if target's node given access to current root BEFORE: - check if current root is super root - check if it's owned by the current root - check if target's node given access to current root - check if target's root given access to current root

ypkang · 2024-08-14T19:03:13Z

jaclang/compiler/tests/test_importer.py

        return super().setUp()

    def test_import_basic_python(self) -> None:
        """Test basic self loading."""
-        Jac.context().init_memory(base_path=self.fixture_abs_path(__file__))
+        ExecutionContext.create(base_path=self.fixture_abs_path(__file__))


We should get the context from the plugin, right? just like the rest of the codebase?

will update

ypkang · 2024-08-14T19:53:15Z

jaclang/plugin/default.py

-        return Jac.context().mem
+    def current_context() -> ExecutionContext:
+        """Get current execution context."""
+        return ExecutionContext.get()


I guess the new_context vs current_context is to handle the difference between jac run via CLI and API request handling? Do we need to handle it explicitly like this? I think the ContextVar is either per thread or per task (in the case of asyncio). so it will automatically be a new one for new request.

From chatgpt

Yes, `ContextVar` in Python is designed to maintain context-specific data, and it behaves differently depending on the concurrency model. - **Thread-based concurrency:** In a traditional thread-based model, each thread has its own separate context, so `ContextVar` instances are indeed per-thread. Each thread will have its own independent value for a `ContextVar`, meaning changes to a `ContextVar` in one thread will not affect the value in another thread. - **Async concurrency (using `asyncio`):** In an asynchronous model, `ContextVar` is per-task. Each `asyncio` task has its own context, and changes to a `ContextVar` in one task will not affect the value in another task. This design ensures that `ContextVar` provides isolation across threads and tasks, making it useful for managing context-specific data in concurrent programming.

my assumptions is context should only be created per request/task and can only be read or update afterwards

in jaclang perspective,
new_context should be triggered only on cli

while current_context is for language level

amadolid mentioned this pull request Jun 22, 2024

[INPROGRESS] #429

Closed

ypkang suggested changes Jun 26, 2024

View reviewed changes

jaclang/core/architype.py Outdated Show resolved Hide resolved

jaclang/core/architype.py Outdated Show resolved Hide resolved

amadolid force-pushed the enhancements/persistent-v2 branch from 715c00a to d677938 Compare June 26, 2024 18:14

amadolid requested a review from ypkang June 27, 2024 14:38

ypkang suggested changes Jun 27, 2024

View reviewed changes

amadolid force-pushed the enhancements/persistent-v2 branch from 0511d22 to 770d8b5 Compare June 27, 2024 18:11

amadolid requested a review from ypkang June 27, 2024 18:11

amadolid force-pushed the enhancements/persistent-v2 branch 5 times, most recently from ab10a4e to a70412a Compare July 3, 2024 20:02

ypkang suggested changes Jul 4, 2024

View reviewed changes

jaclang/core/architype.py Outdated Show resolved Hide resolved

jaclang/core/architype.py Outdated Show resolved Hide resolved

jaclang/core/memory.py Outdated Show resolved Hide resolved

jaclang/core/memory.py Outdated Show resolved Hide resolved

amadolid requested a review from ypkang July 8, 2024 07:14

amadolid force-pushed the enhancements/persistent-v2 branch 2 times, most recently from bad20f1 to 30232ce Compare July 12, 2024 12:30

ypkang previously approved these changes Jul 12, 2024

View reviewed changes

amadolid dismissed ypkang’s stale review via cd22d37 July 15, 2024 06:13

amadolid force-pushed the enhancements/persistent-v2 branch 4 times, most recently from f611dbc to 0e8e210 Compare July 15, 2024 15:37

marsninja reviewed Jul 18, 2024

View reviewed changes

jaclang/cli/cli.py Outdated Show resolved Hide resolved

marsninja suggested changes Jul 18, 2024

View reviewed changes

jaclang/cli/cli.py Outdated Show resolved Hide resolved

jaclang/cli/cli.py Outdated Show resolved Hide resolved

jaclang/cli/cli.py Outdated Show resolved Hide resolved

jaclang/compiler/constant.py Outdated Show resolved Hide resolved

amadolid force-pushed the enhancements/persistent-v2 branch from 909c47c to a1f032b Compare July 18, 2024 15:07

amadolid requested review from marsninja and ypkang July 18, 2024 15:08

amadolid force-pushed the enhancements/persistent-v2 branch from e988be6 to 410bebb Compare July 18, 2024 15:49

amadolid added 18 commits August 14, 2024 12:07

[PERSISTENCE-PR-SERIES]: Rename _jac_ and target_obj

7cd6c67

[PERSISTENCE-PR-SERIES]: Context Refactor

99c1701

[PERSISTENCE-PR-SERIES]: Memory Refactor

8b70518

[PERSISTENCE-PR-SERIES]: Plugin Refactor

562cf6d

[PERSISTENCE-PR-SERIES]: Align cli, constructs and jac_streamlit

a05f47e

[PERSISTENCE-PR-SERIES]: Unit Tests

50bcb3c

[PERSISTENCE-PR-SERIES]: Auto await if necessary

9ff2555

[UPDATES]: Address PR Comments

79c1b6f

[UNIT-TEST]: Access Validation

f6ab241

[ALIGNMENT]: Sync approach to jaclang-jaseci (Context and Memory)

c540479

[BUGFIX]: Missing EdgeAnchor.is_undirected & Rename NO_ACESSS

b014909

[AUTOMATION]: Delete nodes that doesn't have any connection

4760372

[DISCONNECT]: Use EdgeDir class instead of string

db724ac

[REFACTOR]: Remove orjson and dacite

2d68e4b

[REFACTOR]: Make Root and GenericEdge to be dataclass for consistency

bdcc5f8

[REFACTOR]: Prioritize pluggy context

03533ee

[REFACTOR]: Remove ref_id prefixing

58ab554

amadolid force-pushed the enhancements/persistent-v2 branch from 235c9cc to 58ab554 Compare August 14, 2024 04:24

[MINOR]: Return current root instead of super_root on get_object root

9fbeaa7

amadolid force-pushed the enhancements/persistent-v2 branch from 0f4e5e4 to 337e394 Compare August 14, 2024 06:03

[SERIALIZER]: Adjust pickling handler

2d7b101

amadolid force-pushed the enhancements/persistent-v2 branch from 337e394 to 2d7b101 Compare August 14, 2024 07:28

[CLEANUP]: Migrate some tools to jaclang-jaseci

03dbf72

amadolid force-pushed the enhancements/persistent-v2 branch from 4eca7b4 to 03dbf72 Compare August 14, 2024 08:39

[MINOR]: Use shallow cloning instead of copy/deepcopy

d75fc1d

amadolid force-pushed the enhancements/persistent-v2 branch from 4895d09 to d75fc1d Compare August 14, 2024 13:01

ypkang suggested changes Aug 14, 2024

View reviewed changes

[UNIT-TEST]: Use JacFeature instead of ExecutionContext directly

a72b6c4

amadolid requested a review from ypkang August 15, 2024 03:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DATASOURCE]: Persistent data handler update #478

[DATASOURCE]: Persistent data handler update #478

amadolid commented Jun 22, 2024 •

edited

Loading

ypkang left a comment

ypkang Aug 14, 2024 •

edited

Loading

amadolid Aug 14, 2024

ypkang Aug 14, 2024 •

edited

Loading

amadolid Aug 14, 2024

[DATASOURCE]: Persistent data handler update #478

Are you sure you want to change the base?

[DATASOURCE]: Persistent data handler update #478

Conversation

amadolid commented Jun 22, 2024 • edited Loading

ypkang left a comment

Choose a reason for hiding this comment

ypkang Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

amadolid Aug 14, 2024

Choose a reason for hiding this comment

ypkang Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

amadolid Aug 14, 2024

Choose a reason for hiding this comment

amadolid commented Jun 22, 2024 •

edited

Loading

ypkang Aug 14, 2024 •

edited

Loading

ypkang Aug 14, 2024 •

edited

Loading