Too much restriction on pyarrow dependency

[FlorianJacta]

What went wrong? 🤔

There is a breaking change in dependencies going from 3.1.1 to 4.0 by a restriction on pyarrow.

Packages like databricks-sql-connector==3.4.0 cannot be installed now because of this conflict.

INFO: pip is looking at multiple versions of taipy to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install databricks-sql-connector==3.4.0 and taipy==4.0.0.dev4 because these package versions have conflicting dependencies.

The conflict is caused by:
    databricks-sql-connector 3.4.0 depends on pyarrow<17 and >=14.0.1
    taipy 4.0.0.dev4 depends on pyarrow<=17.9.9 and >=17.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Expected Behavior

We should be able to install Taipy alongside databricks-sql-connector==3.4.0.

Steps to Reproduce Issue

pip install git+https://github.com/Avaiga/taipy.git databricks-sql-connector==3.4.0

Version of Taipy

develop - 10/9/24

Acceptance Criteria

• Ensure new code is unit tested, and check code coverage is at least 90%.
• Create related issue in taipy-doc for documentation and Release Notes.

Code of Conduct

• I have checked the existing issues.
• I am willing to work on this issue (optional)

[SaumyaSubham]

Hii @FlorianJacta
I think loosening the pyarrow version range is a viable solution, I can work for this issue.
Kindly assign me this issue under hacktoberfest 2024

[FlorianJacta]

This is not yet open for contribution. We will assign you if it is. Thank you @SaumyaSubham !

[FredLL-Avaiga]

What is databricks take on this ?
Do they have a planning for supporting Arrow version 17+ ?

[FlorianJacta]

I have found this: databricks/databricks-sql-python#427

[jrobinAV]

@FredLL-Avaiga DO you remember any reason to update the minimal supported version?
Supporting 17+ only is very restrictive. I would like to avoid so much restriction.

[jrobinAV]

@SaumyaSubham
I labeled this issue as staff only because we need to ensure the Pyarrow version of the Taipy Community edition matches the version of the Taipy Enterprise Edition. Indeed, Taipy Enterprise has a native integration with Databricks. This requires to have access to some private resources.

Thank you for your understanding.

[FredLL-Avaiga]

@FredLL-Avaiga DO you remember any reason to update the minimal supported version?
Supporting 17+ only is very restrictive. I would like to avoid so much restriction.

I suppose for the time being we can support 16+
There was no API changes in front or back

[toan-quach]

@S-YED thanks for your contribution!

Seems to me that the databricks team is working on a change that will remove the upper bound for dependencies => You will be able to install taipy (even with pyarrow>=17.0.0,<18.0) after their PR is merged.
databricks/databricks-sql-python#452

However, this may take a while, so I would suggest 2 approaches:

1. @S-YED can you taipy and databricks-sql-connector by installing them separately

pip install git+https://github.com/your-username/taipy.git
pip install databricks-sql-connector-3.4.0

2. @jrobinAV we can expand pyarrow on our side to include 16.0 as the baseline as 16.0 is acceptable by databricks-sql-connector

[jrobinAV]

Sounds fair.

[jrobinAV]

@toan-quach
THe related PR #1988 has been closed without merging. PLease proceed with the issue by your own. Thank you.