Editable install not working

[OFR-IIASA]

Up until recently, the editable install worked fine.

Using the latest ixmp version, the editable installation can be executed and installed, but the ixmp module cannot be used from other directories, except the installation directory.

Code sample or context

The code snippet shows the difference of trying to import ixmp in python when it is installed via pip install . followed by pip install -e .

(base) fricko@hpg914:~$ python -c 'import ixmp; print(ixmp.__file__)'
/h/u045/fricko/miniconda3/lib/python3.9/site-packages/ixmp/__init__.py
(base) fricko@hpg914:~$ cd Repo_hpg/ixmp
(base) fricko@hpg914:~/Repo_hpg/ixmp$ pip install -e .
Obtaining file:///h/u045/fricko/Repo_hpg/ixmp
......
Installing collected packages: ixmp
  Attempting uninstall: ixmp
    Found existing installation: ixmp 3.6.1.dev28+g0c2d7de
    Uninstalling ixmp-3.6.1.dev28+g0c2d7de:
      Successfully uninstalled ixmp-3.6.1.dev28+g0c2d7de
Successfully installed ixmp-3.6.1.dev28+g0c2d7de
(base) fricko@hpg914:~/Repo_hpg/ixmp$ cd ../..
(base) fricko@hpg914:~$ python -c 'import ixmp; print(ixmp.__file__)'
None

-->

Expected result

upon installing ixmp via pip install -e ., it should be possible to import the package in python.

Problem description

None is returned upon import.

Versions

ixmp:        3.6.1.dev28+g0c2d7de
message_ix:  3.6.1.dev14+g22de076
     88e5135 (HEAD -> correct_levelized_cost, origin/correct_levelized_cost) Correct formatting in macro related documentation
message_ix_models: installed
message_data: 2020.6.21.dev1471+g05ff5c50.d20220726
     c6d1b99f (HEAD -> issue/366, upstream/issue/366) Update ENGAGE_v4.1.8 workflow to use latest GLOBIOM matrix

click:       8.0.3
dask:        2022.05.0
genno:       installed
graphviz:    0.20
jpype:       1.3.0
… JVM path:  /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
openpyxl:    3.0.9
pandas:      1.4.2
pint:        0.18
xarray:      2022.3.0
yaml:        6.0

iam_units:   installed
jupyter:     1.0.0
matplotlib:  3.5.2
plotnine:    0.8.0
pyam:        1.4.0

GAMS:        32.1.0

python:      3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:37)
[GCC 9.4.0]
python-bits: 64
OS:          Linux
OS-release:  5.4.0-120-generic
machine:     x86_64
processor:   x86_64
byteorder:   little
LC_ALL:      None
LANG:        en_US.UTF-8
LOCALE:      ('en_US', 'UTF-8')

[gidden]

Hi @OFR-IIASA - based on the output here it looks like ixmp is installed, but perhaps not returning an expected output from __file__.

The best way to test that everything works on a given system is to run the tests. Can you confirm if those pass or not? It would be good to confirm that the local editable installed version is the only ixmp version installed on the machine.

[OFR-IIASA]

Correct.

Using the editable option, all tests pass except for ixmp/tests/test_tutorials.py, which yields the following error:

>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           # launch the ix modeling platform using the default local database
E           mp = ixmp.Platform()
E           ------------------
E
E           ---------------------------------------------------------------------------
E           AttributeError                            Traceback (most recent call last)
E           Input In [2], in <cell line: 2>()
E                 1 # launch the ix modeling platform using the default local database
E           ----> 2 mp = ixmp.Platform()
E
E           AttributeError: module 'ixmp' has no attribute 'Platform'
E           AttributeError: module 'ixmp' has no attribute 'Platform'

../../miniconda3/lib/python3.9/site-packages/nbclient/client.py:906: CellExecutionError

with the regular install, no errors occur.

[OFR-IIASA]

I have now tested older versions of ixmp. This same problem applies. Maybe someone else try on hpg914?
I have also tested this on my mac, no problems there - all test pass.
I am currently running scenarios on windows, so no chance to check at the moment.

[OFR-IIASA]

So, I have resolved the issue by going to the directory where ixmp is installed in the case of a non-editable installation, based on the above this would be /h/u045/fricko/miniconda3/lib/python3.9/site-packages, deleted all ixmp related files and then re-ran the installation process.
This may therefore only be an issues when upgrading ixmp versions. On the case above where I tested on mac, the installation was also a clean install.
Maybe therefore someone can see if they can replicate the error by updating their ixmp package and installing it as editable.

[awais307]

@OFR-IIASA should this issue be raised in the weekly list or you think it is resolved and closed.

[jansteinhauser]

Hi, had a similar issue today when trying to install ixmp as editable. Despite getting a "successfully installed" message, it was not found by message-ix and running ixmp show-versions resulted in an error as well.

I tried both fixes suggested by @OFR-IIASA and both worked (individually):

• re-installing from source but not as editable
• finding my site-packages and removing the ixmp folders there.

Likely, this issue resulted from an earlier message-ix installation where ixmp was installed as dependency.

[glatterf42]

Reading about editable installs, there is a direct mention of these specific variables in the 'Limitations' section:

There is no guarantee that attributes like path or file will correspond to the exact location of the original files (e.g., setuptools might employ file links to perform the editable installation). Users are encouraged to use tools like importlib.resources or importlib.metadata when trying to access package files directly.

This makes me think that we're not seeing unintended behaviour here. And indeed, pip install -e . in a fresh virtual environment already "breaks" the install: when running python -c 'import ixmp; print(ixmp.__file__)' inside the ixmp directory, it works, but it doesn't outside of it. Possible solutions as to fix this could be inferred from the site linked above: using one of the mentioned tools to access these variables, or by setting the editable_mode to strict or compat. Both of the latter might stray from the intended behaviour of an editable install, though.
As mentioned by @gidden, the best way to test if your installation works, is by using pytest (not dunder variables). To enable that, run pip install -e .[tests]. Then, pick a random test (e.g., test_utils.py) and run pytest. From within the ixmp directory: pytest ixmp/tests/test_utils.py. And from outside: pytest <path-to-ixmp>/ixmp/tests/test_utils.py. Both should produce something like 15 passed, 4 xfailed, 6 warnings in 6.03s for our chosen example.

Could you please check if these tests are working?

[glatterf42]

When running test_tutorials.py in the same manner, I get 2 passed, 2 deselected, 10 warnings in 19.26s. Your error message from jupyter notebook, @OFR-IIASA, sounds to me like jupyter is using a wrong version of ixmp. To confirm that, you can add a line like print(ixmp.__path__) to the tutorial and compare the stated path to the path you find in the virtual environment.

[glatterf42]

After installing message_ix with a fresh clone via pip install -e .[docs,reporting,tests,tutorial], I receive the following output from both ixmp show-versions and message-ix show-versions:

ixmp:        3.6.1.dev63+ge490f48
     e490f48 (HEAD -> main, origin/main, origin/HEAD) Merge pull request #471 from iiasa/pandas-2.0.0
message_ix:  3.6.1.dev83+g077c407
     077c407 (HEAD -> main, origin/main, origin/HEAD) Retain meaning on typo fix
message_ix_models: None
message_data: None

click:       8.1.3
dask:        2023.3.1
genno:       installed
graphviz:    0.20.1
jpype:       1.4.1
… JVM path:  /usr/lib/jvm/java-19-openjdk-amd64/lib/server/libjvm.so
openpyxl:    3.1.2
pandas:      1.5.3
pint:        0.20.1
xarray:      2023.3.0
yaml:        6.0

iam_units:   installed
jupyter:     1.0.0
matplotlib:  3.7.0
plotnine:    0.10.1
pyam:        1.8.0

GAMS:        41.5.0

python:      3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0]
python-bits: 64
OS:          Linux
OS-release:  5.19.0-35-generic
machine:     x86_64
processor:   x86_64
byteorder:   little
LC_ALL:      None
LANG:        en_US.UTF-8
LOCALE:      ('en_US', 'UTF-8')

So ixmp seems to have been detected as intended. The tests also run without errors for me. Could you please provide more detailed information about your error messages, @jansteinhauser?

[jansteinhauser]

Hi, tests are working fine for me (if I ignore the 4 xfailed). show-versions also works fine.
I can't tell you what the exact message was, as it's now fixed.
Essentially, when running message-ix show-versions, ixmp was 'not found', though I'm not sure if that's just a bad message for ixmp actually running with an error.
I think so because if I ran ixmp show-versions, ixmp did indeed start and threw an error along the lines of 'ixmp does not have an argument scenario' (or something similar).

In general, my installation process leading here was:

1. Install message via pip
   That worked (I was able to run tests and tutorials), but I realized I wanted to install from source after all.
2. Install ixmp and message from source (plus pyam, message_data, message-ix-models).
   Only now did I phase the described issue. After several pip uninstall/install attempts, I was told about this issue and Oliver's solutions. Checking site-packages, I found two ixmp folders, ixmp and ixmp-3.6.1.dev63
3. Remove the ixmp folders (though it probably would have sufficed to just remove ixmp specifically) in site-packages, reinstall ixmp from source

Maybe this helps with pinpointing the issue (or if this is an issue at all)

[glatterf42]

You can safely ignore the xfails as they are expected to fail. So it's good to hear that your issue is fixed now. Thanks also for your instructions on how to reproduce the problem. (Un-)Fortunately, I can't reproduce it. So here's what I ran:

1. Get fresh clones of ixmp and message_ix
2. Create a new virtual environment
3. pip install message_ix
4. Change to ixmp directory; pip install -e .[tests]
5. Change to message_ix directory; pip install -e .[docs,reporting,tests,tutorial]

ixmp show-versions works as expected and the tests are also running. What strikes me in your description is

After several pip uninstall/install attempts

because as you can see from my list, there is no need to explicitly uninstall one of the packages. And it shouldn't take multiple installs either, so my current best guess is that at some point in these attempts, you installed different versions of ixmp, and your python didn't know which to use/used the wrong one. Performing a fresh editable install works for me, either way, and even changing from a pip installation to 'from source' seems to work for me by following the steps I outlined above.

[jansteinhauser]

Just for the record: The uninstall/install attempts started only after I got the message-ix/ixmp errors described. They were an attempt to back to a clean slate and reinstall properly, but didn't change anything for the better.

As far as I can tell, the issue popped up when I had two ixmp folders installed, so after the first installation from source. So, I would agree that it's due to different versions being installed, albeit both versions individually seem to work as intended (the first one did so before I installed from source, the second one did after I removed the first one)

Long story short: We can't reproduce my issue, I guess, but hopefully it's not something that'll likely happen again. Thanks for looking into it!

[glatterf42]

Sounds good. Just for future reference, we recently dealt with a related issue where jupyter was also not using the correct version of ixmp when multiple were installed on slack.
If there is nothing you want to add, @OFR-IIASA, I think we can close this issue.

[glatterf42]

Today, I finally ran into the same problem! After switching from a 'normal' pip install ixmp to an editable one, both ixmp show-versions and the tutorial Jupyter notebooks were not working as described above.

Through printing ixmp.__path__ (I believe), I found the location of the version of ixmp that was in use both in Jupyter and outside of it. Both pointed me to <path_to_venv>/lib/python3.10/site-packages/ixmp, but outside of Jupyter, the search path also included /home/fridolin/ixmp/, i.e. the path to the fresh clone of the main repository. Checking the site-packages, I indeed found two folders referencing ixmp, one for the 'normal' version and one dist-info folder for the editable version.
The 'normal' folder contained nothing apart from testing/__pycache__/, which, in turn, contained some Cpython and Jupyter .pyc files. Since these were the only files, no file provided objects like Platform or Scenario to the package, so they were flagged as missing. For whatever reason, these .pyc files had not been deleted during the uninstalling process that pip performs per default when upgrading a package. One guess of mine is that Ubuntu might block them from being deleted since this might disturb the ixmp terminal command, but it would then be weird that a Jupyter file should have been kept. Unfortunately, even after some hours of looking into this, I couldn't find a good reason for why these files were still there and how they could be deleted automatically in the future. It might work (though I can't try that anymore right now) to run pip uninstall ixmp before attempting to install the editable version. It might also, generally, be best to switch to a new venv and install ixmp and message_ix as editable from fresh clones there.
I also checked pip cache list to see if these files were simply still stored in my cache, but they weren't.
Either way, I started by running pip uninstall ixmp. Again, this did not remove the above files, so I simply deleted their whole directory. From my research online it seems to me that if you are trying to do the same, you should delete mentions of ixmp from potential .egg files and easy-install.pth, should you have them (I didn't). They might also hide in a dist-packages directory at the same level as site-packages (which didn't exist for me either). Once there was nor more mention of ixmp in my site-packages, I re-installed it from my fresh clone. Now, ixmp show-versions and the tutorial tests work fine, and Jupyter shows the correct ixmp.__path__. Fun fact: it even shows the correct ixmp.__file__.

So, if you installed ixmp at some point and now want to switch to an editable version, the best advice would be to use a new virtual environment. If that's not an option, uninstall ixmp completely before installing as editable, checking the site-packages manually. No pytest test should fail after your installation, nor should ixmp show-versions.

If you have any idea of what is causing this error (probably some interplay of pip, jupyter, and ubuntu), please let us know.

[glatterf42]

@khaeru, this is largely for the record and for your information, it might also contain a requirement that we would want our packaging tools to satisfy.

Today, @th-hara pointed me to another issue: even after following the steps as described above, from ixmp import Platform does not work in python terminals started from outside of the ixmp directory:

fridolin@fridolin-Latitude-5520 ~> (message_ix) python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ixmp import Platform
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Platform' from 'ixmp' (unknown location)
>>> quit()
fridolin@fridolin-Latitude-5520 ~> (message_ix) cd ixmp
fridolin@fridolin-Latitude-5520 ~/ixmp (main) [1]> (message_ix) python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ixmp import Platform
>>> quit()

In pip's documentation on editable installs, there is a note that the behaviour of the installs depends on the build-backend. For ixmp, this is setuptools-scm, but for ixmp4, this is poetry_dynamic_versioning.backend. So I tried (using a new venv) to pip install ixmp4, git clone <ixmp4>, pip install -e ., and I found:

fridolin@fridolin-Latitude-5520 ~/t/ixmp4 (main)> (test_ixmp4) python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ixmp4 import Run
>>> quit()
fridolin@fridolin-Latitude-5520 ~/t/ixmp4 (main)> (test_ixmp4) cd ..
fridolin@fridolin-Latitude-5520 ~/tmp> (test_ixmp4) python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ixmp4 import Run
>>> quit()

I.e., the poetry build-backend does not seem to struggle with editable installs being called from outside the package's directory. We would have to try if the same holds for pdm.