Support for Python and Pip

[fralik]

It would be great to have support for Python/Pip as well.

I did some experiments around it and looks like it is possible to have a script similar to write-npm.sh. It could be write-pip.sh:

#!/bin/bash

ARTIFACTS_FEED=${1:-"required"}
FEED_USER=""

# If ARTIFACTS_FEED equals "required" then exit with error message
if [ "${ARTIFACTS_FEED}" = "required" ]; then
    echo "  Usage: write-pip.sh <ARTIFACTS_FEED>"
    echo "example: write-pip.sh pkgs.dev.azure.com/orgname/projectname/_packaging/feedname/pypi/simple"
    echo "IMPORTANT! Do not pass https in artifacts feed!"
    exit 1
fi

pip config set --site global.index-url https://user:\${ARTIFACTS_ACCESSTOKEN}@${ARTIFACTS_FEED}

Pip doesn't support providing auth token independent of feed url. It supports keyring, but my understanding that it is impossible to use keyring in dev container in a non-interactive manner. See https://github.com/jaraco/keyring#using-keyring-on-headless-linux-systems-in-a-docker-container

Running pip via run-pip.sh could be possible as well. In this case we will need a way to read/obtain feed url. Then we may set the environment variable PIP_INDEX_URL,

[markphip]

Correct me if I am wrong, but this is going to store the value of the ARTIFACTS_ACCESSTOKEN environment variable in the config. If so, that will not work and is not how this works.

The tokens that are obtained are short lived, which is why for the package managers we are supporting so far we have to replace the command like dotnet with a bash alias that runs a script. The script then gets the token, updates an environment variable and runs the tool. The tools that work are the ones support getting the value from the environment variable.

Can Python/Pip be configured to get the token from an environment variable? Where we tell it the name of the variable and it reads the value?

The write-npm.sh script is just a small helper to write out a npm config file in the format needed to reference an environment variable. This is mainly only provided because the npm config file is more than a one-liner.

[fralik]

ah, I see about the short living token. You are correct that it won't work in the long run.

There is an option to do it via run script:

#!/bin/bash

if [ -f "${HOME}/ado-auth-helper" ]; then
  export ARTIFACTS_ACCESSTOKEN=$(${HOME}/ado-auth-helper get-access-token)
fi

# Read index-url from pip config get
INDEX_URL=$(pip config get global.index-url)
EXPANDED_INDEX_URL=$(eval echo ${INDEX_URL})
if [ -f "${HOME}/ado-auth-helper" ]; then
    unset ARTIFACTS_ACCESSTOKEN
fi

# Find the pip executable so we do not run the bash alias again
PIP_EXE=$(which pip)

${PIP_EXE} "$@" -i "${EXPANDED_INDEX_URL}"
EXIT_CODE=$?
unset PIP_EXE

if [ -f "${HOME}/ado-auth-helper" ]; then
    unset EXPANDED_INDEX_URL
fi

exit $EXIT_CODE

this would require that Pip is configured like this: pip config set --site global.index-url https://user:${ARTIFACTS_ACCESSTOKEN}:url/pypi/simple, i.e. that it references an environment variable.

[markphip]

Just so I understand ... now you are storing the name of the environment variable in the configuration but since Pip does not support this you dynamically retrieve and expand the value then set it on the command line when running pip?

I still see a couple of problems:

1. If someone runs python -m pip XXX then I do not expect a bash alias will work. We see this problem with similar scenarios for npm
2. Many of the pip subcommands do not support -i so presumably we would need to interpret which subcommand is being run and only selectively add it to the command line

I do not think the first problem can be solved. I also do not think it has to block a solution. It may just be something users need to be aware of

[fralik]

You are correct in your understanding.

1. That is true.
2. This is also true. I got too focused on installing command so that I forgot about the rest.

There is potential solution similar to the above run script. There, we would put auth token in a keyring. Run pip command (it can be any command; if it doesn't need auth, then it will not read it from the keyring). Finally, we remove the token from keyring. The problem with this solution is that there is no easy-usable keyring backend implementation.

[markphip]

I do not have the Python skills for this but looking at the documentation for building a custom keyring backend it looks doable. Wondering if the code for that backend could live in this repository and during install time we configure pip to use it? If configured of course.

Presumably it would not even need to store anything and just has to respond to the getPassword or getCredential method with the results of calling the script?

[delilahw]

I quite like the keyring backend idea. I imagine it would be similar to the implementation in https://github.com/microsoft/artifacts-keyring/blob/master/src/artifacts_keyring/plugin.py, but differing in the underlying executable that we call.

Is anyone working on this atm?

[delilahw]

Update: Got proof of concept working locally. Keen to get this one reviewed and released. I'll reach out, let's discuss on Teams next week!

[markphip]

Feel free to add a PR with your PoC. If you explain what it is doing and what needs to be done at install time, I can probably take care of the glue code for the feature to install and set it up etc. It is pretty straightforward so if you want to tackle that too feel free.

It is hard to test the full final version but usually you can just create a Codespace and manually do some of the steps the install will do to verify it works. That is how I did it for the other providers.

[markphip]

@delilahw I published a new version that installs the helper you created. Can you give it a test? I am only testing that the install completes, not the actual usage. To try it you need this:

  "features": {
    "ghcr.io/microsoft/codespace-features/artifacts-helper:1": {
      "python": true
    }
  }

The installation only happens if it also detects pip is available. Let me know if we need to also handle looking for and using something like pip3

[delilahw]

@delilahw I published a new version that installs the helper you created. Can you give it a test? I am only testing that the install completes, not the actual usage. To try it you need this:

@markphip Thanks!! I've just tested it with a config like this:

{
    "image": "mcr.microsoft.com/devcontainers/base:ubuntu-22.04",
    "features": {
        "ghcr.io/devcontainers/features/python:1": {
            "version": "3.11"
        },
        "ghcr.io/microsoft/codespace-features/artifacts-helper:1": {
            "python": true
        }
    }
}

It downloads the wheel successfully but fails to find pip. I think we do need to handle both python and python3 as they vary between distros. The pip docs now recommend explicitly specifying the python interpreter using python -m pip, rather than just pip, to remove any ambuiguity.

#23 3.392 Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
#23 4.428 2024-05-07 23:11:33 (55.0 MB/s) - 'codespaces_artifacts_helper_keyring-0.1.0-py3-none-any.whl' saved [6613/6613]

#23 4.430 /tmp/install-python-keyring.sh: line 5: pip: command not found

#23 4.431 ERROR: Feature "Azure Artifacts Credential Helper" (ghcr.io/microsoft/codespace-features/artifacts-helper) failed to install!