-
Notifications
You must be signed in to change notification settings - Fork 492
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add trigger for building Linux wheels #663
Changes from 21 commits
128c30c
e94bfad
806501f
0a8d2b3
ffea48b
ec63c3f
d5d3e5e
eecaf4a
dd763d7
147a3ba
35ec67e
cb30f55
bb37b2a
aa1efb4
b722b5f
a9653eb
4b9304a
46f8180
aeb8345
87a38a2
296eba9
8b75669
cecb8d3
3b4e3bd
0659fdc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,175 @@ | ||
# Heavy inspiration taken from: | ||
# https://github.com/pytorch/test-infra/blob/main/.github/workflows/build_wheels_linux.yml | ||
|
||
name: Build Linux Wheels | ||
|
||
|
||
on: | ||
pull_request: | ||
push: | ||
branches: | ||
- nightly | ||
- main | ||
- release/* | ||
tags: | ||
# NOTE: Binary build pipelines should only get triggered on release candidate builds | ||
# Release candidate tags look like: v1.11.0-rc1 | ||
- v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+ | ||
workflow_dispatch: | ||
|
||
|
||
permissions: | ||
id-token: write | ||
contents: read | ||
|
||
|
||
jobs: | ||
setup-variables: | ||
runs-on: ubuntu-latest | ||
outputs: | ||
TEST_INFRA_REPOSITORY: pytorch/test-infra | ||
TEST_INFRA_REF: main | ||
REPOSITORY: pytorch/torchtune | ||
REF: pkg-for-wheel | ||
PACKAGE_NAME: torchtune | ||
PACKAGE_TYPE: wheel | ||
ARCH: x86_64 | ||
SMOKE_TEST_SCRIPT: "" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we had talked about potentially adding sanity checks of importing from the library or something like that here. Any plans to do that in this PR, or should we just do it later on? |
||
steps: | ||
- run: echo "Exposing env vars to all jobs" | ||
generate-matrix: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Any value to just adding a comment at the top of this job about which versions we should be building so folks don't have to go to the test-infra yaml to find this stuff? |
||
needs: | ||
- setup-variables | ||
uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main | ||
with: | ||
package-type: ${{ needs.setup-variables.outputs.PACKAGE_TYPE }} | ||
os: linux | ||
test-infra-repository: ${{ needs.setup-variables.outputs.TEST_INFRA_REPOSITORY }} | ||
test-infra-ref: ${{ needs.setup-variables.outputs.TEST_INFRA_REF }} | ||
with-cuda: disable | ||
with-rocm: disable | ||
build: | ||
needs: | ||
- setup-variables | ||
- generate-matrix | ||
strategy: | ||
fail-fast: false | ||
matrix: ${{ fromJSON(needs.generate-matrix.outputs.matrix) }} | ||
env: | ||
PYTHON_VERSION: ${{ matrix.python_version }} | ||
CU_VERSION: ${{ matrix.desired_cuda }} | ||
UPLOAD_TO_BASE_BUCKET: ${{ matrix.upload_to_base_bucket }} | ||
name: ${{ matrix.build_name }} | ||
runs-on: ${{ matrix.validation_runner }} | ||
container: | ||
image: ${{ matrix.container_image }} | ||
options: ${{ matrix.gpu_arch_type == 'cuda' && '--gpus all' || ' ' }} | ||
# If a build is taking longer than 60 minutes on these runners we need | ||
# to have a conversation | ||
timeout-minutes: 120 | ||
steps: | ||
- name: Clean workspace | ||
shell: bash -l {0} | ||
run: | | ||
set -euxo pipefail | ||
echo "::group::Cleanup debug output" | ||
rm -rf "${GITHUB_WORKSPACE}" | ||
mkdir -p "${GITHUB_WORKSPACE}" | ||
echo "::endgroup::" | ||
- uses: actions/checkout@v3 | ||
with: | ||
repository: ${{ needs.setup-variables.outputs.TEST_INFRA_REPOSITORY }} | ||
ref: ${{ needs.setup-variables.outputs.TEST_INFRA_REF }} | ||
path: test-infra | ||
- uses: pytorch/test-infra/.github/actions/set-channel@main | ||
- name: Set PYTORCH_VERSION | ||
if: ${{ env.CHANNEL == 'test' }} | ||
run: | | ||
# When building RC, set the version to be the current candidate version, | ||
# otherwise, leave it alone so nightly will pick up the latest | ||
echo "PYTORCH_VERSION=${{ matrix.stable_version }}" >> "${GITHUB_ENV}" | ||
- uses: pytorch/test-infra/.github/actions/setup-binary-builds@main | ||
env: | ||
PLATFORM: ${{ env.ARCH == 'aarch64' && 'linux-aarch64' || ''}} | ||
with: | ||
repository: ${{ needs.setup-variables.outputs.REPOSITORY }} | ||
ref: ${{ needs.setup-variables.outputs.REF }} | ||
setup-miniconda: true | ||
python-version: ${{ env.PYTHON_VERSION }} | ||
cuda-version: ${{ env.CU_VERSION }} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we need this if we're not enabling CUDA builds? |
||
arch: ${{ needs.setup-variables.outputs.ARCH }} | ||
- name: Install torch dependency | ||
run: | | ||
set -euxo pipefail | ||
# shellcheck disable=SC1090 | ||
source "${BUILD_ENV_FILE}" | ||
# shellcheck disable=SC2086 | ||
${CONDA_RUN} ${PIP_INSTALL_TORCH} | ||
- name: Install python build dependency | ||
run: | | ||
set -euxo pipefail | ||
# shellcheck disable=SC1090 | ||
source "${BUILD_ENV_FILE}" | ||
# shellcheck disable=SC2086 | ||
${CONDA_RUN} python -m pip install build | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So this will be supplanted by https://github.com/pytorch/test-infra/pull/5071/files? Will we be able to remove some of these other steps as well to rely directly on the test-infra version? |
||
- name: Build the wheel (bdist_wheel) | ||
working-directory: ${{ needs.setup-variables.outputs.REPOSITORY }} | ||
shell: bash -l {0} | ||
run: | | ||
set -euxo pipefail | ||
source "${BUILD_ENV_FILE}" | ||
export PYTORCH_VERSION="$(${CONDA_RUN} pip show torch | grep ^Version: | sed 's/Version: *//' | sed 's/+.\+//')" | ||
${CONDA_RUN} python -m build --wheel | ||
- name: Smoke Test | ||
shell: bash -l {0} | ||
env: | ||
REPOSITORY: ${{ needs.setup-variables.outputs.REPOSITORY }} | ||
PACKAGE_NAME: ${{ needs.setup-variables.outputs.PACKAGE_NAME }} | ||
SMOKE_TEST_SCRIPT: ${{ needs.setup-variables.outputs.SMOKE_TEST_SCRIPT }} | ||
run: | | ||
set -euxo pipefail | ||
source "${BUILD_ENV_FILE}" | ||
WHEEL_NAME=$(ls "${{ env.REPOSITORY }}/dist/") | ||
echo "$WHEEL_NAME" | ||
|
||
${CONDA_RUN} pip install "$REPOSITORY/dist/$WHEEL_NAME" | ||
# Checking that we have a pinned version of torch in our dependency tree | ||
( | ||
pushd "${RUNNER_TEMP}" | ||
unzip -o "${GITHUB_WORKSPACE}/$REPOSITORY/dist/$WHEEL_NAME" | ||
# Ensure that pytorch version is pinned, should output file where it was found | ||
grep "Requires-Dist: torch (==.*)" -r . | ||
) | ||
|
||
if [[ (! -f "$REPOSITORY/${SMOKE_TEST_SCRIPT}) ]]; then | ||
echo "$REPOSITORY/${SMOKE_TEST_SCRIPT} not found" | ||
${CONDA_RUN} python -c "import ${PACKAGE_NAME}; print('package version is ', ${PACKAGE_NAME}.__version__)" | ||
else | ||
echo "$REPOSITORY/${SMOKE_TEST_SCRIPT} found" | ||
${CONDA_RUN} python "$REPOSITORY/${SMOKE_TEST_SCRIPT}" | ||
fi | ||
- name: Upload wheel to GitHub | ||
continue-on-error: true | ||
uses: actions/upload-artifact@v3 | ||
with: | ||
name: ${{ env.ARTIFACT_NAME }} | ||
path: $REPOSITORY/dist/ | ||
upload: | ||
needs: | ||
- setup-variables | ||
- build | ||
- generate-matrix | ||
uses: pytorch/test-infra/.github/workflows/_binary_upload.yml@main | ||
if: always() | ||
with: | ||
repository: $REPOSITORY | ||
ref: $REF | ||
test-infra-repository: $TEST_INFRA_REPOSITORY | ||
test-infra-ref: $TEST_INFRA_REF | ||
build-matrix: ${{ needs.generate-matrix.outputs.matrix }} | ||
architecture: $ARCH | ||
trigger-event: ${{ github.event_name }} | ||
|
||
concurrency: | ||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}-$REPOSITORY-${{ github.event_name == 'workflow_dispatch' }} | ||
cancel-in-progress: true |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,6 +6,7 @@ | |
|
||
from setuptools import find_packages, setup | ||
|
||
# test | ||
|
||
def read_requirements(file): | ||
with open(file) as f: | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand this, aren't we also building nightlies here? (Sorry maybe this is a dumb question)