Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add trigger for building Linux wheels #663

Merged
merged 25 commits into from
Apr 9, 2024
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 175 additions & 0 deletions .github/workflows/build_linux_wheels.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
# Heavy inspiration taken from:
# https://github.com/pytorch/test-infra/blob/main/.github/workflows/build_wheels_linux.yml

name: Build Linux Wheels


on:
pull_request:
push:
branches:
- nightly
- main
- release/*
tags:
# NOTE: Binary build pipelines should only get triggered on release candidate builds
# Release candidate tags look like: v1.11.0-rc1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this, aren't we also building nightlies here? (Sorry maybe this is a dumb question)

- v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+
workflow_dispatch:


permissions:
id-token: write
contents: read


jobs:
setup-variables:
runs-on: ubuntu-latest
outputs:
TEST_INFRA_REPOSITORY: pytorch/test-infra
TEST_INFRA_REF: main
REPOSITORY: pytorch/torchtune
REF: pkg-for-wheel
PACKAGE_NAME: torchtune
PACKAGE_TYPE: wheel
ARCH: x86_64
SMOKE_TEST_SCRIPT: ""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we had talked about potentially adding sanity checks of importing from the library or something like that here. Any plans to do that in this PR, or should we just do it later on?

steps:
- run: echo "Exposing env vars to all jobs"
generate-matrix:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any value to just adding a comment at the top of this job about which versions we should be building so folks don't have to go to the test-infra yaml to find this stuff?

needs:
- setup-variables
uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main
with:
package-type: ${{ needs.setup-variables.outputs.PACKAGE_TYPE }}
os: linux
test-infra-repository: ${{ needs.setup-variables.outputs.TEST_INFRA_REPOSITORY }}
test-infra-ref: ${{ needs.setup-variables.outputs.TEST_INFRA_REF }}
with-cuda: disable
with-rocm: disable
build:
needs:
- setup-variables
- generate-matrix
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.generate-matrix.outputs.matrix) }}
env:
PYTHON_VERSION: ${{ matrix.python_version }}
CU_VERSION: ${{ matrix.desired_cuda }}
UPLOAD_TO_BASE_BUCKET: ${{ matrix.upload_to_base_bucket }}
name: ${{ matrix.build_name }}
runs-on: ${{ matrix.validation_runner }}
container:
image: ${{ matrix.container_image }}
options: ${{ matrix.gpu_arch_type == 'cuda' && '--gpus all' || ' ' }}
# If a build is taking longer than 60 minutes on these runners we need
# to have a conversation
timeout-minutes: 120
steps:
- name: Clean workspace
shell: bash -l {0}
run: |
set -euxo pipefail
echo "::group::Cleanup debug output"
rm -rf "${GITHUB_WORKSPACE}"
mkdir -p "${GITHUB_WORKSPACE}"
echo "::endgroup::"
- uses: actions/checkout@v3
with:
repository: ${{ needs.setup-variables.outputs.TEST_INFRA_REPOSITORY }}
ref: ${{ needs.setup-variables.outputs.TEST_INFRA_REF }}
path: test-infra
- uses: pytorch/test-infra/.github/actions/set-channel@main
- name: Set PYTORCH_VERSION
if: ${{ env.CHANNEL == 'test' }}
run: |
# When building RC, set the version to be the current candidate version,
# otherwise, leave it alone so nightly will pick up the latest
echo "PYTORCH_VERSION=${{ matrix.stable_version }}" >> "${GITHUB_ENV}"
- uses: pytorch/test-infra/.github/actions/setup-binary-builds@main
env:
PLATFORM: ${{ env.ARCH == 'aarch64' && 'linux-aarch64' || ''}}
with:
repository: ${{ needs.setup-variables.outputs.REPOSITORY }}
ref: ${{ needs.setup-variables.outputs.REF }}
setup-miniconda: true
python-version: ${{ env.PYTHON_VERSION }}
cuda-version: ${{ env.CU_VERSION }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this if we're not enabling CUDA builds?

arch: ${{ needs.setup-variables.outputs.ARCH }}
- name: Install torch dependency
run: |
set -euxo pipefail
# shellcheck disable=SC1090
source "${BUILD_ENV_FILE}"
# shellcheck disable=SC2086
${CONDA_RUN} ${PIP_INSTALL_TORCH}
- name: Install python build dependency
run: |
set -euxo pipefail
# shellcheck disable=SC1090
source "${BUILD_ENV_FILE}"
# shellcheck disable=SC2086
${CONDA_RUN} python -m pip install build
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this will be supplanted by https://github.com/pytorch/test-infra/pull/5071/files? Will we be able to remove some of these other steps as well to rely directly on the test-infra version?

- name: Build the wheel (bdist_wheel)
working-directory: ${{ needs.setup-variables.outputs.REPOSITORY }}
shell: bash -l {0}
run: |
set -euxo pipefail
source "${BUILD_ENV_FILE}"
export PYTORCH_VERSION="$(${CONDA_RUN} pip show torch | grep ^Version: | sed 's/Version: *//' | sed 's/+.\+//')"
${CONDA_RUN} python -m build --wheel
- name: Smoke Test
shell: bash -l {0}
env:
REPOSITORY: ${{ needs.setup-variables.outputs.REPOSITORY }}
PACKAGE_NAME: ${{ needs.setup-variables.outputs.PACKAGE_NAME }}
SMOKE_TEST_SCRIPT: ${{ needs.setup-variables.outputs.SMOKE_TEST_SCRIPT }}
run: |
set -euxo pipefail
source "${BUILD_ENV_FILE}"
WHEEL_NAME=$(ls "${{ env.REPOSITORY }}/dist/")
echo "$WHEEL_NAME"

${CONDA_RUN} pip install "$REPOSITORY/dist/$WHEEL_NAME"
# Checking that we have a pinned version of torch in our dependency tree
(
pushd "${RUNNER_TEMP}"
unzip -o "${GITHUB_WORKSPACE}/$REPOSITORY/dist/$WHEEL_NAME"
# Ensure that pytorch version is pinned, should output file where it was found
grep "Requires-Dist: torch (==.*)" -r .
)

if [[ (! -f "$REPOSITORY/${SMOKE_TEST_SCRIPT}) ]]; then
echo "$REPOSITORY/${SMOKE_TEST_SCRIPT} not found"
${CONDA_RUN} python -c "import ${PACKAGE_NAME}; print('package version is ', ${PACKAGE_NAME}.__version__)"
else
echo "$REPOSITORY/${SMOKE_TEST_SCRIPT} found"
${CONDA_RUN} python "$REPOSITORY/${SMOKE_TEST_SCRIPT}"
fi
- name: Upload wheel to GitHub
continue-on-error: true
uses: actions/upload-artifact@v3
with:
name: ${{ env.ARTIFACT_NAME }}
path: $REPOSITORY/dist/
upload:
needs:
- setup-variables
- build
- generate-matrix
uses: pytorch/test-infra/.github/workflows/_binary_upload.yml@main
if: always()
with:
repository: $REPOSITORY
ref: $REF
test-infra-repository: $TEST_INFRA_REPOSITORY
test-infra-ref: $TEST_INFRA_REF
build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
architecture: $ARCH
trigger-event: ${{ github.event_name }}

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}-$REPOSITORY-${{ github.event_name == 'workflow_dispatch' }}
cancel-in-progress: true
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

from setuptools import find_packages, setup

# test

def read_requirements(file):
with open(file) as f:
Expand Down
Loading