Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Do not merge] Implement non-jax versions of IQP models #6

Open
wants to merge 100 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
89826b6
implemented branching logic for different use cases
mariaschuld Feb 28, 2024
ac10674
some minor corrections
mariaschuld Feb 28, 2024
8ff542d
simplify if logic a little
mariaschuld Feb 28, 2024
c57a184
Update src/qml_benchmarks/models/iqp_kernel.py
mariaschuld Feb 29, 2024
bf67629
Add makefile. Run make format. Add catalyst qjit as an option for lig…
vincentmr Feb 28, 2024
c39a4f8
Add qjit in iqp_variational.py module.
vincentmr Feb 29, 2024
98dcf67
Fix iqp_var solver.
vincentmr Feb 29, 2024
c7af8b6
faster qjitting
josephbowles Mar 26, 2024
d357dff
catalyst training
josephbowles Mar 27, 2024
27b4d9d
catalyst port attempt
josephbowles Mar 28, 2024
92b921d
add catalyst support
josephbowles Mar 28, 2024
868a62c
attempt catalyst port
josephbowles Mar 28, 2024
af61f65
add code
josephbowles Apr 9, 2024
43ddf74
test
Apr 10, 2024
56fc153
update
josephbowles Apr 10, 2024
46e6eb5
update
josephbowles Apr 10, 2024
81a614d
update
josephbowles Apr 10, 2024
66169c7
update
josephbowles Apr 10, 2024
5c01770
v1
josephbowles Apr 12, 2024
1596669
perf ind additions
josephbowles Apr 12, 2024
30ffcbb
v1
josephbowles Apr 12, 2024
d8bd09e
v1
josephbowles Apr 12, 2024
7a65c86
new code
josephbowles Apr 15, 2024
eec5fa5
add installs
josephbowles Apr 15, 2024
c18820f
add installs
josephbowles Apr 15, 2024
0f32a6c
add qmetric
josephbowles Apr 15, 2024
ae27b0e
update
josephbowles Apr 15, 2024
2a254be
result
josephbowles Apr 15, 2024
5cf8f2b
add info
josephbowles Apr 15, 2024
881dc94
add info
josephbowles Apr 15, 2024
7dfd26e
update tests
josephbowles Apr 15, 2024
f452d2b
add slurm code
josephbowles Apr 16, 2024
d081626
add slurm code
josephbowles Apr 16, 2024
f513c67
add slurm code
josephbowles Apr 16, 2024
74967ef
add slurm code
josephbowles Apr 16, 2024
9462980
add slurm code
josephbowles Apr 16, 2024
07b49a9
update code
josephbowles Apr 16, 2024
ce6b0ff
delete
josephbowles Apr 16, 2024
fe0a9ce
update
josephbowles Apr 16, 2024
b2bc779
update
josephbowles Apr 22, 2024
595d3ef
update
josephbowles Apr 22, 2024
62ac41e
update
josephbowles Apr 22, 2024
a94578e
update
josephbowles Apr 22, 2024
4a78cf0
update
josephbowles Apr 22, 2024
2698fc5
update
josephbowles Apr 22, 2024
2690831
update
josephbowles Apr 22, 2024
8513a93
.
josephbowles Apr 22, 2024
e813a7f
add use jax
josephbowles Apr 22, 2024
282d90b
.
josephbowles Apr 24, 2024
2cba022
update
Apr 24, 2024
45c4d72
update
Apr 24, 2024
2cb5917
prototype of Slurm job w/ Podman works
balewski Apr 25, 2024
feaf667
double dream works
balewski Apr 26, 2024
8aad113
ok
balewski Apr 26, 2024
63ce6dd
working sbatch script
josephbowles Apr 29, 2024
375b141
update jax version
josephbowles Apr 30, 2024
7b40ba1
working sbatch
josephbowles Apr 30, 2024
5c3ccb2
add perf attributes
josephbowles Apr 30, 2024
c380d43
reads data but crash in Python scalars
balewski Apr 30, 2024
1de6e38
cleanup
balewski May 1, 2024
a76e44a
data gen file
josephbowles May 2, 2024
15f4998
results
josephbowles May 2, 2024
34ca1cf
update
josephbowles May 2, 2024
7f52b53
result
josephbowles May 6, 2024
07b91a9
working sbatch
josephbowles May 6, 2024
241c7ed
rename
josephbowles May 6, 2024
f6c4884
rename
josephbowles May 6, 2024
03cfc6b
cleanup
josephbowles May 6, 2024
e678c96
cleanup
josephbowles May 6, 2024
9cf5550
cleanup
josephbowles May 6, 2024
49b0f80
cleanup
josephbowles May 6, 2024
b2f9bf0
update
josephbowles May 6, 2024
a6b85f4
update
josephbowles May 6, 2024
5fe8150
cleanup
josephbowles May 6, 2024
ada44af
rename
josephbowles May 6, 2024
d0a951e
cleanup
josephbowles May 6, 2024
adf4e33
update
josephbowles May 7, 2024
92fc797
working with catalyst
josephbowles May 29, 2024
3108ca1
model cleanup
josephbowles May 30, 2024
04670ba
catalyst support
josephbowles Jun 4, 2024
fc72c00
qjit update
josephbowles Jun 4, 2024
5016ec4
fix results dir
josephbowles Jun 5, 2024
4b333b1
profile time
josephbowles Jun 17, 2024
1899a0c
.
josephbowles Jun 17, 2024
3a1baa8
.
josephbowles Jun 17, 2024
5e26427
profile
josephbowles Jun 17, 2024
628292a
results
josephbowles Jun 18, 2024
c69895b
results
josephbowles Jun 19, 2024
e5dbfb6
results
josephbowles Jun 19, 2024
0328ca8
results
josephbowles Jun 19, 2024
6bc2c73
no vmap train
josephbowles Jun 19, 2024
362a7fe
results
josephbowles Jun 19, 2024
bea29fa
lax batching
josephbowles Jun 19, 2024
10553dd
update
josephbowles Jul 2, 2024
fd2efba
kernel profiling
josephbowles Jul 2, 2024
3f01d33
results
josephbowles Jul 3, 2024
8231bd0
results
josephbowles Jul 3, 2024
670ecec
results
josephbowles Jul 3, 2024
9c535e5
results
josephbowles Jul 3, 2024
b848381
update
josephbowles Aug 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -179,5 +179,10 @@ cython_debug/
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
.idea/

old/
.bash_history
.local/
.viminfo
hyperplanes_diff/
linearly_separable/

29 changes: 29 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
PYTHON3 := $(shell which python3 2>/dev/null)

PYTHON := python3

.PHONY: help
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " clean to delete all temporary, cache, and build files"
@echo " format [check=1] to apply black formatter; use with 'check=1' to check instead of modify (requires black)"
@echo " lint to run pylint on source files"

.PHONY : clean
clean:
rm -rf src/qml_benchmarks.egg-info/
rm -rf src/qml_benchmarks/__pycache__/
rm -rf src/qml_benchmarks/models/__pycache__/

.PHONY:format
format:
ifdef check
black -l 100 ./src/qml_benchmarks --check
else
black -l 100 ./src/qml_benchmarks
endif

.PHONY: lint
lint:
pylint src/qml_benchmarks --rcfile .pylintrc

51 changes: 51 additions & 0 deletions nersc/Dockerfile.ubu22-PennyLane
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
FROM ubuntu:22.04

# time podman-hpc build -f Dockerfile.ubu22-PennyLane -t balewski/ubu22-pennylane:p5 .
#>>> real 6m47.959s

# podman commit -a "Jan Balewski" c3d48cf15876XXX balewski/XXXkit-qml:p2ch
# on laptop:
# podman run -it balewski/ubu22-pennylane:p0 bash
# time python3 -c 'import pennylane'

# is needed by tzdada which sets sth for one of libs in section 2
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=America/Los_Angeles

RUN echo 1a-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA OS update && \
apt-get update && \
apt-get install -y locales autoconf automake gcc g++ make vim wget ssh openssh-server sudo git emacs aptitude build-essential xterm python3-pip python3-tk python3-scipy python3-dev iputils-ping net-tools screen feh hdf5-tools python3-bitstring mlocate graphviz tzdata x11-apps && \
apt-get clean all

# Forbid installing qiskit-terra for qiskit 1.0
RUN echo 2b-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA Qiskit 1.0 libs && \
pip3 install -c https://qisk.it/1-0-constraints qiskit[visualization] qiskit-ibm-runtime qiskit-machine-learning qiskit_ibm_experiment qiskit-aer


RUN echo 2c-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA ML+QML libs && \
pip3 install scikit-learn pandas seaborn[stats] networkx[default] optuna


RUN echo 2d-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA python libs && \
pip3 install --upgrade pip && \
pip3 install matplotlib h5py ruamel.yaml scipy jupyter notebook bitstring

# notes: python3-tk instals TK for matplotlib to display graphic
RUN echo 2e-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA python libs && \
pip3 install --upgrade Pillow


# based on https://pennylane.ai/install/
# based on https://pytorch.org/get-started/locally/

RUN echo 3a-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA pennylane && \
pip3 install pennylane --upgrade && \
pip3 install pennylane-lightning pennylane-lightning[gpu] pennylane-sf pennylane-qiskit pennylane-cirq pennylane-catalyst scipy==1.11.4 && \
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

RUN echo 3b-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA JAX && \
pip3 install "jax[cpu]"==0.4.23 && \
pip3 install jaxopt optax



51 changes: 51 additions & 0 deletions nersc/container/Dockerfile.ubu22-PennyLane
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
FROM ubuntu:22.04

# time podman-hpc build -f Dockerfile.ubu22-PennyLane -t balewski/ubu22-pennylane:p5 .
#>>> real 6m47.959s

# podman commit -a "Jan Balewski" c3d48cf15876XXX balewski/XXXkit-qml:p2ch
# on laptop:
# podman run -it balewski/ubu22-pennylane:p0 bash
# time python3 -c 'import pennylane'

# is needed by tzdada which sets sth for one of libs in section 2
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=America/Los_Angeles

RUN echo 1a-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA OS update && \
apt-get update && \
apt-get install -y locales autoconf automake gcc g++ make vim wget ssh openssh-server sudo git emacs aptitude build-essential xterm python3-pip python3-tk python3-scipy python3-dev iputils-ping net-tools screen feh hdf5-tools python3-bitstring mlocate graphviz tzdata x11-apps && \
apt-get clean all

# Forbid installing qiskit-terra for qiskit 1.0
RUN echo 2b-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA Qiskit 1.0 libs && \
pip3 install -c https://qisk.it/1-0-constraints qiskit[visualization] qiskit-ibm-runtime qiskit-machine-learning qiskit_ibm_experiment qiskit-aer


RUN echo 2c-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA ML+QML libs && \
pip3 install scikit-learn pandas seaborn[stats] networkx[default] optuna


RUN echo 2d-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA python libs && \
pip3 install --upgrade pip && \
pip3 install matplotlib h5py ruamel.yaml scipy jupyter notebook bitstring

# notes: python3-tk instals TK for matplotlib to display graphic
RUN echo 2e-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA python libs && \
pip3 install --upgrade Pillow


# based on https://pennylane.ai/install/
# based on https://pytorch.org/get-started/locally/

RUN echo 3a-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA pennylane && \
pip3 install pennylane --upgrade && \
pip3 install pennylane-lightning pennylane-lightning[gpu] pennylane-sf pennylane-qiskit pennylane-cirq pennylane-catalyst scipy==1.11.4 && \
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

RUN echo 3b-AAAAAAAAAAAAAAAAAAAAAAAAAAAAA JAX && \
pip3 install -U "jax[cpu]" && \
pip3 install jaxopt optax



135 changes: 135 additions & 0 deletions nersc/container/Readme
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
Usage of Podman container for PennyLane

details are here
https://docs.google.com/document/d/1LHYlTXtOyA5vZSrJF0IdZf3L88DcguMWXyHAPLV-MvU/edit?usp=sharing


Summary:
This is an instruction on how to build and use a Podman image.
The example consists of three parts:
A) Building the image, which I've already completed on PM.
B) Running the image on PM in three modes: simple interactive, advanced interactive, and a Slurm job that executes your code inside a podman image.
C) Running the same image on your laptop. You'll need to build it as described in A), but then running it is straightforward.



= = =
A = = = = = = = = = = building container (one time) = = = = = = = = =
= = =
Dockerfile: Dockerfile.ubu22-PennyLane
Laptop:
podman build -f Dockerfile.ubu22-PennyLane -t balewski/ubu22-pennylane:p5 .
Perlmutter
podman-hpc build -f Dockerfile.ubu22-PennyLane -t balewski/ubu22-pennylane:p5 .
POD_PUB=/cfs/cdirs/nstaff/balewski/podman_common/
podman-hpc --squash-dir /global/$POD_PUB migrate balewski/ubu22-pennylane:p5
chmod -R a+rx /global/$POD_PUB # to allow anyone to use this image

Note, the instruction above is for user:balewski who has access to CFS/nstaff - change the strings as needed



= = =
B = = = = = = = = = = Perlmutter @ NERSC = = = = = = = = =
= = =

B.1 - - - - - - simple interactive use on a worker node - - - - - - -

ssh perlmutter
export PS1="\h:\W> " # to reduce too long names of directoriesexport PS1="\h:\W> " # to reduce too long names of directories

POD_PUB=/cfs/cdirs/nstaff/balewski/podman_common/
export PODMANHPC_ADDITIONAL_STORES=/dvs_ro/$POD_PUB
cd $SCRATCH/tmp
salloc -q interactive -C cpu -t 4:00:00 -A nstaff
podman-hpc run -it balewski/ubu22-pennylane:p5 bash
python3 -c 'import pennylane'
exit


B.2 - - - - - - advanced interactive use on a worker node - - - - - - -
copy & customize https://github.com/balewski/quantumMind/blob/main/PennyLane/pm_podman.source
to configure shortcuts to your liking

ssh perlmutter
salloc -q interactive -C cpu -t 4:00:00 -A nstaff
source pm_podman.source

[ do your work ]

exit #from podman
exit # from salloc



B.3 - - - - - - advanced : Slurm job with multiple task using Podman image- - - - - - -
copy and customize https://github.com/balewski/quantumMind/tree/main/PennyLane/qml_intro/
batchPodman.slr
wrap_podman.sh

You will edit code stored in CFS area but running job will read/write from/to SCRATCH using its copy. Do NOT start Podman image manually

ssh perlmutter
ssh CFS (to your area )
sbatch batchPodman.slr


B.4 - - - - - - advanced : install project spceciffic software, NOT recommended but possible

ssh pm

IMG=jbowles/ubu22-pennylane:p5
POD_PUB=/dvs_ro/cfs/cdirs/m4139/qml-benchmarks/nersc/podman/
export PODMANHPC_ADDITIONAL_STORES=$POD_PUB
CFSH=$SCRATCH/pennylane_wrk # for Jan
#CFSH=/global/cfs/cdirs/m4139 # for Joseph
podman-hpc run -it --volume $CFSH/qml-benchmarks:/root --workdir /root $IMG bash
pip3 install --user . # inside image, create private ./local

exit

Testing:
podman-hpc run -it …
python3 -u -c "import qml_benchmarks "
exit

AND when lanuching podman image you must have this volume mount, on the top of all other volumes you need:

--volume $CFSH/qml-benchmarks:/root

B.5 - - - - - - advanced : jupiter notebook on the laptop powered by Podman PennyLane image running on PM - - - - - - -
Easy to do




= = =
C = = = = = = = = = = Laptop = = = = = = = = =
= = =

C.1 - - - - - - simple interactive use on a laptop - - - - - - -
$ podman run -it balewski/ubu22-pennylane:p3 bash
python3 -c 'import pennylane'
exit


C.2 - - - - - - advanced w/ predefined volume mounts on a laptop - - - - - - -
copy & customize https://github.com/balewski/quantumMind/blob/main/PennyLane/laptop_podman.source
to configure shortcuts to your liking

source laptop_podman.source

[ do your work ]

exit #from podman

If you need to reset podman, do : source restart_podman.source


C.4 - - - - - - advanced : jupiter notebook on the laptop powered by Podman PennyLane image running on the laptop - - - - - - -

source laptop_podman.source jnb
(exec inside running image) jupyter notebook --ip 0.0.0.0 --no-browser --allow-root --port 8833
copy http string and paste into local browser , e.g: http://127.0.0.1:8833/tree?token=7c5cdf5e5c4f1a9d2a616a739988132d59f1a7ca3f4c0779

Troubleshooting: if JNB remains blank change port ID in laptop_podman.source and restart the image
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
construct_kernel_time,training_time,predict_time,hyperparameters
3860.4390046596527,3860.4435765743256,3755.371912240982,"{'repeats': 10, 'use_jax': True, 'vmap': True, 'jit': True}"
Loading