[email protected] #84

jaelynlitz · 2023-11-21T22:32:42Z

Merge request type

New feature
Resolves bug
Documentation
Other

Relates to

This MR updates

Summary

Resolve issues interfacing with [email protected] onwards. Some function signature changes resulted in changes in the interface, other bugs TBD

Linked Issue(s)

cameronrutherford · 2023-11-21T23:51:32Z

Turns out you have to mention the "closing keywords" for each issue individually. Don't have the time at the moment, but would be good to explore prettier issue templates like spack has

buildsystem/spack/deception/spack.yaml

jaelynlitz · 2023-11-30T02:35:29Z

trying [email protected] on incline and am getting this error? have you seen this before @cameronrutherford?

https://gitlab.pnnl.gov/exasgd/frameworks/exago-github-mirror/-/jobs/137810#L128

  1 ==> petsc: Executing phase: 'configure'
  2 ==> [2023-11-29-17:25:41.538594] Find (not recursive): /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ck    x5653ahtbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  3 ==> [2023-11-29-17:25:41.542749] Find complete: /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ah    tbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  4 ==> [2023-11-29-17:25:41.543214] Find (not recursive): /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ck    x5653ahtbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  5 ==> [2023-11-29-17:25:41.543359] Find complete: /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ah    tbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  6 ==> [2023-11-29-17:25:41.545360] '/share/apps/python/3.11.4/bin/python3.11' 'configure' '--prefix=/qfs/projects/exasgd/src/ci-incline/install/linux-    centos7-zen/clang-15.0.0-rocm5.3.0/petsc-3.20.1-qwd245ym3rxespvefk3rvqivxpv4xtqx' '--with-ssl=0' '--download-c2html=0' '--download-sowing=0' '--down    load-hwloc=0' '--with-make-exec=make' 'CXXFLAGS=--gcc-toolchain=/share/apps/gcc/8.4.0/' 'CXXOPTFLAGS=' '--with-cc=/share/apps/openmpi/4.1.4/gcc/8.4.    0/bin/mpicc' '--with-cxx=/share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpic++' '--with-fc=/share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpif90' '--with-precisio    n=double' '--with-scalar-type=real' '--with-shared-libraries=1' '--with-debugging=0' '--with-openmp=0' '--with-64-bit-indices=0' '--with-blas-lapack    -lib=/qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ahtbucwlxhwv4lkwnavfui/lib/libopenblas.so' '-    -with-x=0' '--with-sycl=0' '--with-clanguage=C' '--with-cuda=0' '--with-hip=0' '--with-metis=0' '--with-hypre=0' '--with-parmetis=0' '--with-kokkos=    0' '--with-kokkos-kernels=0' '--with-superlu_dist=0' '--with-ptscotch=0' '--with-suitesparse=0' '--with-hdf5=0' '--with-zlib=0' '--with-mumps=0' '--    with-trilinos=0' '--with-fftw=0' '--with-valgrind=0' '--with-gmp=0' '--with-libpng=0' '--with-giflib=0' '--with-mpfr=0' '--with-netcdf=0' '--with-pn    etcdf=0' '--with-moab=0' '--with-random123=0' '--with-exodusii=0' '--with-cgns=0' '--with-memkind=0' '--with-p4est=0' '--with-saws=0' '--with-yaml=0    ' '--with-hwloc=0' '--with-libjpeg=0' '--with-scalapack=0' '--with-strumpack=0' '--with-mmg=0' '--with-parmmg=0' '--with-tetgen=0'
  7 =============================================================================================
  8                          Configuring PETSc to compile on your system
  9 =============================================================================================
 10 =============================================================================================
 11                                      ***** WARNING *****
 12   Found environment variable:
 13   CC=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/clang.
 14   Ignoring it, since its also set on command line
 15 =============================================================================================
 16 =============================================================================================
 17                                      ***** WARNING *****
 18   Found environment variable:
 19   CXX=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/clang++.
 20   Ignoring it, since its also set on command line
 21 =============================================================================================
 22 =============================================================================================
 23                                      ***** WARNING *****
 24   Found environment variable:
 25   FC=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/gfortran.
 26   Ignoring it, since its also set on command line
 27 =============================================================================================
 28 =============================================================================================
 29                                      ***** WARNING *****
 30   Found environment variable:
 31   F77=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/gfortran.
 32   Ignoring it! Use "./configure F77=$F77" if you really want to use this value
 33 =============================================================================================
 34 =============================================================================================
 35                                      ***** WARNING *****
 36   Found environment variable: F90=/share/apps/gcc/8.4.0/bin/gfortran. Ignoring it! Use
 37   "./configure F90=$F90" if you really want to use this value
 38 =============================================================================================
 39 =============================================================================================
 40                                      ***** WARNING *****
 41   Using default C optimization flags "-g -O3". You might consider manually setting optimal
 42   optimization flags for your system with COPTFLAGS="optimization flags" see
 43   config/examples/arch-*-opt.py for examples
 44 =============================================================================================
 45 =============================================================================================
 46                                      ***** WARNING *****
 47   Using default FC optimization flags "-g -O". You might consider manually setting optimal
 48   optimization flags for your system with FOPTFLAGS="optimization flags" see
 49   config/examples/arch-*-opt.py for examples
 50 =============================================================================================
 51
 52 *********************************************************************************************
 53            UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
 54 ---------------------------------------------------------------------------------------------
 55   Timeout: Unable to run MPI program with /share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpiexec
 56   --oversubscribe -n 1
 57   (1) make sure this is the correct program to run MPI jobs
 58   (2) your network may be misconfigured; see
 59   https://petsc.org/release/faq/#mpi-network-misconfigure
 60   (3) you may have VPN running whose network settings may not play nice with MPI
 61 *********************************************************************************************

cameronrutherford · 2023-11-30T02:52:22Z

trying [email protected] on incline and am getting this error? have you seen this before @cameronrutherford?

  1 ==> petsc: Executing phase: 'configure'
  2 ==> [2023-11-29-17:25:41.538594] Find (not recursive): /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ck    x5653ahtbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  3 ==> [2023-11-29-17:25:41.542749] Find complete: /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ah    tbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  4 ==> [2023-11-29-17:25:41.543214] Find (not recursive): /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ck    x5653ahtbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  5 ==> [2023-11-29-17:25:41.543359] Find complete: /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ah    tbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  6 ==> [2023-11-29-17:25:41.545360] '/share/apps/python/3.11.4/bin/python3.11' 'configure' '--prefix=/qfs/projects/exasgd/src/ci-incline/install/linux-    centos7-zen/clang-15.0.0-rocm5.3.0/petsc-3.20.1-qwd245ym3rxespvefk3rvqivxpv4xtqx' '--with-ssl=0' '--download-c2html=0' '--download-sowing=0' '--down    load-hwloc=0' '--with-make-exec=make' 'CXXFLAGS=--gcc-toolchain=/share/apps/gcc/8.4.0/' 'CXXOPTFLAGS=' '--with-cc=/share/apps/openmpi/4.1.4/gcc/8.4.    0/bin/mpicc' '--with-cxx=/share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpic++' '--with-fc=/share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpif90' '--with-precisio    n=double' '--with-scalar-type=real' '--with-shared-libraries=1' '--with-debugging=0' '--with-openmp=0' '--with-64-bit-indices=0' '--with-blas-lapack    -lib=/qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ahtbucwlxhwv4lkwnavfui/lib/libopenblas.so' '-    -with-x=0' '--with-sycl=0' '--with-clanguage=C' '--with-cuda=0' '--with-hip=0' '--with-metis=0' '--with-hypre=0' '--with-parmetis=0' '--with-kokkos=    0' '--with-kokkos-kernels=0' '--with-superlu_dist=0' '--with-ptscotch=0' '--with-suitesparse=0' '--with-hdf5=0' '--with-zlib=0' '--with-mumps=0' '--    with-trilinos=0' '--with-fftw=0' '--with-valgrind=0' '--with-gmp=0' '--with-libpng=0' '--with-giflib=0' '--with-mpfr=0' '--with-netcdf=0' '--with-pn    etcdf=0' '--with-moab=0' '--with-random123=0' '--with-exodusii=0' '--with-cgns=0' '--with-memkind=0' '--with-p4est=0' '--with-saws=0' '--with-yaml=0    ' '--with-hwloc=0' '--with-libjpeg=0' '--with-scalapack=0' '--with-strumpack=0' '--with-mmg=0' '--with-parmmg=0' '--with-tetgen=0'
  7 =============================================================================================
  8                          Configuring PETSc to compile on your system
  9 =============================================================================================
 10 =============================================================================================
 11                                      ***** WARNING *****
 12   Found environment variable:
 13   CC=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/clang.
 14   Ignoring it, since its also set on command line
 15 =============================================================================================
 16 =============================================================================================
 17                                      ***** WARNING *****
 18   Found environment variable:
 19   CXX=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/clang++.
 20   Ignoring it, since its also set on command line
 21 =============================================================================================
 22 =============================================================================================
 23                                      ***** WARNING *****
 24   Found environment variable:
 25   FC=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/gfortran.
 26   Ignoring it, since its also set on command line
 27 =============================================================================================
 28 =============================================================================================
 29                                      ***** WARNING *****
 30   Found environment variable:
 31   F77=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/gfortran.
 32   Ignoring it! Use "./configure F77=$F77" if you really want to use this value
 33 =============================================================================================
 34 =============================================================================================
 35                                      ***** WARNING *****
 36   Found environment variable: F90=/share/apps/gcc/8.4.0/bin/gfortran. Ignoring it! Use
 37   "./configure F90=$F90" if you really want to use this value
 38 =============================================================================================
 39 =============================================================================================
 40                                      ***** WARNING *****
 41   Using default C optimization flags "-g -O3". You might consider manually setting optimal
 42   optimization flags for your system with COPTFLAGS="optimization flags" see
 43   config/examples/arch-*-opt.py for examples
 44 =============================================================================================
 45 =============================================================================================
 46                                      ***** WARNING *****
 47   Using default FC optimization flags "-g -O". You might consider manually setting optimal
 48   optimization flags for your system with FOPTFLAGS="optimization flags" see
 49   config/examples/arch-*-opt.py for examples
 50 =============================================================================================
 51
 52 *********************************************************************************************
 53            UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
 54 ---------------------------------------------------------------------------------------------
 55   Timeout: Unable to run MPI program with /share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpiexec
 56   --oversubscribe -n 1
 57   (1) make sure this is the correct program to run MPI jobs
 58   (2) your network may be misconfigured; see
 59   https://petsc.org/release/faq/#mpi-network-misconfigure
 60   (3) you may have VPN running whose network settings may not play nice with MPI
 61 *********************************************************************************************

Maybe this isn't the right thing to do, but one idea is to rebuild ROCm 5.6.0 and a new clang version like we have already done for ReSolve https://github.com/ORNL/ReSolve/pull/53/files.

#89 and @nkoukpaizan was mentioning test failures anyway, so maybe that's a bad idea.

I don't know what could be going wrong here, but there should be some better logs being printed in configure.log that you need to look at that will say more of what is going wrong.

I would suggest maybe you look at the incline PR and review tricky SLURM configuration issues that might be tripping you up.

cameronrutherford · 2023-11-30T02:54:02Z

If you don't have any ideas, we can also ping Satish and xSDK folks for advice. You can quickly try PETSc 3.19 as well since that should also work in theory

jaelynlitz · 2023-11-30T21:12:00Z

If you don't have any ideas, we can also ping Satish and xSDK folks for advice. You can quickly try PETSc 3.19 as well since that should also work in theory

I am very confused bc the first pipeline petsc 3.19.6 failed, but on the retry it built? https://gitlab.pnnl.gov/exasgd/frameworks/exago-github-mirror/-/jobs/138077 I'm stumped but won't question it...

cameronrutherford · 2023-11-30T21:18:28Z

If you don't have any ideas, we can also ping Satish and xSDK folks for advice. You can quickly try PETSc 3.19 as well since that should also work in theory

I am very confused bc the first pipeline petsc 3.19.6 failed, but on the retry it built? https://gitlab.pnnl.gov/exasgd/frameworks/exago-github-mirror/-/jobs/138077 I'm stumped but won't question it...

Glad 3.19 builds, but 3.20 should work.

None of the pipelines or error messages you have linked are at the right level for us to diagnose the problem. Please get the right configure.log or error message that describes an actual issue occurring. Currently we are blindly speculating and praying, and 3.20 should still work just fine.

tests/functionality/opflow/CMakeLists.txt

jaelynlitz · 2023-12-01T19:35:02Z

Waiting for the ascent rebuild to run - then this is good to go! I just opened a spack PR, so when that is merged, we can move the spack submodule back to spack proper instead of my fork.

…build] [incline-rebuild] [newell-rebuild]

cameronrutherford added this to the 1.6.1 Python bindings + Documentation Release milestone Nov 21, 2023

cameronrutherford reviewed Nov 21, 2023

View reviewed changes

buildsystem/spack/deception/spack.yaml Show resolved Hide resolved

jaelynlitz mentioned this pull request Nov 30, 2023

OPFLOW HiOp functionality tests failing with [email protected] #43

Closed

jaelynlitz force-pushed the hiop-1.0.1 branch from 0d56a1a to 62c2c6a Compare November 30, 2023 22:13

jaelynlitz mentioned this pull request Nov 30, 2023

Incline Test Failures #92

Open

13 tasks

cameronrutherford previously approved these changes Dec 1, 2023

View reviewed changes

tests/functionality/opflow/CMakeLists.txt Show resolved Hide resolved

jaelynlitz dismissed cameronrutherford’s stale review via 0cf8f4b December 1, 2023 19:24

jaelynlitz requested review from pelesh and ryandanehy December 1, 2023 19:33

jaelynlitz and others added 14 commits December 1, 2023 16:30

enable more scopflow tests and rebuild with latest hiop [deception-re…

b2aa6e0

…build] [incline-rebuild] [newell-rebuild]

update hiop interface due to the latest changes in hiop

c2e233d

use spack fork for dev [deception-rebuild]

4dda529

Update deception spack built tcl modules - [deception-test]

117320c

dev_path hiop [newell-rebuild]

05dde80

Update newell spack built tcl modules - [newell-test]

fe58426

enable failing opflow tests on newell

5a9d731

change tolerance [newell-test]

a2841fc

Update deception spack built tcl modules - [deception-test]

9bbf68d

enable failing opflow tests on deception [deception-test]

b5b9ed0

Update incline spack built tcl modules - [incline-test]

e4c8e0a

enable all tests on incline [incline-test]

9281f22

enable all ascent tests [ascent-rebuild]

86913b9

build incline petsc3.20.1 with 20 cores [incline-rebuild]

cef655d

jaelynlitz and others added 4 commits December 1, 2023 16:30

disable two incline tests [incline-rebuild]

25d5dbf

Update incline spack built tcl modules - [incline-test]

28dc55a

add dev_path hiop for ascent [ascent-rebuild]

eef3efb

Update Ascent spack built tcl modules

c15cb47

jaelynlitz force-pushed the hiop-1.0.1 branch from fa2eae5 to c15cb47 Compare December 2, 2023 00:31

ryandanehy approved these changes Dec 2, 2023

View reviewed changes

cameronrutherford merged commit 2bdd5b7 into develop Dec 2, 2023
11 checks passed

This was referenced Dec 4, 2023

Update spack & try petsc3.20.2 #93

Merged

SCOPFLOW HiOp Patch #2

Closed

update hiop interface due to the latest changes in hiop #30

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[email protected] #84

[email protected] #84

jaelynlitz commented Nov 21, 2023 •

edited by cameronrutherford

Loading

cameronrutherford commented Nov 21, 2023

jaelynlitz commented Nov 30, 2023 •

edited

Loading

cameronrutherford commented Nov 30, 2023

cameronrutherford commented Nov 30, 2023 •

edited

Loading

jaelynlitz commented Nov 30, 2023

cameronrutherford commented Nov 30, 2023 •

edited

Loading

jaelynlitz commented Dec 1, 2023

[email protected] #84

[email protected] #84

Conversation

jaelynlitz commented Nov 21, 2023 • edited by cameronrutherford Loading

cameronrutherford commented Nov 21, 2023

jaelynlitz commented Nov 30, 2023 • edited Loading

cameronrutherford commented Nov 30, 2023

cameronrutherford commented Nov 30, 2023 • edited Loading

jaelynlitz commented Nov 30, 2023

cameronrutherford commented Nov 30, 2023 • edited Loading

jaelynlitz commented Dec 1, 2023

jaelynlitz commented Nov 21, 2023 •

edited by cameronrutherford

Loading

jaelynlitz commented Nov 30, 2023 •

edited

Loading

cameronrutherford commented Nov 30, 2023 •

edited

Loading

cameronrutherford commented Nov 30, 2023 •

edited

Loading