Multithreading: only detecting 1 core when 20 specified (has omp support)

[tkerr1159]

Hello, I am looking to process a lot of lidar files into CHMs, and I wish to do this with multiple cores. However, I am getting a warning "Number of cores requested 20 but only 1 available.

I have done some digging and see that while parallel::detectCores(logical = TRUE) returns 32, Sys.getenv("OMP_NUM_THREADS") returns 1. It seems openmp is not detecting the correct number of cores?

I have specified 20 cores using the ncores = 20 within the exec() function, in order to take advantage of the concurrent files strategy.

If anyone has any input or wants some additional information, please let me know!

[Jean-Romain]

Are you on MacOS ?

[tkerr1159]

No, I'm running it on linux.

[Jean-Romain]

Ok, are you working in some kind of container? I'm highly confident that the problem does not come from me. Your system is setup in such a way that it is constrain to 1 thread. Why? I don't know. It is not my computer. What does data.table::getDTthreads() return?

[tkerr1159]

data.table::getDTthreads() returns 1.

I am running R from a conda environment, which I had previously set up when I built out this CHM workflow using lidR. In fact, I used the doParallel library with a foreach loop to create the CHMs, and didn't have any issues with specifying multiple cores for parallelization.

[Jean-Romain]

So you have no native C/C++ parallelism with OpenMP in your conda environment. doParallel and other backend capable of parallelizing R code are not native parallelization. They open multiple R sessions and serialize the I/O to communicate the inputs and outputs within the sessions. This is the strategy used by lidR with its pros and cons but lasR is a pure C++ program not relying on R.

In your conda environment no package relying on OpenMP and C/C++ parallelism can leverage multi-threading and this is on your end to configure/fix it. I have absolutely zero knowledge about conda to help you. Why not using a regular installation of R? You are not the first one to report issues because of fancy/custom installation.

[tkerr1159]

I see, this is insightful. I will talk to our server manager about switching to a regular install of R, since he is the one who recommended and set up the R anaconda environment. Thank you for all your help!