`sdmTMB` not working within a `foreach () %dopar%` loop

[dmkaplan2000]

Hi,

I am having a strange problem that I am scratching my head about. I need to run a bunch of sdmTMB models, so I decided to run them in parallel using foreach() %dopar%. I do something along the lines of:

library(sdmTMB)
library(foreach)
library(doParallel)
cl <- makeCluster(4)  
registerDoParallel(cl)  

ss = foreach(formula=formulas,.packages=c("sdmTMB")) %dopar% {
  
  m = sdmTMB(
      data = mydata,
      formula = formula,
      mesh = mymesh, 
      time = 'year', 
      family = lognormal(),
      spatial = "off",
      spatiotemporal = "iid",
      offset = log(mydata$num_pos_sets)
    )
  
  saveRDS(m,"myfile.RDS")
}

I have tested that the model runs fine outside of the %dopar%, but when I run the full script, I see that the 4 R processes start up and are using heavy amounts of CPU and memory (often greatly exceeding 100% of CPU), but the models seem to never finish (a 15 minute model run individually never finishes over 12 hours). I have confirmed this both on my desktop computer and on a cluster I have access to. I have also confirmed that if I remove registerDoParallel so that essentially %dopar% becomes %do%, then the code does slowly advance with a single R process using >600% CPU (note that I have not executed TMB::openmp so if I understand correctly TMB itself should not be running in parallel).

Do you understand why this might be? Is it possible that the internal parallelization of sdmTMB is not playing nicely with foreach? What is the best way to parallelize the running of these models?

Thanks,
David

[seananderson]

I don't see a reason why foreach shouldn't work here, but maybe it's using parallel processing within openBLAS on top of the foreach loop? It's possible it's not reading your .Rprofile settings within the processes. That would be consistent with high CPUE usage and slow fits. https://gist.github.com/seananderson/08a51e296a854f227a908ddd365fb9c1

RhpcBLASctl::blas_set_num_threads(1)
RhpcBLASctl::omp_set_num_threads(1)

Example

library(sdmTMB)
library(foreach)
library(doParallel)
#> Loading required package: iterators
#> Loading required package: parallel
cl <- makeCluster(2)
registerDoParallel(cl)

formulas <- c(present ~ 1, present ~ depth)
out <- foreach(formula = formulas, .packages = c("sdmTMB")) %dopar% {
  sdmTMB(
    data = pcod,
    formula = formula,
    family = binomial(),
    spatial = "off"
  )
}

^{Created on 2025-05-31 with reprex v2.1.1}

I've tended to use the future package along with furrrr or sometimes future.apply. I just find it easier to work with including a better ability to automatically pick up globals and packages that need to be available in each process. Plus, with furrr, you can develop/debug with the matching purrr function and then switch to furrr. E.g. purrr::map() -> furrr::future_map().

library(sdmTMB)
library(future)
plan(multisession, workers = 2)
# for a 'forked' process where memory is shared:
# plan(multicore, workers = 2)
# but not available in RStudio

formulas <- c(present ~ 1, present ~ depth)
out <- furrr::future_map(formulas, \(f) {
  sdmTMB(
    data = pcod,
    formula = f,
    family = binomial(),
    spatial = "off"
  )
})

^{Created on 2025-05-31 with reprex v2.1.1}