vignettes/memory-and-parallelization.Rmd
memory-and-parallelization.RmdIn order to be able to work with the whole database at the sapwood or plant level it is recommended at least \(16GB\) of RAM memory. This is because loading all data objects already consumes \(4GB\) and any operation like aggregation or metric calculation results in extra memory needed:
library(sapfluxnetr) # This will need at least 5GB of memory during the process folder <- 'RData/plant' sfn_metadata <- read_sfn_metadata(folder) daily_results <- sfn_sites_in_folder(folder) %>% filter_sites_by_md( si_biome %in% c("Temperate forest", 'Woodland/Shrubland'), sites = sites, metadata = sfn_metadata ) %>% read_sfn_data(folder) %>% daily_metrics(tidy = TRUE, metadata = sfn_metadata) # Important to save, this way you will have access to the object in the future save(daily_results, file = 'daily_results.RData')
To circumvent this in less powerful systems, we recommend to work in small subsets of sites (25-30) and join the tidy results afterwards:
library(sapfluxnetr) folder <- 'RData/plant' metadata <- read_sfn_metadata(folder) sites <- sfn_sites_in_folder(folder) %>% filter_sites_by_md( si_biome %in% c("Temperate forest", 'Woodland/Shrubland'), sites = sites, metadata = sfn_metadata ) daily_results_1 <- read_sfn_data(sites[1:30], folder) %>% daily_metrics(tidy = TRUE, metadata = sfn_metadata) daily_results_2 <- read_sfn_data(sites[31:60], folder) %>% daily_metrics(tidy = TRUE, metadata = sfn_metadata) daily_results_3 <- read_sfn_data(sites[61:90], folder) %>% daily_metrics(tidy = TRUE, metadata = sfn_metadata) daily_results_4 <- read_sfn_data(sites[91:110], folder) %>% daily_metrics(tidy = TRUE, metadata = sfn_metadata) daily_results_steps <- bind_rows( daily_results_1, daily_results_2, daily_results_3, daily_results_4 ) rm(daily_results_1, daily_results_2, daily_results_3, daily_results_4) save(daily_results_steps, file = 'daily_results_steps.RData')
sapfluxnetr includes tha capability to parallelize the metrics calculation when performed on a sfn_data_multi object. This is made thanks to the furrr package, which uses the future package behind the scenes. By default, the code will run in a sequential process, which is the usual way the R code runs. But setting the future::plan to multicore (in Linux), multisession (in Windows) or multiprocess (automatically choose between the previous plans depending on the system) will run the code in parallel, dividing the sites between the available cores.
> Be advised, parallelization usually means more RAM used, so in systems
with less than 16GB maybe is not a good idea.
Also, the time benefits start to show when analysing 10 sites or more.# loading future package library(future) # setting the plan plan('multiprocess') # metrics!! daily_results_parallel <- sfn_sites_in_folder(folder) %>% filter_sites_by_md( si_biome %in% c("Temperate forest", 'Woodland/Shrubland'), sites = sites, metadata = sfn_metadata ) %>% read_sfn_data(folder) %>% daily_metrics(tidy = TRUE, metadata = sfn_metadata) # Important to save, this way you will have access to the object in the future save(daily_results_parallel, file = 'daily_results_parallel.RData')
When using furrr, even in the sequential plan, the future package sets a limit of \(500MB\) for each core. With sapfluxnet data this limit is easily exceeded, causing an error. To avoid this we may want to set the future.globals.maxSize limit to a higher value (\(1GB\) for example, but the limit wanted really depend on the plan and the number of sites):
# future library library(future) # plan sequential, not really needed, as it is the default, but for the sake of # clarity plant('sequential') # up the limit to 1GB, this in bytes is 1014*1024^2 options('future.globals.maxSize' = 1014*1024^2) # do the metrics daily_results_limit <- sfn_sites_in_folder(folder) %>% filter_sites_by_md( si_biome %in% c("Temperate forest", 'Woodland/Shrubland'), sites = sites, metadata = sfn_metadata ) %>% read_sfn_data(folder) %>% daily_metrics(tidy = TRUE, metadata = sfn_metadata) # Important to save, this way you will have access to the object in the future save(daily_results_limit, file = 'daily_results_limit.RData')