Evaluate model runs for calibration

Evaluate the model runs produced by a call to alternate() with user-defined error functions.

Usage

evaluate_models(
  x,
  eval_funs = NULL,
  eval_weights = 1,
  param_cols,
  eval_cols,
  n_best = NULL,
  f_best = 0.01,
  scaling_fun = scale_min_median,
  ...
)

# S3 method for class 'cfp_altres'
evaluate_models(
  x,
  eval_funs = NULL,
  eval_weights = 1,
  param_cols = cfp_id_cols(cfp_layers_map(cfp_og_model(x))),
  eval_cols = NULL,
  n_best = NULL,
  f_best = 0.01,
  scaling_fun = scale_min_median,
  ...
)

Arguments

x

A cfp_altres object, as returned by alternate().

eval_funs

A named list of evaluation functions. Each function must accept the arguments x and param_cols that are passed from this function.

eval_weights

A vector of weights the same length of eval_funs or one. Alternatively a data.frame() that specifies the weight for any wished error_parameter (names of eval_funs) and param_cols combinations. Provide the weights as a numeric in the parameter_weight column.

param_cols

The columns that, together, define different parameters (e.g. different gases) for which NRMSEs should be calculated separately (e.g. "gas"). Defaults to the id_cols of layers_map. If no such distinction is wished, set to character()

eval_cols

A character vector of columns for which the model error should be returned separately. Must be a subset of param_cols and defaults to the complete set.

n_best

An integer number of runs to select as the best runs.

f_best

A numeric between 0 to 1 as the fraction of runs to select as the best. Defaults to 0.01.

scaling_fun

A scaling function. Defaults to min-median scaling.

...

Any arguments that need to be passed to the error_funs. Note that all matching arguments will be applied to each function!

Value

A list with components best_runs the runs with the lowest model error (ME), model_error the model error for all runs, models_evaluated the raw values returned by error_funs and best_runs_runmap, a cfp_run_map() which can be used to rerun the best_runs model configurations. Note, that for best_runs_runmap the value of run_id is remapped to values 1:n_best.

Examples

# \donttest{
PROFLUX <- pro_flux(base_dat |> filter(site == "site_a"))

run_map <-
 cfp_run_map(
   PROFLUX,
   params = list(TPS = c(0.9, 1.1)),
   type = "factor",
   n_runs = 5)

PF_alt <- alternate(
  PROFLUX,
  \(x) complete_soilphys(x, DSD0_formula = "a*AFPS^b", quiet = TRUE),
  run_map)

evaluate_models(
  PF_alt,
  eval_funs = list("NRMSE_conc" = error_concentration)
    )
#> Joining with `by = join_by(error_parameter)`
#> $best_runs
#> # A tibble: 1 × 2
#>   run_id    ME
#>    <dbl> <dbl>
#> 1      4     0
#> 
#> $model_error
#> # A tibble: 5 × 2
#>   run_id      ME
#>    <dbl>   <dbl>
#> 1      4   0    
#> 2      5   0.777
#> 3      2   1    
#> 4      3   7.60 
#> 5      1 830.   
#> 
#> $models_evaluated
#> # A tibble: 5 × 7
#> # Groups:   error_parameter, site, gas [1]
#>   error_parameter site   gas   NRMSE run_id  sNRMSE parameter_weight
#>   <chr>           <chr>  <chr> <dbl>  <dbl>   <dbl>            <dbl>
#> 1 NRMSE_conc      site_a CO2   0.276      1 830.                   1
#> 2 NRMSE_conc      site_a CO2   0.209      2   1                    1
#> 3 NRMSE_conc      site_a CO2   0.209      3   7.60                 1
#> 4 NRMSE_conc      site_a CO2   0.209      4   0                    1
#> 5 NRMSE_conc      site_a CO2   0.209      5   0.777                1
#> 
#> $best_runs_runmap
#> 
#> A cfp_run_map to be used in alternate(). 
#> number of runs:  1 
#> parameters to alternate:
#>   param param_id
#> 1   TPS        1
#>   run_id param   value   type gas param_id
#> 1      1   TPS 1.07765 factor CO2        1
#> 
# }