Pipeline Helpers - One Call from Path to Multipliers
Source:vignettes/pipeline-helpers.Rmd
pipeline-helpers.Rmd1. When to reach for the convenience helpers
The four-step SUBE chain - import_suts() /
read_figaro() → extract_domestic_block() →
build_matrices() → compute_sube() - gives you
full control over every stage. If you already have the intermediate
objects in hand, call the four steps directly; it is two extra lines.
The manual workflow is covered in detail in the modeling-and-outputs vignette; see
the data-preparation vignette for
input contracts and the canonical format.
These helpers work with any SUT source in the canonical long format,
not just WIOD and FIGARO. The source parameter selects the
matching importer; once data is in canonical sube_suts
shape, the downstream pipeline is identical regardless of origin.
Reach for the convenience helpers in two cases:
- You have a SUT file (or directory) on disk and want multipliers back
in one call: use
run_sube_pipeline(). - You have a pre-imported
sube_sutstable and want to sweep across many country-year slices, collecting tidy merged outputs: usebatch_sube().
Both helpers surface the same structured diagnostics contract so silent data-quality issues (dropped rows, misaligned inputs, singular branches) never disappear from the console.
2. run_sube_pipeline() on sample data
The package ships a tiny WIOD-format sample under
system.file("extdata", "sample", package = "sube"). Feed
its sut_data.csv straight into
run_sube_pipeline():
sut_path <- system.file("extdata", "sample", "sut_data.csv", package = "sube")
result <- run_sube_pipeline(
path = sut_path,
cpa_map = sube_example_data("cpa_map"),
ind_map = sube_example_data("ind_map"),
inputs = sube_example_data("inputs"),
source = "wiod"
)
#> Warning: Pipeline completed with issues: 1 inputs_misaligned. See
#> result$diagnostics for details.
class(result)
#> [1] "sube_pipeline_result" "list"
names(result)
#> [1] "results" "models" "diagnostics" "call"3. Inspecting $results, $diagnostics, and
$call
The compute output lives under $results:
result$results$summary
#> YEAR COUNTRY CPAagg GO VA EMP CO2 FD GOe
#> <int> <char> <char> <num> <num> <num> <num> <num> <num>
#> 1: 2020 AAA P01 1.625000 0.5416667 0.3993056 0.2569444 6 3.192982
#> 2: 2020 AAA P02 1.428571 0.4761905 0.3214286 0.1666667 5 2.339181
#> VAe EMPe CO2e
#> <num> <num> <num>
#> 1: 3.192982 3.324157 3.639344
#> 2: 2.339181 2.229869 1.967213The unified diagnostics table has a 6-column schema -
country, year, stage,
status, message, n_rows:
result$diagnostics
#> country year stage status
#> <char> <int> <char> <char>
#> 1: AAA 2020 build inputs_misaligned
#> 2: AAA 2020 compute ok
#> message
#> <char>
#> 1: Country-year AAA_2020 present in both SUT and inputs but absent from build_matrices model_data.
#> 2: ok
#> n_rows
#> <int>
#> 1: NA
#> 2: NAWhen every status is "ok" no warning is emitted.
Provenance about the call itself lives under $call:
result$call[c("source", "path", "n_countries", "n_years", "estimate",
"package_version")]
#> $source
#> [1] "wiod"
#>
#> $path
#> [1] "/home/runner/work/_temp/Library/sube/extdata/sample/sut_data.csv"
#>
#> $n_countries
#> [1] 1
#>
#> $n_years
#> [1] 1
#>
#> $estimate
#> [1] FALSE
#>
#> $package_version
#> [1] "0.1.2"4. Switching to the FIGARO path
The same helper imports FIGARO flatfiles when
source = "figaro". Because FIGARO data is ~400 MB per
flatfile, the block below is illustrative
(eval = FALSE):
figaro_dir <- Sys.getenv("SUBE_FIGARO_DIR") # user-provided
run_sube_pipeline(
path = figaro_dir,
cpa_map = my_cpa_map, # user-supplied correspondence
ind_map = my_ind_map,
inputs = my_inputs, # industry-level GO/VA/EMP/CO2
source = "figaro",
year = 2023L
)See vignette("figaro-workflow") for the full FIGARO
narrative including the env-var gate (SUBE_FIGARO_DIR) and
map-table construction.
5. batch_sube() across groups
The sample data has one country (AAA) and one year (2020). To illustrate multi-group batching, duplicate the sample into a second year:
sut <- sube_example_data("sut_data")
sut2 <- data.table::copy(sut); sut2[, YEAR := 2021L]
sut_multi <- rbind(sut, sut2)
class(sut_multi) <- c("sube_suts", class(sut_multi))
inp <- sube_example_data("inputs")
inp2 <- data.table::copy(inp); inp2[, YEAR := 2021L]
inp_multi <- rbind(inp, inp2)
batch <- batch_sube(
sut_data = sut_multi,
cpa_map = sube_example_data("cpa_map"),
ind_map = sube_example_data("ind_map"),
inputs = inp_multi
)
#> Warning: Batch completed with 0 error(s) across 2 group(s); issues: 2
#> inputs_misaligned. See result$diagnostics for details.
names(batch$results)
#> [1] "AAA_2020" "AAA_2021"
batch$summary
#> YEAR COUNTRY CPAagg GO VA EMP CO2 FD GOe
#> <int> <char> <char> <num> <num> <num> <num> <num> <num>
#> 1: 2020 AAA P01 1.625000 0.5416667 0.3993056 0.2569444 6 3.192982
#> 2: 2020 AAA P02 1.428571 0.4761905 0.3214286 0.1666667 5 2.339181
#> 3: 2021 AAA P01 1.625000 0.5416667 0.3993056 0.2569444 6 3.192982
#> 4: 2021 AAA P02 1.428571 0.4761905 0.3214286 0.1666667 5 2.339181
#> VAe EMPe CO2e
#> <num> <num> <num>
#> 1: 3.192982 3.324157 3.639344
#> 2: 2.339181 2.229869 1.967213
#> 3: 3.192982 3.324157 3.639344
#> 4: 2.339181 2.229869 1.967213
batch$diagnostics
#> country year stage status
#> <char> <int> <char> <char>
#> 1: AAA 2020 build inputs_misaligned
#> 2: AAA 2020 compute ok
#> 3: AAA 2021 build inputs_misaligned
#> 4: AAA 2021 compute ok
#> message
#> <char>
#> 1: Country-year AAA_2020 present in both SUT and inputs but absent from build_matrices model_data.
#> 2: ok
#> 3: Country-year AAA_2021 present in both SUT and inputs but absent from build_matrices model_data.
#> 4: ok
#> n_rows group_key
#> <int> <char>
#> 1: NA AAA_2020
#> 2: NA AAA_2020
#> 3: NA AAA_2021
#> 4: NA AAA_2021Per-group results are preserved under batch$results;
merged tables under $summary, $tidy, and
$diagnostics (with a group_key column naming
each batch key) are ready for downstream analysis.
6. Turning on estimate = TRUE
When you also want elasticities from
estimate_elasticities(), pass estimate = TRUE.
The sample data is tiny so the regression may return NULL;
on production-sized inputs a sube_models object is
attached:
result_est <- run_sube_pipeline(
path = sut_path,
cpa_map = sube_example_data("cpa_map"),
ind_map = sube_example_data("ind_map"),
inputs = sube_example_data("inputs"),
source = "wiod",
estimate = TRUE
)
#> Warning: Pipeline completed with issues: 1 inputs_misaligned. See
#> result$diagnostics for details.
class(result_est$models) # "sube_models" when model_data is non-empty, else NULL
#> [1] "NULL"7. Reading the diagnostic warnings
The $diagnostics table categorises silent issues into
four statuses:
-
coerced_na(stage = "import") - rows whoseVALUEbecameNAduringas.numeric()coercion at import. Pipeline-level aggregate; count lives inn_rows. -
skipped_alignment(stage = "build") - country-years present in the SUT input but dropped bybuild_matrices()because the CPA or industry correspondence left no aligned rows. -
inputs_misaligned(stage = "build") - country-years present in both the SUT data and theinputstable but absent frombuild_matrices()’s$model_data. Typical cause: industry codes ininputsthat do not match the matrix industries. -
singular_supply/singular_go/singular_leontief(stage = "compute") -compute_sube()could not invert the relevant matrix for this country-year. Pass-through from the compute stage’s existing diagnostics.
When any row has status != "ok", a single
warning() summarises counts per category. Query the
structured table for programmatic follow-up; read the warning for a fast
interactive heads-up.