Skip to contents

Standalone Bayesian BTL with CmdStan

Source: vignettes/bayesian-btl.Rmd
bayesian-btl.Rmd

fit_bayes_btl_mcmc() fits a Bayesian Bradley–Terry–Luce (BTL) model to an existing fixed set of pairwise outcomes. It reuses the adaptive fit and summary contracts but does not perform adaptive pair selection.

The integrated CJ workflow explains how Bayesian EAP reliability differs from frequentist SSR, empirical score recovery, and held-out prediction. It also introduces the experimental frequentist lapse/position estimator that matches the btl_e_b likelihood without Bayesian priors.

When to use this guide

Use this workflow when you already have pairwise winners and want Bayesian score estimates, uncertainty, and convergence diagnostics. It is also a route to the completed Bayesian input required for rubric calibration. To let the package choose new comparisons during collection, use the adaptive guide instead.

You need: recorded comparisons, their item IDs, and CmdStan. You get: one score summary per item plus fit diagnostics. The code below uses synthetic outcomes; the short sampling settings demonstrate the interface, not a validated assessment or a sufficient sampling budget for your study.

Prerequisites

The fit requires the suggested cmdstanr and parallelly packages, CmdStan, and a working C++ toolchain. Installation is a machine-level setup and does not require provider credentials. Fitting chunks are shown but not evaluated during document builds. Run them interactively after setup; no provider requests are needed for this example.

install.packages("parallelly")
install.packages(
  "cmdstanr",
  repos = c("https://stan-dev.r-universe.dev", getOption("repos"))
)
cmdstanr::check_cmdstan_toolchain(fix = TRUE)
cmdstanr::install_cmdstan()
cmdstan_available <- requireNamespace("cmdstanr", quietly = TRUE) &&
  tryCatch(!is.null(cmdstanr::cmdstan_version()), error = function(e) FALSE)
cmdstan_available
#> [1] FALSE

A TRUE value means CmdStan was found. Compilation and model fitting are separate checks performed when you run the fit.

Prepare modeling input

“Canonical” means the standard column layout expected by this workflow. The public builder accepts ID1, ID2, and better_id, validates that each winner belongs to its pair, and adds deterministic keys, iteration values, timestamps, and provenance columns.

library(pairwiseLLM)

data("example_writing_pairs", package = "pairwiseLLM")
observed <- example_writing_pairs[seq_len(min(40L, nrow(example_writing_pairs))), ]
results_tbl <- build_btl_results_data(
  observed,
  backend = "offline_fixture",
  model = "deterministic_observations"
)
ids <- sort(unique(c(results_tbl$A_id, results_tbl$B_id)))

results_tbl[, c("pair_uid", "A_id", "B_id", "better_id", "winner_pos", "phase")]
#> # A tibble: 40 × 6
#>    pair_uid  A_id  B_id  better_id winner_pos phase 
#>    <chr>     <chr> <chr> <chr>          <int> <chr> 
#>  1 S01:S02#1 S01   S02   S02                2 phase2
#>  2 S01:S03#1 S01   S03   S03                2 phase2
#>  3 S01:S04#1 S01   S04   S04                2 phase2
#>  4 S01:S05#1 S01   S05   S01                1 phase2
#>  5 S01:S06#1 S01   S06   S06                2 phase2
#>  6 S01:S07#1 S01   S07   S07                2 phase2
#>  7 S01:S08#1 S01   S08   S08                2 phase2
#>  8 S01:S09#1 S01   S09   S09                2 phase2
#>  9 S01:S10#1 S01   S10   S10                2 phase2
#> 10 S01:S11#1 S01   S11   S11                2 phase2
#> # ℹ 30 more rows

The four likelihood variants are:

  • "btl": item parameters only;
  • "btl_e": item parameters plus a lapse/error component;
  • "btl_b": item parameters plus a presentation-position effect;
  • "btl_e_b": both lapse and position components (the default).

Choose the variant before inspecting results, based on the study design and estimands. A more complex variant is not automatically preferable, especially with sparse data.

Fit and summarize

The following seeded sampling example is not evaluated during documentation builds. Run it after the availability check returns TRUE. The small iteration count keeps the walkthrough practical and is for workflow demonstration, not production inference.

fit <- fit_bayes_btl_mcmc(
  results = results_tbl,
  ids = ids,
  model_variant = "btl_e_b",
  cmdstan = list(
    chains = 2L,
    parallel_chains = 2L,
    iter_warmup = 250L,
    iter_sampling = 250L,
    seed = 7007L,
    core_fraction = 1
  )
)

refits <- summarize_refits(fit)
items <- summarize_items(fit)

refits[, c(
  "round_id", "total_pairs", "diagnostics_pass",
  "divergences", "max_rhat", "min_ess_bulk", "reliability_EAP"
)]
items[, c("ID", "theta_mean", "theta_sd", "rank_mean", "deg")]

The item scale is relative and identified by the model constraints; its absolute origin is not an external score. Posterior SDs and intervals are conditional on the selected model and observed judgments. Check divergences, R-hat, effective sample size, and sampling warnings before interpreting ranks. A failed diagnostic is not repaired by hiding the warning or reporting only posterior means.

What the output means

Output Interpretation
theta_mean, theta_sd Posterior mean relative quality and posterior uncertainty for each item.
rank_mean Posterior mean rank; uncertainty can make it non-integer.
deg Number of committed comparisons involving that item.
diagnostics_pass Whether the fit meets the implemented diagnostic checks; inspect individual diagnostics too.
reliability_EAP Posterior mean score variance divided by that variance plus average posterior variance; an internal model-based summary, not empirical recovery.
max_rhat, min_ess_bulk, divergences Sampling checks for chain agreement, effective information, and numerical problems.

Higher quality locations indicate stronger writing, but adjacent ranks can be uncertain. Keep both the item summaries and diagnostic table when saving or reporting an analysis.

Chains and CPU allocation

chains sets the total number of chains; parallel_chains sets how many run at once. Four chains can therefore run two at a time. threads_per_chain defaults to one; the bundled models do not use within-chain parallel likelihoods, so increasing it does not accelerate likelihood evaluation.

When parallel_chains is omitted, scheduling uses at most two available CPU slots and applies core_fraction = 0.8, retaining at least one slot. On a two-slot allocation this schedules one chain at a time. Detection respects scheduler, container, and check constraints without changing the total chains. The historical default chain count remains min(8, detected physical cores), with logical-core and one-core fallbacks when hardware detection is unavailable. Set chains explicitly when you need the same count across machines.

An explicit parallel_chains request bypasses core_fraction and is capped at the total chains. Its product with threads_per_chain must fit the available allocation; otherwise fitting stops with guidance. For example, cmdstan = list(chains = 4L, parallel_chains = 4L) requests four CPU slots and prints a resource message when sampling. Active check limits still cap the allocation at two. Examples and package checks stay within that limit.

If resource detection fails, fitting warns and uses one CPU slot. Independent concurrent fits each see the same allocation, so divide resources among them yourself. summarize_refits(fit) reports mcmc_cores_available, mcmc_parallel_chains_requested, mcmc_concurrency_budget, and mcmc_concurrency_used. An automatic request is NA; historical fits without this metadata also report NA for the new fields.

Cumulative refits

pair_counts fits increasing subsets of the same data and records one round/item-log view per subset. subset_method = "first" follows row order. "sample" creates one seeded permutation and uses nested prefixes, so provide seed for reproducibility.

cumulative <- fit_bayes_btl_mcmc(
  results_tbl,
  ids = ids,
  pair_counts = c(20L, 40L),
  subset_method = "sample",
  seed = 7007L,
  cmdstan = list(chains = 2L, parallel_chains = 2L, seed = 7007L)
)

summarize_refits(cumulative)
summarize_items(cumulative, refit_id = 2L)

Files, failures, and reproducibility

CmdStan writes generated C++/executables and sampling CSV files to its cache or configured cmdstan$output_dir. Preserve the seed, model variant, CmdStan configuration, package version, and diagnostics with reported results. Missing CmdStan, invalid schemas, unknown IDs, invalid winners, empty data, invalid subset sizes, and unsupported variants abort explicitly. Sampling failures and poor diagnostics should be investigated rather than converted to partial rankings.

For adaptive selection and stopping, use Guide: Adaptive Pairing. For fixed-pair frequentist alternatives, see fit_bt_model() and fit_elo_model().

Predictive warm-start priors

fit_bayes_btl_mcmc() accepts warm_start_prior, constructed with make_warm_start_prior() from calibrated model predictions or explicit expert scores. The four active variants use centered predictive locations and a default raw-theta SD of 0.5. Omitting the argument preserves raw means zero and SDs one. Ensemble prediction SD is diagnostic disagreement, not an automatic prior SD. See Guide: Adaptive Warm Start for the offline workflow.

For downstream performance levels or calibration to human rubric scores, see Guide: Rubric Calibration. It consumes the completed fit directly and supports all four Bayesian BTL variants.

Citation

Mercer, S. H. (2026). Standalone Bayesian BTL with CmdStan [R package vignette]. Comprehensive R Archive Network. https://doi.org/10.32614/CRAN.package.pairwiseLLM