Train a task-specific warm-start model with nested validation

A warm-start model predicts relative writing quality before pairwise comparisons. One fitting call represents one assessment; do not pool unlinked BT/BTL scales.

Usage

fit_warm_start_model(
  ids,
  theta,
  task_id,
  texts = NULL,
  features = NULL,
  schema = "writing_features_v1",
  python = NULL,
  seed = 1L,
  outer_folds = 5L,
  inner_folds = 5L,
  alpha_grid = seq(0, 1, by = 0.025),
  lambda_rule = c("lambda.1se", "lambda.min"),
  engine = c("glmnet", "pls", "svr_rbf"),
  cv_plan = NULL,
  engine_control = NULL
)

Arguments

ids: Unique item IDs, aligned positionally with theta and texts.
theta: Finite numeric BT/BTL scores from one assessment, one per ID.
task_id: One nonblank assessment label, stored as provenance. It does not group rows, affect fitting, or identify a registered model.
texts: Optional character vector of texts. Supply exactly one of texts and features. Extraction runs once before resampling.
features: Optional precomputed feature table accepted by predict.pairwiseLLM_warm_model(), aligned to ids by its item_id column.
schema: Frozen feature schema identifier.
python: Existing Python interpreter path for text extraction only.
seed: Integer from zero through .Machine$integer.max. Local randomization preserves the caller's RNG state. Defaults to 1.
outer_folds, inner_folds: Integer fold counts, at least two; defaults are five each. Every fitting split needs three rows and a nonconstant outcome. Requested fold counts are never reduced automatically.
alpha_grid: Unique finite alpha candidates in [0, 1], sorted internally. Defaults to 41 values from ridge to lasso in increments of 0.025.
lambda_rule: Deployment rule: "lambda.1se" (default) or expert override "lambda.min". The rule applies to outer models and the final model alike.
engine: Fitting engine: "glmnet" (default), "pls", or "svr_rbf". Explicit alpha_grid or lambda_rule arguments are accepted only for glmnet.
cv_plan: Optional make_warm_start_cv_plan() object with exactly matching task, ordered IDs and outcomes. Omitted seed/fold arguments defer to the plan; explicitly conflicting arguments fail. A supplied plan is never regenerated.
engine_control: For PLS, an optional named list containing ncomp, a vector of unique positive integer candidate component counts, at most 10. Every candidate must be legal in every required inner fit and context refit. The default uses all counts from one through the common legal maximum in each tuning context. For SVR, only cost and gamma_multiplier are accepted: unique finite positive candidate vectors, defaulting independently to 2^(-2:4) and 2^(-2:2). Epsilon is fixed at 0.10. Unknown controls fail. For glmnet use alpha_grid and lambda_rule instead.

Value

A portable pairwiseLLM_warm_model with deployment preprocessing, numeric engine parameters and OOF calibration, plus tuning traces, fold assignments, outer raw/calibrated predictions, transformed held-out outcomes, validation metrics and warnings. Audit records include item IDs and outcomes but no raw training texts. Each outer record includes its tuning, scaling, preprocessing, calibration and hyperparameters (linear models also record nonzero counts). Final tuning metadata is separate from outer validation. No backend fit is retained. PLS coefficients and Ymeans - Xmeans %*% beta reproduce backend predictions on the stored preprocessed predictor scale without requiring pls at deployment. SVR stores numeric support vectors, dual coefficients, rho and actual gamma; prediction needs no e1071. Its linear coefficients/intercept are NULL.

Details

New public fits use engine-neutral format 3 with explicit full audit status, a reusable CV plan, and a numeric deployment payload. Legacy formats 1/2 remain supported. Construct a plan once to reuse partitions across feature schemas.

Folds balance outcome ranges using consecutive outcome-ranked blocks, with randomized ties and distinct randomized fold labels within each block. Predictor missingness filtering, median imputation, near-zero-variance removal and sample-SD scaling are learned separately in every inner training split. Each outer training set defines its outcome mean and sample SD; both its inner fits and its held-out outcomes use that scale. See pairwiseLLM_warm_model for preprocessing defaults. No PCA or outcome-based feature screening is used.

For each alpha, glmnet constructs a reference path on the current tuning dataset with its own preprocessing: 100 requested lambdas, minimum ratio 0.01 if retained predictors outnumber rows, otherwise 0.0001. The actual returned path is stored. Each inner training split fits those exact penalties with its own preprocessing; reference preprocessing is not used to transform inner rows. Candidate penalties use the entire tuning dataset, including its inner holdouts. No interpolation, extrapolation, or omitted fold is accepted for an eligible candidate. When an inner fit reports an iteration-limit failure and returns a verified prefix of the requested path, only its missing tail is invalidated. A lambda is eligible only if it converges in every inner fold. CV summaries and both lambda rules use eligible candidates only; an alpha with none is excluded. If all alphas are excluded, tuning fails. Reference-path generation, malformed engine output, other engine errors, and selected-lambda refit failures remain fatal. Full audits retain versioned candidate-validity records with requested and returned fold lambdas, convergence codes, counts, masks, invalid-tail counts, alpha eligibility and whether selection reached the smallest eligible lambda. Selection indices refer to the original path; unavailable losses and summaries are NA, never imputed. Historical fully converged audits remain valid.

CV error is the observation-count-weighted mean of fold mean squared errors. Its SE is sqrt(weighted.mean((fold_mse - cvm)^2, fold_sizes) / (K - 1)). Errors within 1e-10 * max(1, abs(a), abs(b)) are numerical ties. Lambda minimum ties favor the largest penalty. Alpha selection compares CV error at each alpha's lambda minimum and favors smaller alpha in a tie. The default selected penalty is the largest lambda within one SE of that alpha's minimum error.

PLS uses the optional pls package with explicit method = "kernelpls", scale = FALSE, validation = "none", and centering. Preprocessing remains owned by this package. Each tuning context uses a common component grid bounded by every inner training matrix and the context refit: centered QR rank at tolerance 1e-7, retained predictor count, training row count minus one, and 10. Explicit candidates are never silently dropped. Weighted MSE and SE follow the rules above; minimum-error ties and eligible 1-SE choices favor fewer components. Nonfinite or degenerate fits fail with context, without fallback. All candidate OOF predictions, fold losses, rank bounds and selection evidence are retained. Stored rank bounds are checked for consistency; recomputing rank itself requires the original feature table, which is not stored in the model.

RBF-SVR uses optional e1071, with type = "eps-regression", kernel = "radial", scale = FALSE, cross = 0, probability = FALSE, and fixed epsilon = 0.10. Every fit divides its gamma multiplier by its own retained predictor count. The complete Cartesian grid is tuned with the weighted MSE/SE above. Both minimum-error ties and eligible 1-SE choices favor lower cost, then lower gamma multiplier. Full audits retain candidate OOF values, fold losses, retained counts and actual gammas. Failed candidates are never omitted.

OOF (out-of-fold) predictions are predictions for rows excluded from their corresponding engine fit. Calibration regresses standardized outcomes on selected-hyperparameter OOF predictions using ordinary least squares. Those same folds select hyperparameters: calibration fit statistics are not independent performance estimates. Only untouched outer predictions define validation metrics. Calibration needs at least three rows and full rank at QR tolerance 1e-7; constant/degenerate predictions or nonfinite coefficients cause an explicit error. Finite negative slopes are allowed. Undefined validation diagnostics are recorded as NA with reasons, without an identity-calibration fallback.

After outer validation, full-data tuning and OOF calibration precede the final all-row engine refit. Raw and calibrated predictions use within-task standardized units, not original BTL units. They are not Bayesian prior SDs. The selected engine package and withr are required only for model development; Python is required only for the text-input path. This function never installs software or downloads data.

Examples

if (requireNamespace("glmnet", quietly = TRUE) &&
    requireNamespace("withr", quietly = TRUE)) {
  local({
    # Synthetic features illustrate the interface, not predictive validity.
    example_features <- function(seed) {
      withr::local_seed(seed)
      fields <- warm_start_feature_schema()$feature
      x <- as.data.frame(matrix(runif(15 * length(fields)), nrow = 15))
      names(x) <- fields
      x$n_tokens <- 11:25
      x$token_length_mean <- 2 + 10 * x$token_length_mean
      x$token_length_std <- 0.2 + x$token_length_std
      x$dale_chall_readability_score <- 5 + 20 * x$dale_chall_readability_score
      x <- data.frame(item_id = as.character(1:15), x)
      attr(x, "warm_start_schema") <- "writing_features_v1"
      x
    }
    features <- example_features(3103)
    theta <- 10 + 0.4 * features$n_tokens - 2 * features$token_length_mean
    # A small alpha grid keeps this example fast; the default has 41 values.
    model <- fit_warm_start_model(features$item_id, theta, "synthetic-a",
      features = features, alpha_grid = c(0, 1))
    summary(model)
    model$validation$metrics
  })
}
#> $pearson_r
#> [1] 0.9897764
#> 
#> $squared_pearson_r
#> [1] 0.9796574
#> 
#> $spearman_rho
#> [1] 0.9535714
#> 
#> $rmse
#> [1] 0.1600619
#> 
#> $mae
#> [1] 0.08329452
#> 
#> $calibration_intercept
#> [1] -0.04764372
#> 
#> $calibration_slope
#> [1] 0.9501558
#> 
#> $undefined_reasons
#> character(0)
#>

Usage

Arguments

Value

Details

See also

Examples