Skip to contents

Changelog

Source: NEWS.md

pairwiseLLM 1.5.2

Warm-start prediction

  • Added ensemble_warm_start_algorithms() for equal-weight same-task predictions with honest aligned outer-CV validation. Full/reduced portable artifacts work with storage, registries, bundles and BTL/TrueSkill warm starts. Component disagreement remains diagnostic; cross-task ensemble semantics are unchanged (#259).

  • Added optional RBF-SVR warm starts with the fixed epsilon/grid conventions, split-specific gamma, weighted nested-CV tuning and deterministic 1-SE selection. Complete format-3 audits and portable numeric support vectors support full/reduced deployment without e1071; nonlinear coefficient requests fail explicitly (#259).

  • Added optional PLS warm-start models using explicit kernel PLS, legal common component grids, weighted nested-CV tuning and fewer-component 1-SE selection. Full format-3 audits retain candidate OOF evidence; portable full/reduced models predict and supply warm priors without pls (#259).

  • Added explicit writing_features_v2 extraction with 46 source-audited features, schema-specific hashes/decoding, and pinned readability/missingness behavior. The default 20-feature v1 schema and Python environment remain unchanged (#259).

  • Added reusable task/outcome-bound warm-start CV plans and portable format-3 glmnet models, including full/reduced storage, registries, bundles, and warm priors. Legacy formats 1/2 and frozen v1/glmnet numerical behavior remain supported (#259).

Documentation

  • Added a complete offline v2 workflow using fabricated texts and outcomes, shared validation splits, three algorithms, and BTL/TrueSkill initialization. Updated warm-start and adaptive-pairing guidance for non-technical readers, distinguishing cross-task and same-task ensembles and user-chosen prior SD.
  • Corrected model, reduction, and Python audit documentation; synchronized current package version references. R >= 4.4 and optional fitting dependencies are unchanged.

pairwiseLLM 1.5.1

Improvements

  • Added sparse adaptive replay reservoirs with one frozen observation per unordered edge. All four within-set pairing strategies respect unused allowed edges and recorded presentation orientation. Reservoir runs use a common seeded spanning-tree bootstrap, validate identity on resume, and distinguish exhaustion from a temporarily unavailable preferred candidate (#257).

  • MCMC scheduling now respects available CPU allocations through the suggested parallelly package, required when sampling. Automatic concurrency retains the two-slot ceiling and core_fraction headroom. Explicit parallel-chain requests remain supported, with joint chain/thread budget validation and clear errors for excessive requests. Fits and logs record requested and effective resource settings without changing total chains or sampling settings (#255).

Documentation

  • Reworked the README and Getting Started around a runnable offline example, score interpretation, and a sequential path to collecting new comparisons.
  • Added practical reading guidance, prerequisites, output explanations, and recovery advice across the vignettes, while retaining formal statistical design content. Corrected pair-count, example-order, and CmdStan setup guidance.
  • Organized pkgdown navigation and function reference by user task, separated practical guides from statistical design articles, and clarified introductory function help. Existing article URLs and package interfaces are preserved.

Bug fixes

  • Batch polling and result downloads now retry transient HTTP and network failures consistently across OpenAI, Anthropic, and Gemini, honoring Retry-After. Multi-batch resume defers exhausted transient retrievals while surfacing permanent errors immediately. Retrieval retries reuse existing batch IDs without adding scientific attempts; polling timeouts remain checked between status requests. Requires httr2 >= 1.1.0 (#251).

  • OpenAI Batch requests now accept store on both endpoints and max_output_tokens on Responses, including through the batch wrappers. Explicit storage values are preserved and omitted controls remain absent. The new openai_download_batch_errors() helper downloads request failures separately from successful output for ID-based reconciliation (#249).

  • Both OpenAI live endpoints now preserve explicit service_tier requests across model families. "standard" and "default" send "default"; "auto", "flex", and "priority" are forwarded unchanged. Omitted or NULL tiers leave the field absent, preserving OpenAI’s project-configured auto behavior (#247).

  • OpenAI live comparisons now validate and forward explicit store = TRUE and store = FALSE for both Responses and Chat Completions, including row-wise wrappers (#245).

  • Gemini Developer API live and batch comparisons now validate and forward store to control request logging. Omitted or NULL values preserve provider defaults (and Gemini project settings). These controls do not imply a general data-retention guarantee.

pairwiseLLM 1.5.0

Rubric calibration

  • Added fit_rubric_calibration(), S3 predict(), and evaluate_rubric_predictions() for completed Bayesian comparative judgments. All four variants (btl, btl_e, btl_b, btl_e_b) and completed fixed or adaptive results share the downstream interface. Fixed live/batch collection remains provenance only.
  • Added percentile scoring with type-8 quantiles, preserved ties, and requested versus achieved proportions. These are norm-referenced, distribution-matched performance levels; human rubric labels are not fitted.
  • Added default ordinal_linear proportional-odds calibration and explicit ordinal_monotone calibration with a nondecreasing penalized spline. Ordinal predictions retain full probabilities, median-category scores by default, optional modal scores, expected internal level, and extrapolation flags.
  • Both ordinal methods support a labeled subset within one CJ fit (same_set) and a reusable Phase A rubric reference (linked_anchors). Linked target prediction requires completed existing Phase B outputs and reuses the stored reference transformation after identity and fit-contract checks. Phase B estimation, selection, probes, and stopping are unchanged.
  • Added normalized ranked probability score, log loss, cumulative calibration summaries, hard-score metrics, and optional training-model diagnostics. Internal label-only cross-validation retains completed CJ evidence and fits scaling and calibration from training labels alone. No automatic method selection is provided.
  • Each analytic trait requires its own CJ analysis and calibration. Missing intended calibration categories fail explicitly; orientation is never silently reversed. All methods condition on accepted CJ point locations: CJ uncertainty is retained where available but is not propagated. Joint CJ/rubric likelihood estimation is not implemented.

Requirements and documentation

  • Raised the minimum supported R version to 4.4.
  • Linear fitting uses optional ordinal; monotone fitting and prediction use optional mgcv (>= 1.9-4), with optional withr for reproducible fitting. Missing backends give installation guidance without substituting methods.
  • Added a practical rubric-calibration guide and integrated reference help, README, and website navigation, including same-set validation and historical reference reuse through Phase B.

pairwiseLLM 1.4.0

Adaptive ranking

  • Added predictive warm-start modes cold, btl_only, trueskill_only, and both. Omitted mode preserves historical behavior: cold without predictive input and btl_only with it. Request both explicitly to initialize both models.
  • TrueSkill-warm modes initialize item locations as 25 + (25/3) * prior_mean, with fixed scale 1 and unchanged sigma. Every mode retains the same seeded shuffled connected bootstrap of N - 1 valid comparisons.
  • Added post-bootstrap random, trueskill_p50, and trueskill_pollitt strategies through adaptive_config$pairing_strategy; hybrid remains the default. Direct strategies use a minimum-degree focal item and currently support only within-set runs. Pollitt-inspired targets use TrueSkill probabilities.
  • Within-set and Phase-A hybrid rolling anchors and the long-link probability gate now use TrueSkill throughout. Bayesian BTL still supplies item estimates, uncertainty, reliability, diagnostics, stopping, and the global_identified signal that can affect later hybrid routing. Phase B linking is unchanged.
  • Sessions persist effective mode, strategy, predictive provenance, and current TrueSkill state. Legacy sessions migrate to cold or historical btl_only and hybrid, preserving saved bootstrap and round progress. Phase-A artifact identity distinguishes generation settings while preserving explicit imports.
  • Added validate_adaptive_replay() and make_adaptive_judge_replay() for offline replay of exact directed judgments, with strict reuse protection. Replay users can set adaptive_config$dup_max_obs_relaxed = 2L; the general hybrid default retains its conditional third observation.

Fixes and documentation

  • Within-set continuation retains saved BTL settings when no override is supplied; round recovery uses the exact saved round field.
  • Updated adaptive help, guides, design documentation, and replay examples to distinguish predictive initialization, connected bootstrap, and pairing strategy.
  • Strengthened adaptive-session validation and resume handling, including preservation of named Phase-A evidence hashes, legacy pair-resume behavior, and improved detection of Ollama results.
  • Made stratified cost estimation robust to singleton strata and expanded boundary validation for adaptive state, warm-start models, provider responses, batch payloads, optional dependencies, and sampler interfaces.
  • Simplified linking internals around the anchored-joint refit path by removing obsolete transform-refit Stan support and unreachable legacy Phase-B branches, while preserving legacy state normalization, audit fields, persistence compatibility, and current linking behavior.
  • Added regression coverage for fresh-process resume, refit diagnostics, selection and information updates, state reporting, schema validation, and runtime boundary behavior.

pairwiseLLM 1.3.2

New Features

Adaptive pairing warm start

  • Added warm start model development workflow: task-specific
    elastic-net training with nested cross-validation and out-of-fold calibration, and portable model storage.
  • Added user model registration, equal-weight task-model ensembles, and centered predictive priors for standalone and adaptive Bayesian BTL estimation. Adaptive sessions retain numeric priors for resume without the original model or Python.
  • Added the adaptive warm start help family and a practical workflow guide. Python is optional for extraction; precomputed prediction needs neither Python nor glmnet. Real bundled predictors remain deferred; no default model ships.
  • Added warm_start_coefficients() for inspecting calibrated standardized elastic-net coefficients from individual models and ensemble components, preserving the distinction between preprocessing removal and shrinkage to zero.

pairwiseLLM 1.3.1

CRAN release: 2026-09-07

New Features

Adaptive linking

  • Added cross-set adaptive linking through adaptive_rank(), with one-spoke and multi-spoke hub-and-spoke run modes for placing separately ranked sets on a shared scale.
  • Phase A can run canonical within-set ranking, import saved Phase A artifacts, or combine imported and newly run sets. Phase B uses anchored-joint estimation with hard-locked hub parameters, globally shared judge parameters, and concurrent spoke processing.
  • Added D-optimal cross-set candidate selection, spoke-aware comparison budgets, and fixed per-refit held-out probes with explicit coverage, calibration, and stopping gates.
  • Added persistent Phase A manifests and linking sessions, resume support, link_stage_log, and explicit blocker and invalid-judgment reporting for auditable long-running workflows.

Model support & live API improvements

  • Added a separate Vertex AI Gemini live backend via backend = "vertex" with VERTEX_API_KEY.
    • Vertex remains live-only in this series; generic batch wrappers reject backend = "vertex" explicitly rather than routing it through Gemini batch helpers.
  • Clarified provider-specific service_tier behavior for Gemini Developer API and Vertex AI Gemini API in the package docs, README, and vignettes.
  • Added request-shape handling for the OpenAI GPT-5.x and GPT-5.6 model families.

Bayesian BTL usability

  • Added exported build_btl_results_data() to convert conventional ID1/ID2/better_id outcomes into the canonical input required by fit_bayes_btl_mcmc().
  • Standalone Bayesian BTL fits now report reliability and sampler diagnostics through the same result and log surfaces used by adaptive workflows.

Improvements

  • Reduced adaptive-linking refit, candidate-generation, logging, and checkpoint overhead for large runs, including bounded candidate construction and analytic anchored-joint derivatives.
  • Hardened adaptive-linking resume behavior, Phase A stopping scope, spoke routing, probe scheduling, and Phase B pair accounting.

Documentation

  • Updated the backend support matrix to distinguish Gemini Developer API from Vertex AI Gemini API and to document the live-only Vertex scope.
  • Replaced the undated supported-model list with a dated, machine-readable compatibility registry that separates backend implementation, live/batch testing, and current provider availability.
  • Corrected stale provider identifiers, API-key and CmdStan setup guidance, OpenAI Flex wording, cost-estimator summaries, and broken public examples.

Bug fixes

  • Fixed Anthropic extended-thinking live requests when users explicitly set the required temperature = 1.
  • OpenAI Responses live requests now forward and enforce an explicitly supplied max_output_tokens limit.
  • Changed omitted temperature and top_p controls to use model/provider defaults instead of injecting deterministic sampling values. Provider-required reasoning constraints are still enforced.
  • Provider smoke runs now exit unsuccessfully when a selected API key is absent, unless missing-key skips were explicitly allowed.

pairwiseLLM 1.3.0

New Features

Adaptive pairing & ranking framework

  • Introduced an adaptive pairing / adaptive ranking framework designed to reduce unnecessary comparisons through uncertainty-aware pair selection and Bayesian inference.

  • Added adaptive_rank(), the primary user-facing wrapper that runs the complete adaptive workflow end-to-end, including warm start, adaptive pairing rounds, Bayesian BTL refits, diagnostics, and stopping.

  • Advanced control is available via:

  • Adaptive pairing is organized into rounds that balance global scale identification and local refinement using a mixture of anchor, long-range, mid-range, and local comparisons.

  • The adaptive controller tracks a global identifiability state based on Bayesian diagnostics and agreement between online (TrueSkill) and global (BTL) rankings. Once the global scale is identified:

    • long-range comparisons are automatically tapered,
    • comparison budget is reallocated toward local and boundary-refining pairs,
    • exploration rates are reduced to focus on decision-relevant uncertainty.

  • Long-range comparisons are additionally posterior-gated in later stages, preventing wasted comparisons on pairs that are already decisively ordered.

  • Late-stage local pairing prioritizes near-tie pairs, with limited, auditable overrides to degree caps when especially informative comparisons are blocked.

  • Adaptive runs produce fully auditable step-, round-, and refit-level logs, recording candidate generation, fallbacks, gating decisions, quota reallocations, and stopping criteria.

  • All adaptive workflows use standardized configuration, state, and logging contracts to ensure reproducibility and future extensibility.

Bayesian Bradley–Terry–Luce (BTL) modeling

  • Added a fully Bayesian Bradley–Terry–Luce (BTL) model implemented via CmdStan, providing posterior uncertainty estimates for item skill parameters.
  • New entrypoint fit_bayes_btl_mcmc() enables direct posterior inference from pairwise comparison data, independent of or integrated with adaptive workflows.
  • Supports multiple model variants (including error and positional bias extensions) and optional refitting on increasing subsets of comparisons.
  • Bayesian BTL outputs integrate seamlessly with adaptive ranking utilities (summarize_items(), summarize_refits()), serving as the statistical backbone for adaptive pairing decisions.

Model support & live API improvements

  • Added support for the Gemini Flash model gemini-3-flash-preview for live pairwise comparisons.
  • Added support for OpenAI service tiers / priority routing via service_tier for applicable live models.
    • Enables tiers such as "flex" and "priority" when supported by the selected model.
    • Integrated into the live submission path without requiring changes to calling code.

Documentation

  • Expanded and clarified documentation for adaptive ranking, Bayesian BTL, and live model configuration.
  • Updated examples to reflect new adaptive and Bayesian APIs.

pairwiseLLM 1.2.0

New Features

  • Parallel Processing:
    • submit_llm_pairs() and backend-specific live functions (OpenAI, Anthropic, Gemini, Together, Ollama) now support parallel execution via parallel = TRUE and workers = n (requires the future package).
  • Incremental Saving & Resume:
    • Added save_path argument to live submission functions. Results are saved to CSV incrementally, allowing interrupted jobs to resume automatically by skipping previously processed pairs.
  • Robust Error Handling:
    • Failed API calls no longer stop the entire process. Failures are captured and returned separately, allowing for easier inspection and re-submission.
  • Added estimate_llm_pairs_cost() to estimate costs in live and batch mode.
  • Introduced llm_submit_pairs_multi_batch() and llm_resume_multi_batches() to split large comparison sets across multiple batches and resume polling later. These helpers support writing per‑batch and combined results, along with an optional jobs registry.

Bug fixes

  • The prompt format for anthropic batch comparisons now match the anthropic live format.
  • Reverse consistency functions can now handle duplicate pairs.

Breaking Changes

  • submit_llm_pairs() and its backend-specific counterparts now return a list containing two elements: $results (a tibble of successful comparisons) and $failed_pairs (a tibble of inputs that failed). Previous versions returned a single tibble.

pairwiseLLM 1.1.0

CRAN release: 2025-12-22

Models

  • Added GPT-5.2
  • Ensured models can be called with date format, e.g. gpt-5.2-2025-12-11
  • Default temperature setting is set to 0 for non-reasoning models, provider default for reasoning models (typically 1)

Tests

  • Tests added to improve coverage

Documentation

  • Changed pkgdown site layout
  • Added codemeta.json
  • Added repo logo
  • Updated function examples
  • Add references to Description

Miscellaneous

  • No longer set global variables, now done in individual functions
  • Added verbose option in fit_bt_model() and summarize_bt_fit()
  • Moved null coalescing helper to separate R file
  • Changed validation of API keys in multiple functions

pairwiseLLM 1.0.0

  • Initial release.
  • Unified live and batch LLM comparison framework (OpenAI / Anthropic / Gemini).
  • Live support for Together.ai and local Ollama backends.
  • Tools for Bradley–Terry and Elo models, positional bias checks