Skip to contents

Changelog

Source: NEWS.md

pairwiseLLM (development version)

  • Document OpenAI Decisions and TypeSafe Jev setup, probability interpretation, live collection, caching and recovery (#337). The dated October 16 synthetic pilot confirms compatibility for gpt-6-luna/decisions and jev-1.13.0/systemone, with differing reversal consistency; it does not establish probability calibration or educational validity. Offline qualification extends bulk request integrity, modeling compatibility and privacy regressions. Default saved runs omit raw responses; in-memory failure objects can still contain sensitive provider content when serialized by callers.

  • OpenAI Decisions and TypeSafe Jev support bounded concurrent live collection through submit_llm_pairs() and provider wrappers (#336). Shared integrity validation binds every transmission and saved result to its scientific request. Locked atomic RDS journals preserve full probabilities, valid results and attempt history; CSV exports can be rebuilt safely. Retry budgets survive restarts, including at most one automatic reissue of ambiguous interrupted work. Optional promises/later enable concurrency; filelock is required for saved decision runs. See ?decision_bulk for controls, recovery guarantees and privacy limitations.

  • Single-pair comparisons support opt-in backend = "typesafe" and the direct typesafe_compare_pair_live() helper (#335). The default is versioned jev-1.13.0; explicit Jev versions and documented moving aliases are supported without availability guarantees or fallback. TYPESAFE_API_KEY is included in credential checks. Jev reuses the shared decision prompt, request-to-wire integrity checks, response parser, and probability-preserving normalization. TypeSafe overload/rate-limit retries preserve validated bytes; existing OpenAI Decisions, generative endpoints, and binary modeling are unchanged. Resumable live bulk execution is available; managed Batch remains unsupported.

  • Single-pair OpenAI comparisons now support opt-in endpoint = "decisions" with gpt-6-luna (#334). The frozen decision instructions, selected winner, both probabilities, independent confidence, usage and provenance are retained. Scientific request bodies and final serialized bytes are checked before HTTP, including retries. Unsupported controls fail early; existing generative endpoints are unchanged. Resumable live bulk Decisions is available; managed Batch remains unsupported.

  • Internal decision-model foundations freeze the approved decision_cj_v1 instructions and construct equivalent offline OpenAI Decisions/TypeSafe Jev requests (#333). Parsers retain both probabilities, the selected winner, independent confidence, and sample-free request provenance. Strict identity and completeness checks exclude malformed, refused, or choice/probability-mismatched decisions from model evidence. Existing generative defaults and binary BT inputs are preserved; live integration and resumable execution use these contracts.

  • Require rlang >= 1.3.0 for stable linked-input and runtime hashing, and optional withr >= 3.0.3 for correct local RNG restoration. glmnet remains optional without a new minimum version. macOS CI updates its dependencies through the existing installer without installing recursive optional packages.

  • Failed verification of linking hashes with recorded pre-1.3.0 rlang provenance now reports a distinct pairwiseLLM_link_hash_incompatibility error, including stored/current versions and explicit recovery guidance. New standalone rubric references record hash-engine provenance. Existing hashes and artifacts are never migrated; mismatches without an established version incompatibility remain integrity errors.

  • Frequentist BT bootstrap refits reuse item contrasts and pair designs within each call (#324). Indexed counts replace repeated aggregation, and alpha information matrices use the equivalent weighted graph Laplacian. Original Firth mean bias reduction, all-pairs alpha pseudo-counts, solver controls, uncertainty checks and public interfaces are preserved. Frozen-source oracles and offline benchmarks cover numerical equivalence and worker scaling.

  • Direct trueskill_p50 and trueskill_pollitt candidate scoring validates and aligns the TrueSkill state once per candidate vector (#326). Frozen scalar references check exact probabilities, pair selection, presentation order, diagnostics and RNG behavior. An offline benchmark measures scoring, complete selection and synthetic trajectories; fitting and stopping rules are unchanged.

  • OpenAI Batch disables implicit prompt caching by default for verified GPT-5.6 and later model IDs (#318). prompt_caching = "implicit" restores provider caching; recognized earlier models keep their defaults and cannot explicitly disable caching. Unknown IDs require the explicit opt-in. Validation occurs before IO, and segmented submissions/retries preserve the policy. Parsed results now include prompt_cache_write_tokens alongside cached reads from both endpoints, with NA for missing usage. The advanced batch guide explains model-specific cache rates and Batch pricing. Live APIs, frozen requests and study libraries are unchanged; savings depend on observed usage.

  • Five-arm synthetic qualification (#317) checks matched initialization, outcome-leakage boundaries, Pollitt selection, evidence budgets and continuation across the connected-bootstrap boundary. The warm-start guide now executes all five arms with B defined as mean exposures per item: floor(B * N / 2) total committed pairs, including bootstrap. The tree benchmark separately reports unprofiled wall time, fresh-process peak RSS and cumulative allocations. These are package engineering checks, not estimates of study effects.

  • An internal frozen predictive spanning-tree builder (#315) ranks allowed edges using initial TrueSkill probabilities near one third or two thirds. Degree caps start at two and double only when stalled; exact ties are seeded, recorded orientations are retained, and diagnostics describe relaxation and exposure. Construction uses only permitted endpoints and frozen predictions. Public bootstrap_policy = "predictive_connected" integration (#316) requires a selectable replay reservoir, coherent predictive TrueSkill uncertainty and Pollitt subsequent pairing. Existing bootstrap defaults are unchanged.

  • Adaptive ranking can opt into warm_start_trueskill = "predictive_distribution" to initialize item-specific TrueSkill uncertainty from saved predictive prior SD (#314). Model input requires explicit SD; prebuilt priors retain their stored SD. Versioned mapping and SD provenance survive validated resume without remapping. Legacy initialization, BTL prior rules, and bootstrap behavior are unchanged.

  • A new integrated CJ workflow guide connects pair selection, frequentist and Bayesian estimation, SSR and EAP reliability, empirical score recovery, scale dispersion, schedule-aware bootstrap, and held-out predictive evaluation (#306). Seeded offline examples include experimental lapse-model interior and boundary fits. Topic guides and reference help link to the shared conceptual entry point; statistical implementations, defaults, dependencies, and version are unchanged.

  • fit_bt_model(engine = "lapse") adds an experimental unpenalized frequentist prototype matching the Bayesian btl_e_b likelihood (#305). Bounded optimization estimates centered item strengths, positional bias and lapse probability. Valid epsilon-zero boundary fits retain point estimates and predictions with explicit nonregular uncertainty status; ordinary joint SEs and covariance are unavailable. Interior fits retain validated joint uncertainty. Genuine numerical and identification failures still fail closed. Conventional SSR and simple-BT bootstrap integration remain unavailable. The unchanged frozen qualification recovered all 450 cases: 352 interior and 98 boundary fits, with no numerical or recovery failures. Numerical tolerances and the original evidence are preserved. Existing defaults, dependencies and package version are unchanged.

  • bootstrap_bt_model() adds provider-free, schedule-aware parametric BT bias correction (#304). Fixed schedules retain their pairs; adaptive replicates rerun canonical selection, state updates and scheduled refits from frozen initialization. Results include centered corrections, Monte Carlo summaries, explicit failed-replicate diagnostics and reproducible seeds, with optional parallel execution and artifact retention. See the new bootstrap guide.

  • fit_bt_model(engine = "alpha", alpha = ...) adds Hamilton-style alpha-adjusted Bradley-Terry estimation for adaptive schedules (#303), including explicit penalty settings, centered covariance/SEs, numerical diagnostics and pairwise prediction. SEs are model-based conditional on the realized comparison graph; schedule-aware uncertainty remains separate from these model-based SEs. Numerical failures retain auditable diagnostics without changing estimators. Existing engine defaults and optional dependencies are unchanged.

  • fit_bt_model(engine = "brglm2") adds genuine Firth mean bias reduction for random/nonadaptive schedules (#307), using optional brglm2. Fits include centered covariance/SEs, estimator provenance, SSR, and pairwise predict(). Connected separated data are supported. Valid equal-strength fits retain predictions and uncertainty with explicitly unavailable SSR. Existing engine defaults and fallback behavior are unchanged.

  • Frequentist BT fits now expose independently checked scale-separation reliability (SSR), its variance components, and engine/settings provenance (#302). scale_separation_reliability() provides the calculation directly, and fit_bt_model(sirt_eps = ...) makes sirt’s epsilon adjustment explicit while retaining legacy calls. Disconnected comparisons and invalid/nonfinite reliability inputs now error, including fixed-theta sirt fits with missing SEs. Explicit engines never fall back; negative SSR remains unmodified.

  • Hybrid starvation reports now distinguish unused arithmetic pair capacity from the restrictions encountered in each attempted candidate pool (#298). Use summarize_adaptive(state, include_starvation = TRUE) for the optional report. Fixed-budget guidance explains why repeat rules can stop a run early; selection rules, default summaries and existing log formats are unchanged.

  • Hybrid pairing no longer stops when exploration misses pairs that are still eligible (#296). After trying every fallback, it uses the usual exploitation rule to choose from the last eligible pool. Existing successful selections, pairing limits and saved-session formats are unchanged.

  • E2/E3 Gaussian prediction uses deterministic, error-budgeted quadrature refinement to resolve global-tolerance failures on valid saved fits (#294). The prediction target and requested global tolerances are unchanged; old serialized fits remain usable without refitting or changing their hashes.

  • Sampler draws are now stored as plain numeric matrices in BTL fit contracts, preserving values and item alignment. This fixes standalone linked-reference preparation for real CmdStan fits without relaxing serialization checks or requiring new sampling to canonicalize existing draws (#292).

  • prepare_linked_rubric_reference() saves a completed standalone Bayesian ranking as a reusable reference for linked_anchors rubric calibration. This includes rankings based on pooled historical comparisons. E1, E2, and E3 can use the same saved reference with their respective inputs (#290).

  • New standalone fits record the comparisons used in each refit so reference preparation can verify them. Older standalone fits must be refitted from their original comparisons before becoming reusable linked references; their existing same-set uses and adaptive Phase A reference support are unchanged.

pairwiseLLM 1.6.0

Breaking changes to linking

  • Removed the anchored-joint estimator, its priors, initialization and estimator-specific selection machinery. Saved anchored-joint Phase B sessions cannot be resumed, reported or used for rubric scoring. Restart from compatible original Phase A artifacts and intentionally supplied cross-set comparisons. The Phase B posterior is never silently migrated into a new method.
  • Linking requires an explicit method: fixed_shape_offset (E1), gaussian_posterior_bridge (E2), or joint_offset (E3). There is no default until the downstream study applies its frozen selection rule (#273).
  • Adaptive Phase B D-optimal selection, including legacy aliases, is unavailable pending separate selector validation. Explicit-evidence fitting and continuation remain available. Ordinary within-set adaptive ranking and Phase A preparation retain their existing behavior.

Three linking methods

  • Added prepare_link_input(), fit_link() and predict_link() to separate link estimation from pair selection. Inputs identify hub/spoke samples, compatible Phase A information, explicit ordered cross-set judgments and fixed shared judge settings. Each method uses an explicit offset and enforces single-use evidence.
  • E1 estimates only the offset, using adaptive one-dimensional quadrature and posterior-integrated prediction. It accepts Phase A point scores or compatible artifacts. Its uncertainty is conditional on fixed Phase A scores.
  • E2 uses saved joint posterior draws to construct full-covariance Gaussian bridges. Only cross-set judgments enter its new likelihood. MAP/Laplace fitting propagates approximate shape and offset uncertainty; covariance jitter is bounded and recorded, with no fallback to separate marginal standard errors.
  • E3 jointly fits the original raw within-set and cross-set comparisons, each once, under compatible original model/prior assumptions. MAP/Laplace fitting uses deterministic optimization, analytic derivatives and observed-Hessian checks. An explicitly selected engine = "mcmc" supplies a matching optional Stan audit model, sampler diagnostics and retained serializable draws.
  • All methods preserve the offset prior at zero cross-set evidence and report that the link is not identified. Numerical failures and missing uncertainty are explicit; no engine silently falls back to another estimator.

Sessions, reporting and rubric scoring

  • Added start_link_session(), resume_link_session(), save_link_session() and load_link_session() for exact save/load and append-only continuation. Unchanged resume reuses the saved result. Multi-spoke sessions preserve separate hub/spoke fits, and statuses describe the caller’s workflow without enabling automatic stopping.
  • Common input, result and session schemas use version 1. Estimator-neutral logs and summaries replace obsolete anchored-joint fields with estimator identity, offset estimates, uncertainty scope, fit diagnostics and evidence counts/hashes. theta_link_eap aliases theta_link_mean: posterior averages for E1/E3-MCMC, MAP locations for E2/E3 Laplace. Unknown source provenance remains missing.
  • Linked rubric scoring reuses the exact original hub calibration on valid linked spoke scores. Compatible Phase A artifacts can be reused where they retain the selected method’s required points, joint draws or original raw observations.
  • Probe and stopping adapters consume common results, preserve presentation order, and keep held-out outcomes outside fitting. E1 conditional uncertainty cannot satisfy a full-uncertainty reliability criterion. These adapters do not enable or validate adaptive Phase B selection or stopping.

Validation and documentation

  • Added a shared provider-free release matrix for invariance, repeated judgments, evidence single-use, known-offset simulations and integration. Real optional E3-MCMC audits cover density agreement, MAP parity, symmetry, repeated likelihood information and seeded repeatability. Tests never request provider judgments.
  • Rewrote linking usage guides with offline examples for all three methods and practical save/resume guidance. Expanded the design guide with full statistical models and plain-language explanations; audited the series’ rubric and README changes for the same accessible style.
  • Added an inspectable downstream linking contract and standalone synthetic smoke runner that records the exact package revision for candidate production pinning.

Bug fixes

  • Keep each item in exactly one ranking band when the top band includes every item. Added regression tests for recovery, diagnostics and numerical failures, and removed unreachable execution code behind the Phase B selection gate.
  • Rubric Phase A imports verify evidence hashes against the original stored table before normalization, preserving valid artifacts with different serialized table attributes. Tampered evidence still fails validation.
  • Warm-start glmnet tuning invalidates only verified nonconvergent inner-fold lambda tails. Selection requires complete converged evidence across all folds; full audits retain candidate eligibility and convergence evidence. Reference-path failures and selected-lambda refit failures remain fatal (#271).
  • Increased the internal warm-start glmnet iteration ceiling from 100,000 to 10,000,000 while preserving the 1e-12 convergence threshold, exact lambda paths and strict convergence checks (#268, #271).

pairwiseLLM 1.5.2

Warm-start prediction

  • Added ensemble_warm_start_algorithms() for equal-weight same-task predictions with honest aligned outer-CV validation. Full/reduced portable artifacts work with storage, registries, bundles and BTL/TrueSkill warm starts. Component disagreement remains diagnostic; cross-task ensemble semantics are unchanged (#259).

  • Added optional RBF-SVR warm starts with the fixed epsilon/grid conventions, split-specific gamma, weighted nested-CV tuning and deterministic 1-SE selection. Complete format-3 audits and portable numeric support vectors support full/reduced deployment without e1071; nonlinear coefficient requests fail explicitly (#259).

  • Added optional PLS warm-start models using explicit kernel PLS, legal common component grids, weighted nested-CV tuning and fewer-component 1-SE selection. Full format-3 audits retain candidate OOF evidence; portable full/reduced models predict and supply warm priors without pls (#259).

  • Added explicit writing_features_v2 extraction with 46 source-audited features, schema-specific hashes/decoding, and pinned readability/missingness behavior. The default 20-feature v1 schema and Python environment remain unchanged (#259).

  • Added reusable task/outcome-bound warm-start CV plans and portable format-3 glmnet models, including full/reduced storage, registries, bundles, and warm priors. Legacy formats 1/2 and frozen v1/glmnet numerical behavior remain supported (#259).

Documentation

  • Added a complete offline v2 workflow using fabricated texts and outcomes, shared validation splits, three algorithms, and BTL/TrueSkill initialization. Updated warm-start and adaptive-pairing guidance for non-technical readers, distinguishing cross-task and same-task ensembles and user-chosen prior SD.
  • Corrected model, reduction, and Python audit documentation; synchronized current package version references. R >= 4.4 and optional fitting dependencies are unchanged.

pairwiseLLM 1.5.1

Improvements

  • Added sparse adaptive replay reservoirs with one frozen observation per unordered edge. All four within-set pairing strategies respect unused allowed edges and recorded presentation orientation. Reservoir runs use a common seeded spanning-tree bootstrap, validate identity on resume, and distinguish exhaustion from a temporarily unavailable preferred candidate (#257).

  • MCMC scheduling now respects available CPU allocations through the suggested parallelly package, required when sampling. Automatic concurrency retains the two-slot ceiling and core_fraction headroom. Explicit parallel-chain requests remain supported, with joint chain/thread budget validation and clear errors for excessive requests. Fits and logs record requested and effective resource settings without changing total chains or sampling settings (#255).

Documentation

  • Reworked the README and Getting Started around a runnable offline example, score interpretation, and a sequential path to collecting new comparisons.
  • Added practical reading guidance, prerequisites, output explanations, and recovery advice across the vignettes, while retaining formal statistical design content. Corrected pair-count, example-order, and CmdStan setup guidance.
  • Organized pkgdown navigation and function reference by user task, separated practical guides from statistical design articles, and clarified introductory function help. Existing article URLs and package interfaces are preserved.

Bug fixes

  • Batch polling and result downloads now retry transient HTTP and network failures consistently across OpenAI, Anthropic, and Gemini, honoring Retry-After. Multi-batch resume defers exhausted transient retrievals while surfacing permanent errors immediately. Retrieval retries reuse existing batch IDs without adding scientific attempts; polling timeouts remain checked between status requests. Requires httr2 >= 1.1.0 (#251).

  • OpenAI Batch requests now accept store on both endpoints and max_output_tokens on Responses, including through the batch wrappers. Explicit storage values are preserved and omitted controls remain absent. The new openai_download_batch_errors() helper downloads request failures separately from successful output for ID-based reconciliation (#249).

  • Both OpenAI live endpoints now preserve explicit service_tier requests across model families. "standard" and "default" send "default"; "auto", "flex", and "priority" are forwarded unchanged. Omitted or NULL tiers leave the field absent, preserving OpenAI’s project-configured auto behavior (#247).

  • OpenAI live comparisons now validate and forward explicit store = TRUE and store = FALSE for both Responses and Chat Completions, including row-wise wrappers (#245).

  • Gemini Developer API live and batch comparisons now validate and forward store to control request logging. Omitted or NULL values preserve provider defaults (and Gemini project settings). These controls do not imply a general data-retention guarantee.

pairwiseLLM 1.5.0

Rubric calibration

  • Added fit_rubric_calibration(), S3 predict(), and evaluate_rubric_predictions() for completed Bayesian comparative judgments. All four variants (btl, btl_e, btl_b, btl_e_b) and completed fixed or adaptive results share the downstream interface. Fixed live/batch collection remains provenance only.
  • Added percentile scoring with type-8 quantiles, preserved ties, and requested versus achieved proportions. These are norm-referenced, distribution-matched performance levels; human rubric labels are not fitted.
  • Added default ordinal_linear proportional-odds calibration and explicit ordinal_monotone calibration with a nondecreasing penalized spline. Ordinal predictions retain full probabilities, median-category scores by default, optional modal scores, expected internal level, and extrapolation flags.
  • Both ordinal methods support a labeled subset within one CJ fit (same_set) and a reusable Phase A rubric reference (linked_anchors). Linked target prediction requires completed existing Phase B outputs and reuses the stored reference transformation after identity and fit-contract checks. Phase B estimation, selection, probes, and stopping are unchanged.
  • Added normalized ranked probability score, log loss, cumulative calibration summaries, hard-score metrics, and optional training-model diagnostics. Internal label-only cross-validation retains completed CJ evidence and fits scaling and calibration from training labels alone. No automatic method selection is provided.
  • Each analytic trait requires its own CJ analysis and calibration. Missing intended calibration categories fail explicitly; orientation is never silently reversed. All methods condition on accepted CJ point locations: CJ uncertainty is retained where available but is not propagated. Joint CJ/rubric likelihood estimation is not implemented.

Requirements and documentation

  • Raised the minimum supported R version to 4.4.
  • Linear fitting uses optional ordinal; monotone fitting and prediction use optional mgcv (>= 1.9-4), with optional withr for reproducible fitting. Missing backends give installation guidance without substituting methods.
  • Added a practical rubric-calibration guide and integrated reference help, README, and website navigation, including same-set validation and historical reference reuse through Phase B.

pairwiseLLM 1.4.0

Adaptive ranking

  • Added predictive warm-start modes cold, btl_only, trueskill_only, and both. Omitted mode preserves historical behavior: cold without predictive input and btl_only with it. Request both explicitly to initialize both models.
  • TrueSkill-warm modes initialize item locations as 25 + (25/3) * prior_mean, with fixed scale 1 and unchanged sigma. Every mode retains the same seeded shuffled connected bootstrap of N - 1 valid comparisons.
  • Added post-bootstrap random, trueskill_p50, and trueskill_pollitt strategies through adaptive_config$pairing_strategy; hybrid remains the default. Direct strategies use a minimum-degree focal item and currently support only within-set runs. Pollitt-inspired targets use TrueSkill probabilities.
  • Within-set and Phase-A hybrid rolling anchors and the long-link probability gate now use TrueSkill throughout. Bayesian BTL still supplies item estimates, uncertainty, reliability, diagnostics, stopping, and the global_identified signal that can affect later hybrid routing. Phase B linking is unchanged.
  • Sessions persist effective mode, strategy, predictive provenance, and current TrueSkill state. Legacy sessions migrate to cold or historical btl_only and hybrid, preserving saved bootstrap and round progress. Phase-A artifact identity distinguishes generation settings while preserving explicit imports.
  • Added validate_adaptive_replay() and make_adaptive_judge_replay() for offline replay of exact directed judgments, with strict reuse protection. Replay users can set adaptive_config$dup_max_obs_relaxed = 2L; the general hybrid default retains its conditional third observation.

Fixes and documentation

  • Within-set continuation retains saved BTL settings when no override is supplied; round recovery uses the exact saved round field.
  • Updated adaptive help, guides, design documentation, and replay examples to distinguish predictive initialization, connected bootstrap, and pairing strategy.
  • Strengthened adaptive-session validation and resume handling, including preservation of named Phase-A evidence hashes, legacy pair-resume behavior, and improved detection of Ollama results.
  • Made stratified cost estimation robust to singleton strata and expanded boundary validation for adaptive state, warm-start models, provider responses, batch payloads, optional dependencies, and sampler interfaces.
  • Simplified linking internals around the anchored-joint refit path by removing obsolete transform-refit Stan support and unreachable legacy Phase-B branches, while preserving legacy state normalization, audit fields, persistence compatibility, and current linking behavior.
  • Added regression coverage for fresh-process resume, refit diagnostics, selection and information updates, state reporting, schema validation, and runtime boundary behavior.

pairwiseLLM 1.3.2

New Features

Adaptive pairing warm start

  • Added warm start model development workflow: task-specific
    elastic-net training with nested cross-validation and out-of-fold calibration, and portable model storage.
  • Added user model registration, equal-weight task-model ensembles, and centered predictive priors for standalone and adaptive Bayesian BTL estimation. Adaptive sessions retain numeric priors for resume without the original model or Python.
  • Added the adaptive warm start help family and a practical workflow guide. Python is optional for extraction; precomputed prediction needs neither Python nor glmnet. Real bundled predictors remain deferred; no default model ships.
  • Added warm_start_coefficients() for inspecting calibrated standardized elastic-net coefficients from individual models and ensemble components, preserving the distinction between preprocessing removal and shrinkage to zero.

pairwiseLLM 1.3.1

CRAN release: 2026-09-07

New Features

Adaptive linking

  • Added cross-set adaptive linking through adaptive_rank(), with one-spoke and multi-spoke hub-and-spoke run modes for placing separately ranked sets on a shared scale.
  • Phase A can run canonical within-set ranking, import saved Phase A artifacts, or combine imported and newly run sets. Phase B uses anchored-joint estimation with hard-locked hub parameters, globally shared judge parameters, and concurrent spoke processing.
  • Added D-optimal cross-set candidate selection, spoke-aware comparison budgets, and fixed per-refit held-out probes with explicit coverage, calibration, and stopping gates.
  • Added persistent Phase A manifests and linking sessions, resume support, link_stage_log, and explicit blocker and invalid-judgment reporting for auditable long-running workflows.

Model support & live API improvements

  • Added a separate Vertex AI Gemini live backend via backend = "vertex" with VERTEX_API_KEY.
    • Vertex remains live-only in this series; generic batch wrappers reject backend = "vertex" explicitly rather than routing it through Gemini batch helpers.
  • Clarified provider-specific service_tier behavior for Gemini Developer API and Vertex AI Gemini API in the package docs, README, and vignettes.
  • Added request-shape handling for the OpenAI GPT-5.x and GPT-5.6 model families.

Bayesian BTL usability

  • Added exported build_btl_results_data() to convert conventional ID1/ID2/better_id outcomes into the canonical input required by fit_bayes_btl_mcmc().
  • Standalone Bayesian BTL fits now report reliability and sampler diagnostics through the same result and log surfaces used by adaptive workflows.

Improvements

  • Reduced adaptive-linking refit, candidate-generation, logging, and checkpoint overhead for large runs, including bounded candidate construction and analytic anchored-joint derivatives.
  • Hardened adaptive-linking resume behavior, Phase A stopping scope, spoke routing, probe scheduling, and Phase B pair accounting.

Documentation

  • Updated the backend support matrix to distinguish Gemini Developer API from Vertex AI Gemini API and to document the live-only Vertex scope.
  • Replaced the undated supported-model list with a dated, machine-readable compatibility registry that separates backend implementation, live/batch testing, and current provider availability.
  • Corrected stale provider identifiers, API-key and CmdStan setup guidance, OpenAI Flex wording, cost-estimator summaries, and broken public examples.

Bug fixes

  • Fixed Anthropic extended-thinking live requests when users explicitly set the required temperature = 1.
  • OpenAI Responses live requests now forward and enforce an explicitly supplied max_output_tokens limit.
  • Changed omitted temperature and top_p controls to use model/provider defaults instead of injecting deterministic sampling values. Provider-required reasoning constraints are still enforced.
  • Provider smoke runs now exit unsuccessfully when a selected API key is absent, unless missing-key skips were explicitly allowed.

pairwiseLLM 1.3.0

New Features

Adaptive pairing & ranking framework

  • Introduced an adaptive pairing / adaptive ranking framework designed to reduce unnecessary comparisons through uncertainty-aware pair selection and Bayesian inference.

  • Added adaptive_rank(), the primary user-facing wrapper that runs the complete adaptive workflow end-to-end, including warm start, adaptive pairing rounds, Bayesian BTL refits, diagnostics, and stopping.

  • Advanced control is available via:

  • Adaptive pairing is organized into rounds that balance global scale identification and local refinement using a mixture of anchor, long-range, mid-range, and local comparisons.

  • The adaptive controller tracks a global identifiability state based on Bayesian diagnostics and agreement between online (TrueSkill) and global (BTL) rankings. Once the global scale is identified:

    • long-range comparisons are automatically tapered,
    • comparison budget is reallocated toward local and boundary-refining pairs,
    • exploration rates are reduced to focus on decision-relevant uncertainty.

  • Long-range comparisons are additionally posterior-gated in later stages, preventing wasted comparisons on pairs that are already decisively ordered.

  • Late-stage local pairing prioritizes near-tie pairs, with limited, auditable overrides to degree caps when especially informative comparisons are blocked.

  • Adaptive runs produce fully auditable step-, round-, and refit-level logs, recording candidate generation, fallbacks, gating decisions, quota reallocations, and stopping criteria.

  • All adaptive workflows use standardized configuration, state, and logging contracts to ensure reproducibility and future extensibility.

Bayesian Bradley–Terry–Luce (BTL) modeling

  • Added a fully Bayesian Bradley–Terry–Luce (BTL) model implemented via CmdStan, providing posterior uncertainty estimates for item skill parameters.
  • New entrypoint fit_bayes_btl_mcmc() enables direct posterior inference from pairwise comparison data, independent of or integrated with adaptive workflows.
  • Supports multiple model variants (including error and positional bias extensions) and optional refitting on increasing subsets of comparisons.
  • Bayesian BTL outputs integrate seamlessly with adaptive ranking utilities (summarize_items(), summarize_refits()), serving as the statistical backbone for adaptive pairing decisions.

Model support & live API improvements

  • Added support for the Gemini Flash model gemini-3-flash-preview for live pairwise comparisons.
  • Added support for OpenAI service tiers / priority routing via service_tier for applicable live models.
    • Enables tiers such as "flex" and "priority" when supported by the selected model.
    • Integrated into the live submission path without requiring changes to calling code.

Documentation

  • Expanded and clarified documentation for adaptive ranking, Bayesian BTL, and live model configuration.
  • Updated examples to reflect new adaptive and Bayesian APIs.

pairwiseLLM 1.2.0

New Features

  • Parallel Processing:
    • submit_llm_pairs() and backend-specific live functions (OpenAI, Anthropic, Gemini, Together, Ollama) now support parallel execution via parallel = TRUE and workers = n (requires the future package).
  • Incremental Saving & Resume:
    • Added save_path argument to live submission functions. Results are saved to CSV incrementally, allowing interrupted jobs to resume automatically by skipping previously processed pairs.
  • Robust Error Handling:
    • Failed API calls no longer stop the entire process. Failures are captured and returned separately, allowing for easier inspection and re-submission.
  • Added estimate_llm_pairs_cost() to estimate costs in live and batch mode.
  • Introduced llm_submit_pairs_multi_batch() and llm_resume_multi_batches() to split large comparison sets across multiple batches and resume polling later. These helpers support writing per‑batch and combined results, along with an optional jobs registry.

Bug fixes

  • The prompt format for anthropic batch comparisons now match the anthropic live format.
  • Reverse consistency functions can now handle duplicate pairs.

Breaking Changes

  • submit_llm_pairs() and its backend-specific counterparts now return a list containing two elements: $results (a tibble of successful comparisons) and $failed_pairs (a tibble of inputs that failed). Previous versions returned a single tibble.

pairwiseLLM 1.1.0

CRAN release: 2025-12-22

Models

  • Added GPT-5.2
  • Ensured models can be called with date format, e.g. gpt-5.2-2025-12-11
  • Default temperature setting is set to 0 for non-reasoning models, provider default for reasoning models (typically 1)

Tests

  • Tests added to improve coverage

Documentation

  • Changed pkgdown site layout
  • Added codemeta.json
  • Added repo logo
  • Updated function examples
  • Add references to Description

Miscellaneous

  • No longer set global variables, now done in individual functions
  • Added verbose option in fit_bt_model() and summarize_bt_fit()
  • Moved null coalescing helper to separate R file
  • Changed validation of API keys in multiple functions

pairwiseLLM 1.0.0

  • Initial release.
  • Unified live and batch LLM comparison framework (OpenAI / Anthropic / Gemini).
  • Live support for Together.ai and local Ollama backends.
  • Tools for Bradley–Terry and Elo models, positional bias checks