Package index • pairwiseLLM

Start here

New users: follow the Getting Started guide for an offline example and a first collection workflow.

pairwiseLLM-package pairwiseLLM: pairwiseLLM: Pairwise comparisons and adaptive ranking with LLM judges

Prepare writing samples and pairs

Create pair tables locally before making provider requests.

read_samples_df(): Read writing samples from a data frame
read_samples_dir(): Read writing samples from a directory of .txt files
make_pairs(): Create all unordered pairs of writing samples
sample_pairs(): Randomly sample pairs of writing samples
sample_reverse_pairs(): Sample reversed versions of a subset of pairs
randomize_pair_order(): Randomly assign samples to positions SAMPLE_1 and SAMPLE_2
alternate_pair_order(): Deterministically alternate sample order in pairs

Choose a trait and prepare prompts

trait_description(): Get a trait name and description for prompts
set_prompt_template(): Read or validate a prompt template for pairwise comparisons
build_prompt(): Build a concrete LLM prompt from a template
register_prompt_template(): Register a named prompt template
get_prompt_template(): Retrieve a named prompt template
list_prompt_templates(): List available prompt templates
remove_prompt_template(): Remove a registered prompt template

Collect comparisons

Use the generic helpers first. Cost estimation runs a paid pilot; see provider controls and recovery.

check_llm_api_keys(): Check configured API keys for LLM backends
submit_llm_pairs(): Backend-agnostic live comparisons for a tibble of pairs
llm_compare_pair(): Backend-agnostic live comparison for a single pair of samples
llm_submit_pairs_batch(): Submit pairs to an LLM backend via batch API
llm_download_batch_results(): Extract results from a pairwiseLLM batch object
llm_submit_pairs_multi_batch(): Multi‑batch submission and polling wrappers
llm_resume_multi_batches(): Resume polling and download results for multiple batch jobs
estimate_llm_pairs_cost(): Estimate LLM token usage and cost for a set of pairwise comparisons
print(<pairwiseLLM_cost_estimate>): Print a pairwiseLLM cost estimate

Turn comparisons into rankings

BT and Elo use recorded winners; Bayesian BTL additionally requires CmdStan.

build_bt_data(): Build Bradley-Terry comparison data from pairwise results
fit_bt_model(): Fit a Bradley–Terry model with sirt and fallback to BradleyTerry2
summarize_bt_fit(): Summarize a Bradley–Terry model fit
build_elo_data(): Build EloChoice comparison data from pairwise results
fit_elo_model(): Fit an EloChoice model to pairwise comparison data
build_btl_results_data(): Build canonical results_tbl data for Bayesian BTL MCMC
fit_bayes_btl_mcmc(): Full Bayesian BTL inference via CmdStanR (adaptive-compatible)

Choose comparisons adaptively

Start with adaptive_rank(); see the practical guide.

adaptive_rank(): Run adaptive ranking end-to-end from data and model settings
make_adaptive_judge_llm(): Build an LLM judge function for adaptive ranking

Inspect rankings and diagnostics

print(<adaptive_state>): Print an adaptive state summary.
adaptive_get_logs(): Retrieve canonical adaptive logs.
adaptive_step_log(): Adaptive step log accessor.
adaptive_round_log(): Adaptive round log accessor.
adaptive_item_log(): Adaptive item log accessor.
adaptive_results_history(): Adaptive results history in build_bt_data() format.
summarize_adaptive(): Summarize an adaptive state.
summarize_refits(): Summarize adaptive refits
summarize_items(): Summarize adaptive items

Save and resume adaptive sessions

save_adaptive_session(): Save an adaptive session to disk.
validate_session_dir(): Validate an adaptive session directory.
load_adaptive_session(): Load an adaptive session from disk.

Convert Bayesian rankings to rubric levels

Requires completed Bayesian CJ results. See the rubric guide for human labels and interpretation.

fit_rubric_calibration(): Fit a rubric calibration to completed comparative judgments
predict(<pairwiseLLM_rubric_calibration>): Predict rubric scores from a calibration
evaluate_rubric_predictions(): Evaluate rubric predictions on observed ordered labels

Check consistency and position bias

compute_reverse_consistency(): Compute consistency between forward and reverse pair comparisons
check_positional_bias(): Check positional preference and bootstrap reversal agreement

Advanced: predictive warm start

ensemble_warm_start_algorithms(): Average algorithms trained on the same task
ensemble_warm_start_models(): Combine independently trained warm-start models
extract_warm_start_features(): Extract frozen writing features for warm-start prediction
fit_warm_start_model(): Train a task-specific warm-start model with nested validation
make_warm_start_cv_plan(): Construct reusable warm-start cross-validation partitions
make_warm_start_prior(): Convert warm-start predictions to Bayesian BTL priors
summary(<pairwiseLLM_warm_model>) print(<pairwiseLLM_warm_model>): Portable task-specific warm-start models
predict(<pairwiseLLM_warm_algorithm_ensemble>): Predict with a same-task algorithm ensemble
predict(<pairwiseLLM_warm_ensemble>): Predict with an equal-weight warm-start ensemble
predict(<pairwiseLLM_warm_model>): Predict relative quality from a portable warm-start model
prepare_warm_start_model(): Prepare metadata or a summary-only warm-start artifact
register_warm_start_model() remove_warm_start_model() list_warm_start_models(): Register, inspect, or remove user warm-start models
save_warm_start_model() load_warm_start_model(): Save or load a portable warm-start model
summary(<pairwiseLLM_warm_algorithm_ensemble>) print(<pairwiseLLM_warm_algorithm_ensemble>): Inspect a same-task algorithm ensemble
summary(<pairwiseLLM_warm_ensemble>) print(<pairwiseLLM_warm_ensemble>): Inspect a warm-start ensemble
summary(<pairwiseLLM_warm_predictions>) print(<pairwiseLLM_warm_predictions>): Inspect ensemble predictions
warm_start_coefficients(): Inspect calibrated standardized warm-start coefficients
warm_start_feature_schema(): Inspect the frozen warm-start writing feature schema
warm_start_python_status(): Check the optional warm-start feature environment

Advanced: adaptive lifecycle and replay

make_adaptive_replay_reservoir(): Create a sparse frozen adaptive replay reservoir
make_adaptive_judge_replay(): Create an offline judge from frozen directed outcomes
validate_adaptive_replay(): Validate frozen directed judgments for an adaptive panel
adaptive_rank_start(): Adaptive ranking
adaptive_rank_run_live(): Adaptive ranking live runner
adaptive_rank_resume(): Adaptive ranking resume

Provider-specific live helpers

openai_compare_pair_live(): Live OpenAI comparison for a single pair of samples
submit_openai_pairs_live(): Live OpenAI comparisons for a tibble of pairs
anthropic_compare_pair_live(): Live Anthropic (Claude) comparison for a single pair of samples
submit_anthropic_pairs_live(): Live Anthropic (Claude) comparisons for a tibble of pairs
gemini_compare_pair_live(): Live Google Gemini comparison for a single pair of samples
submit_gemini_pairs_live(): Live Google Gemini comparisons for a tibble of pairs
vertex_compare_pair_live(): Live Vertex AI Gemini comparison for a single pair of samples
submit_vertex_pairs_live(): Live Vertex AI Gemini comparisons for a tibble of pairs
together_compare_pair_live(): Live Together.ai comparison for a single pair of samples
submit_together_pairs_live(): Live Together.ai comparisons for a tibble of pairs
ollama_compare_pair_live(): Live Ollama comparison for a single pair of samples
submit_ollama_pairs_live(): Live Ollama comparisons for a tibble of pairs

Provider-specific batch helpers

build_openai_batch_requests(): Build OpenAI batch JSONL lines for paired comparisons
write_openai_batch_file(): Write an OpenAI batch table to a JSONL file
openai_upload_batch_file(): Upload a JSONL batch file to OpenAI
openai_create_batch(): Create an OpenAI batch from an uploaded file
openai_get_batch(): Retrieve an OpenAI batch
openai_poll_batch_until_complete(): Poll an OpenAI batch until it completes or fails
openai_download_batch_output(): Download the output file for a completed batch
openai_download_batch_errors(): Download the error file for an OpenAI batch
run_openai_batch_pipeline(): Run a full OpenAI batch pipeline for pairwise comparisons
build_anthropic_batch_requests(): Build Anthropic Message Batch requests from a tibble of pairs
anthropic_create_batch(): Create an Anthropic Message Batch
anthropic_get_batch(): Retrieve an Anthropic Message Batch by ID
anthropic_poll_batch_until_complete(): Poll an Anthropic Message Batch until completion
anthropic_download_batch_results(): Download Anthropic Message Batch results (.jsonl)
run_anthropic_batch_pipeline(): Run an Anthropic batch pipeline for pairwise comparisons
build_gemini_batch_requests(): Build Gemini batch requests from a tibble of pairs
gemini_create_batch(): Create a Gemini Batch job from request objects
gemini_get_batch(): Retrieve a Gemini Batch job by name
gemini_poll_batch_until_complete(): Poll a Gemini Batch job until completion
gemini_download_batch_results(): Download Gemini Batch results to a JSONL file
run_gemini_batch_pipeline(): Run a Gemini batch pipeline for pairwise comparisons

Parse downloaded batch output

parse_openai_batch_output(): Parse an OpenAI Batch output JSONL file
parse_anthropic_batch_output(): Parse Anthropic Message Batch output into a tibble
parse_gemini_batch_output(): Parse Gemini batch JSONL output into a tibble of pairwise results

Manage local Ollama resources

ensure_only_ollama_model_loaded(): Ensure only one Ollama model is loaded in memory

Example data

Synthetic texts, bundled comparison outcomes, Bayesian input rows, and a parser fixture for offline exploration.

example_writing_samples: Example dataset of writing samples
example_writing_samples1000: Synthetic Writing Samples with Controlled Quality Levels (N = 1000)
example_writing_pairs: Example dataset of paired comparisons for writing samples
example_writing_results: Example canonical results table for writing comparisons
example_openai_batch_output: Example OpenAI Batch output (JSONL lines)