Backend-agnostic live comparison for a single pair of samples

llm_compare_pair() is a thin wrapper around backend-specific comparison functions. It currently supports the "openai", "anthropic", "gemini", "vertex", "together", "ollama", and "typesafe" backends and forwards the call to the appropriate live comparison helper:

• "openai" → openai_compare_pair_live()

• "anthropic" → anthropic_compare_pair_live()

• "gemini" → gemini_compare_pair_live()

• "vertex" → vertex_compare_pair_live()

• "together" → together_compare_pair_live()

• "ollama" → ollama_compare_pair_live()

• "typesafe" → typesafe_compare_pair_live()

Usage

llm_compare_pair(
  ID1,
  text1,
  ID2,
  text2,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  backend = c("openai", "anthropic", "gemini", "vertex", "together", "ollama",
    "typesafe"),
  endpoint = c("chat.completions", "responses", "decisions"),
  api_key = NULL,
  include_raw = FALSE,
  ...
)

Arguments

ID1

Character ID for the first sample.

text1

Character string containing the first sample's text.

ID2

Character ID for the second sample.

text2

Character string containing the second sample's text.

model

Model identifier for the chosen backend. For "openai" this should be an OpenAI model name (for example "gpt-4.1", "gpt-5.1"). For "anthropic", use Anthropic model names such as "claude-sonnet-4-5". For "gemini" and "vertex", use the corresponding Gemini model names (for example "gemini-3.5-flash-lite" or "gemini-2.5-flash"). For "together", use Together.ai model identifiers such as "deepseek-ai/DeepSeek-V4-Flash-0731". For "ollama", use a local model name known to the Ollama server (for example "mistral-small3.2:24b", "qwen3:32b", "gemma3:27b"). For "typesafe", omission defaults to "jev-1.13.0"; explicit versioned Jev identifiers and documented aliases are accepted without an availability guarantee or automatic fallback. Other backends still require model.

trait_name

Short label for the trait (for example "Overall Quality").

trait_description

Full-text definition of the trait.

prompt_template

Prompt template string, typically from set_prompt_template(). For Decisions, omit or use NULL; custom decision instructions belong in decision_template in .... The same rule applies to TypeSafe; its omitted generative default is never evaluated.

backend

Character scalar indicating which LLM provider to use. One of "openai", "anthropic", "gemini", "vertex", "together", "ollama", or "typesafe".

endpoint

Character scalar specifying which endpoint family to use for backends that support multiple live APIs. For the "openai" backend this must be one of "chat.completions", "responses", or "decisions", matching openai_compare_pair_live(). For "anthropic", "gemini", "vertex", "together", and "ollama", this argument is currently ignored. For "typesafe", omit it or use "systemone"; other values fail.

api_key

Optional API key for the selected backend. If NULL, the backend-specific helper will use its own default environment variable (for example OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, VERTEX_API_KEY, TOGETHER_API_KEY, TYPESAFE_API_KEY). For "ollama", this argument is ignored (no API key is required for local inference).

include_raw

Logical; if TRUE, the returned tibble includes a raw_response list-column with the parsed JSON body (or NULL on parse failure). Support for this may vary across backends.

...

Additional backend-specific parameters. For "openai" these are passed on to openai_compare_pair_live() and typically include arguments such as temperature, top_p, logprobs, reasoning, and include_thoughts. For "anthropic", "gemini", and "vertex" they are forwarded to the corresponding live helper and may include parameters such as reasoning, include_thoughts, max_output_tokens, service_tier, thinking_level, or provider-specific options. For "ollama", arguments are forwarded to ollama_compare_pair_live() and may include host, think, num_ctx, and other Ollama-specific controls.

service_tier is provider-specific: OpenAI, Gemini Developer API, and Vertex each validate and encode it using their own request surfaces. For Gemini Developer API and Vertex, supported public values are "standard" (default / omitted), "flex", and "priority". For both OpenAI generative live endpoints, "standard" and "default" send "default"; "auto", "flex", and "priority" are sent unchanged. Omission or NULL allows OpenAI's project-configured "auto" behavior. Model/tier availability is validated by OpenAI.

For OpenAI and Gemini Developer API, store accepts NULL, TRUE, or FALSE. It controls OpenAI response storage or Gemini request logging; omission or NULL preserves provider/project defaults. See the selected provider helper for details; this is not a cross-provider retention policy. Decisions accepts only pair_uid, decision_template, and prompt_cache_options in ...; generative, storage, and tier controls fail before HTTP. Use model = "gpt-6-luna". Its omitted prompt is never replaced with the generative default. Use submit_llm_pairs() for resumable bulk Decisions execution. TypeSafe accepts only pair_uid and decision_template in ..., including rejection of unsupported settings explicitly supplied as NULL. Both decision backends retain independent confidence and full request provenance; probabilities never change binary BT/adaptive winner semantics. See decision_bulk for bulk execution; managed Batch remains unsupported.

Value

A tibble with one row and the same columns as the underlying backend-specific live helper (for example openai_compare_pair_live() for "openai"). All backends are intended to return a compatible structure including thoughts, content, and token counts.

Details

All backends are expected to return a tibble with a compatible structure, including:

• custom_id, ID1, ID2

• model, object_type, status_code, error_message

• thoughts (reasoning / thinking text when available)

• content (visible assistant output)

• better_sample, better_id

• prompt_tokens, completion_tokens, total_tokens

For the "openai" backend, the endpoint argument controls whether the Chat Completions API ("chat.completions"), Responses API ("responses"), or opt-in Decisions API ("decisions") is used. Decisions uses the approved decision instructions and retains the complete binary probability distribution; see openai_compare_pair_live() for its supported controls, return fields, validation, and errors. TypeSafe uses its Choice endpoint and the same decision instructions and probability-preserving contracts; see typesafe_compare_pair_live(). For the "anthropic", "gemini", "vertex", "together", and "ollama" backends, endpoint is currently ignored and the default live API for that provider is used.

Decision failure retention

Decision failures may retain raw provider responses in memory even when include_raw = FALSE; saving the complete return value may persist sensitive sample text. See decision_bulk for default checkpoint/export retention.

See also

• openai_compare_pair_live(), anthropic_compare_pair_live(), gemini_compare_pair_live(), vertex_compare_pair_live(), together_compare_pair_live(), ollama_compare_pair_live(), and typesafe_compare_pair_live() for backend-specific implementations.

• submit_llm_pairs() for row-wise comparisons over a tibble of pairs.

• build_bt_data() and fit_bt_model() for Bradley–Terry modelling of comparison results.

Other live backends: anthropic_compare_pair_live(), check_llm_api_keys(), gemini_compare_pair_live(), ollama_compare_pair_live(), openai_compare_pair_live(), submit_anthropic_pairs_live(), submit_gemini_pairs_live(), submit_llm_pairs(), submit_ollama_pairs_live(), submit_openai_pairs_live(), submit_together_pairs_live(), submit_typesafe_pairs_live(), submit_vertex_pairs_live(), together_compare_pair_live(), typesafe_compare_pair_live(), vertex_compare_pair_live()

Examples

if (FALSE) { # \dontrun{
# Requires an API key for the chosen cloud backend. For OpenAI, set
# OPENAI_API_KEY in your environment. Running these examples will incur
# API usage costs.
#
# For local Ollama use, an Ollama server must be running and the models
# must be pulled in advance. No API key is required for the `"ollama"`
# backend.

data("example_writing_samples", package = "pairwiseLLM")
samples <- example_writing_samples[1:2, ]

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# Single live comparison using the OpenAI backend and chat.completions
res_live <- llm_compare_pair(
  ID1               = samples$ID[1],
  text1             = samples$text[1],
  ID2               = samples$ID[2],
  text2             = samples$text[2],
  model             = "gpt-4.1",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  backend           = "openai",
  endpoint          = "chat.completions",
  temperature       = 0
)

res_live$better_id

# Using the OpenAI responses endpoint with gpt-5.1 and reasoning = "low"
res_live_gpt5 <- llm_compare_pair(
  ID1               = samples$ID[1],
  text1             = samples$text[1],
  ID2               = samples$ID[2],
  text2             = samples$text[2],
  model             = "gpt-5.1",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  backend           = "openai",
  endpoint          = "responses",
  reasoning         = "low",
  include_thoughts  = TRUE,
  temperature       = NULL,
  top_p             = NULL,
  logprobs          = NULL,
  include_raw       = TRUE
)

str(res_live_gpt5$raw_response[[1]], max.level = 2)

# Example: single live comparison using a local Ollama backend
res_ollama <- llm_compare_pair(
  ID1 = samples$ID[1],
  text1 = samples$text[1],
  ID2 = samples$ID[2],
  text2 = samples$text[2],
  model = "mistral-small3.2:24b",
  trait_name = td$name,
  trait_description = td$description,
  prompt_template = tmpl,
  backend = "ollama",
  host = getOption(
    "pairwiseLLM.ollama_host",
    "http://127.0.0.1:11434"
  ),
  think = FALSE
)

res_ollama$better_id
} # }