Skip to contents

Build an LLM judge function for adaptive ranking

Source: R/adaptive_rank.R
make_adaptive_judge_llm.Rd

Creates a judge function compatible with adaptive_rank_run_live() by wrapping llm_compare_pair() and converting provider responses into adaptive binary outcomes (Y in {0,1}).

Usage

make_adaptive_judge_llm(
  backend = c("openai", "anthropic", "gemini", "vertex", "together", "ollama"),
  model,
  trait = "overall_quality",
  trait_name = NULL,
  trait_description = NULL,
  prompt_template = set_prompt_template(),
  endpoint = "chat.completions",
  api_key = NULL,
  include_raw = FALSE,
  text_col = "text",
  judge_args = list()
)

Arguments

backend

Backend passed to llm_compare_pair(). Choices are "openai", "anthropic", "gemini", "vertex", "together", and "ollama". Default is "openai".

model

Model identifier passed to llm_compare_pair(). Required.

trait

Built-in trait key used when no custom trait is supplied. Ignored when both trait_name and trait_description are supplied. Default is "overall_quality".

trait_name

Optional custom trait display name.

trait_description

Optional custom trait definition.

prompt_template

Prompt template string. Defaults to set_prompt_template().

endpoint

Endpoint family passed to llm_compare_pair(). Only used when backend = "openai"; choices are "chat.completions" and "responses". Default is "chat.completions". Ignored for other backends.

api_key

Optional API key passed to llm_compare_pair().

include_raw

Logical; forwarded to llm_compare_pair(). Default is FALSE.

text_col

Name of the text column expected in adaptive item rows. Default is "text".

judge_args

Named list of additional fixed arguments forwarded to llm_compare_pair(). Use this for provider-specific controls such as reasoning, service_tier, temperature, top_p, logprobs, host, or include_thoughts. Default is list().

Value

A function judge(A, B, state, ...) returning a list with required fields is_valid, Y, and invalid_reason, plus optional canonical audit fields such as judge_backend, judge_model, judge_endpoint, llm_status_code, llm_error_message, llm_custom_id, prompt_tokens, completion_tokens, total_tokens, and raw_response_json.

Details

The returned function has signature judge(A, B, state, ...) and enforces the adaptive transactional contract: it returns is_valid = TRUE with Y in {0,1} when the model response identifies one of the two presented items, and returns is_valid = FALSE otherwise. In addition to the required contract fields, the returned judge preserves canonical audit metadata from the first llm_compare_pair() row, including backend/model provenance, status/error fields, token counts, and a serialized raw_response_json payload when available.

Model configuration is split into:

Collectively this supports all llm_compare_pair() options, including backend-specific parameters such as OpenAI reasoning and service_tier.

See also

adaptive_rank(), adaptive_rank_run_live(), llm_compare_pair()

Other adaptive ranking: adaptive_rank(), adaptive_rank_resume(), adaptive_rank_run_live(), adaptive_rank_start(), make_adaptive_judge_replay(), make_adaptive_replay_reservoir(), summarize_adaptive(), validate_adaptive_replay()

Examples

judge <- make_adaptive_judge_llm(
  backend = "openai",
  model = "gpt-5.1",
  endpoint = "responses",
  judge_args = list(
    reasoning = "low",
    service_tier = "flex",
    include_thoughts = FALSE
  )
)