Run a full OpenAI batch pipeline for pairwise comparisons

This helper wires together the existing pieces:

build_openai_batch_requests()
write_openai_batch_file()
openai_upload_batch_file()
openai_create_batch()
optionally openai_poll_batch_until_complete()
optionally openai_download_batch_output()
optionally parse_openai_batch_output()

Usage

run_openai_batch_pipeline(
  pairs,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  include_thoughts = FALSE,
  include_raw = FALSE,
  endpoint = NULL,
  batch_input_path = tempfile("openai_batch_input_", fileext = ".jsonl"),
  batch_output_path = tempfile("openai_batch_output_", fileext = ".jsonl"),
  poll = TRUE,
  interval_seconds = 5,
  timeout_seconds = 600,
  max_attempts = Inf,
  metadata = NULL,
  api_key = NULL,
  ...
)

Arguments

pairs: Tibble of pairs with at least ID1, text1, ID2, text2. Typically produced by make_pairs(), sample_pairs(), and randomize_pair_order().
model: OpenAI model name (e.g. "gpt-4.1", "gpt-5.6-sol").
trait_name: Trait name to pass to build_openai_batch_requests().
trait_description: Trait description to pass to build_openai_batch_requests().
prompt_template: Prompt template string, typically from set_prompt_template().
include_thoughts: Logical; if TRUE and using endpoint = "responses", requests reasoning-style summaries to populate the thoughts column in the parsed output. When endpoint is not supplied, include_thoughts = TRUE causes the responses endpoint to be selected automatically.
include_raw: Logical; if TRUE, attaches the raw model response as a list-column raw_response in the parsed results.
endpoint: One of "chat.completions" or "responses". If NULL (or omitted), it is chosen automatically as described above.
batch_input_path: Path to write the batch input .jsonl file. Defaults to a temporary file.
batch_output_path: Path to write the batch output .jsonl file if poll = TRUE. Defaults to a temporary file.
poll: Logical; if TRUE, the function will poll the batch until it reaches a terminal status using openai_poll_batch_until_complete() and then download and parse the output. If FALSE, it stops after creating the batch and returns without polling or parsing.
interval_seconds: Polling interval in seconds (used when poll = TRUE).
timeout_seconds: Maximum total time in seconds for polling before giving up (used when poll = TRUE).
max_attempts: Maximum number of polling attempts (primarily useful for testing).
metadata: Optional named list of metadata key–value pairs to pass to openai_create_batch().
api_key: Optional OpenAI API key. Defaults to Sys.getenv("OPENAI_API_KEY").
...: Additional arguments passed through to build_openai_batch_requests(), e.g. temperature, top_p, logprobs, reasoning, store, prompt_caching, and max_output_tokens. Explicitly select endpoint = "responses" when supplying max_output_tokens.

Value

A list with elements:

batch_input_path – path to the input .jsonl file.
batch_output_path – path to the output .jsonl file (or NULL if poll = FALSE).
file – File object returned by openai_upload_batch_file().
batch – Batch object; if poll = TRUE, this is the final batch after polling, otherwise the initial batch returned by openai_create_batch().
results – Parsed tibble from parse_openai_batch_output() if poll = TRUE, otherwise NULL.

Details

It is a convenience wrapper around these smaller functions and is intended for end-to-end batch runs on a set of pairwise comparisons. For more control (or testing), you can call the components directly.

When endpoint is not specified, it is chosen automatically:

if include_thoughts = TRUE or GPT-5 reasoning is requested, the "responses" endpoint is used and a default reasoning effort of "low" is applied for GPT-5 series models unless overridden via reasoning.
otherwise, "chat.completions" is used.

Omitted or NULL store preserves provider defaults. Responses are stored for later retrieval by default; store = FALSE disables that response storage, not Batch input/output/error file retention or all data retention. See build_openai_batch_requests() for endpoint-specific details. Use openai_download_batch_errors() separately to retrieve request failures. For an errors-only batch, use poll = FALSE, then openai_get_batch() or openai_poll_batch_until_complete() and the error downloader; this pipeline's poll = TRUE path still requires an output file.

Batch prompt caching

The default disables prompt caching for these verified model IDs: gpt-5.6-luna, gpt-5.6-terra, gpt-5.6-sol, gpt-6-luna, gpt-6-sol, gpt-6-astra, and gpt-6.1-sol. Each request body gets prompt_cache_options = list(mode = "explicit"), with no breakpoints, on both endpoints. Set prompt_caching = "implicit" to omit that field and retain provider caching when prefix reuse is expected to be worthwhile.

Recognized earlier OpenAI models retain their historical request bodies when the policy is omitted or NULL; explicitly disabling caching on those models errors. Unknown IDs (including unlisted snapshots, future models and custom aliases) require an explicit "implicit" opt-in or a verified model ID. Support is an offline allowlist; version numbers and snapshot suffixes never establish support for disabling caching. Older GPT-3.5/4, GPT-5 through GPT-5.5 and o-series naming forms, including recognized legacy date-suffixed names, retain provider defaults.

Invalid policies and manual cache controls fail before submission, even with zero pairs. Raw prompt_cache_options, prompt_cache_breakpoint, prompt_cache_key, retention and TTL controls are not accepted by these builders. Keys affect cache routing/accounting; retention and TTL affect lifetime; store = FALSE affects response storage. None substitutes for disabling implicit cache breakpoints. Prompt text and keys are not rewritten. This policy affects future package-built Batch requests only, not live requests or already submitted jobs. See the prompt-caching guide and vignette("advanced-batch-workflows") for usage-based cost comparisons.

Examples

# The OpenAI batch pipeline requires:
# - Internet access
# - A valid OpenAI API key in OPENAI_API_KEY (or supplied via `api_key`)
# - Billable API usage
#
if (FALSE) { # \dontrun{
data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 2, seed = 123) |>
  randomize_pair_order(seed = 456)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# Run a small batch using chat.completions
out <- run_openai_batch_pipeline(
  pairs             = pairs,
  model             = "gpt-4.1",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  endpoint          = "chat.completions",
  poll              = TRUE,
  interval_seconds  = 5,
  timeout_seconds   = 600
)

print(out$batch$status)
print(utils::head(out$results))
} # }