Data Schemas and Prompt Management

This article follows data from source samples to modeling inputs and shows how prompt templates are managed. All examples are deterministic and run without provider credentials.

A schema is the required set of column names and types. You need a data frame of texts; you will get pair rows, prompts, and modeling inputs. This page explains those transitions. Start with Getting Started if you want to see the complete analysis before customizing it.

Bring a spreadsheet or data frame

Use one row per writing sample. Keep a unique, non-missing character ID for each sample across collection, saved results, and scoring; do not use row position as identity. A CSV import can map your existing column names:

library(pairwiseLLM)
writing <- read.csv("writing.csv", colClasses = "character")
samples <- read_samples_df(writing, id_col = "essay_id", text_col = "response")

For example, essay_id = "E001" and response = "The essay text..." become ID = "E001" and text = "The essay text...". Remove blank or missing IDs and texts before continuing. The reader checks duplicate IDs; it is not a substitute for inspecting the quality of your input data.

Samples and pairs

read_samples_df() normalizes a data frame and read_samples_dir() reads one text file per item. Both return an ID/text table suitable for the fixed-pair workflow. Existing data frames can be used directly when they already have those columns.

library(pairwiseLLM)

data("example_writing_samples", package = "pairwiseLLM")
samples <- read_samples_df(example_writing_samples[, c("ID", "text")])

pairs <- samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 4L, seed = 10L) |>
  randomize_pair_order(seed = 11L)

pairs[, c("ID1", "text1", "ID2", "text2")]
#> # A tibble: 4 × 4
#>   ID1   text1                                                        ID2   text2
#>   <chr> <chr>                                                        <chr> <chr>
#> 1 S12   "Evaluating writing is challenging because no rubric can fu… S10   "The…
#> 2 S09   "Assessing writing is difficult because the construct is mu… S05   "Wri…
#> 3 S07   "Writing assessment is difficult because writing is a compl… S20   "Wri…
#> 4 S19   "Writing assessment is difficult because it asks us to quan… S16   "Ass…

The fixed-pair schema uses ID1, text1, ID2, and text2. Reversal helpers preserve the unordered pair while changing presentation order. Keep the identifiers: they are needed to map a provider’s SAMPLE_1 or SAMPLE_2 decision back to better_id.

Traits and prompt templates

Generative templates require all four placeholders: {TRAIT_NAME}, {TRAIT_DESCRIPTION}, {SAMPLE_1}, and {SAMPLE_2}. build_prompt() substitutes them but does not submit anything to a provider.

trait <- trait_description("overall_quality")
template <- set_prompt_template()

prompt <- build_prompt(
  template = template,
  trait_name = trait$name,
  trait_desc = trait$description,
  text1 = pairs$text1[[1]],
  text2 = pairs$text2[[1]]
)

substr(prompt, 1L, 180L)
#> [1] "You are a debate adjudicator. Your task is to weigh the comparative strengths of two writing samples regarding a specific trait.\n\nTRAIT: Overall Quality\nDEFINITION: Overall quality"

The named registry lasts only for the current R session. Registration validates placeholders; overwrite = FALSE protects an existing name. Persist a custom template by storing its text in your project and registering it from a startup script, not by modifying package files.

example_name <- "task07_example"
register_prompt_template(example_name, template = template)
example_name %in% list_prompt_templates()
#> [1] TRUE
identical(get_prompt_template(example_name), template)
#> [1] TRUE

register_prompt_template(example_name, template = template, overwrite = TRUE)
remove_prompt_template(example_name)

set_prompt_template() returns the built-in default or validates an inline/file template. Save its return value and pass it as prompt_template; it does not change a global active prompt. get_prompt_template() resolves a user registration before a built-in template of the same name. remove_prompt_template() removes only session registrations; it cannot delete built-in files.

Decision instructions and probabilities

OpenAI Decisions and TypeSafe Jev ask a binary choice question instead of asking the model to generate a tagged answer. Omitting prompt_template selects the same built-in decision_cj_v1 instructions for both. This default is separate from set_prompt_template() and its session registry. Pass your complete trait_name and trait_description; the description is preserved in full. For a custom trait, include its boundaries and what the assessor should ignore.

You can prepare the settings entirely offline:

decision_settings <- list(
  trait_name = "Organization",
  trait_description = paste(
    "Judge logical sequencing and connections between ideas.",
    "Exclude spelling and grammar except when they obscure organization."
  )
)
# Optional replacement instructions; omit to use decision_cj_v1.
custom_instructions <- paste(
  "Assess {TRAIT_NAME} using the full guidance below.",
  "{TRAIT_DESCRIPTION}",
  "Select the sample with stronger performance on this trait.", sep = "\n"
)

Supply custom instructions as decision_template, retaining both trait placeholders. Sample text is supplied separately in presentation order. Do not put sample placeholders or the generative <BETTER_SAMPLE> tag into decision instructions. See llm_compare_pair() for supported controls and validation.

This invented result illustrates the output without making a provider call:

synthetic_decision <- data.frame(
  ID1 = "A", ID2 = "B", better_id = "B",
  p_sample_1 = 0.3, p_sample_2 = 0.7, p_chosen = 0.7,
  decision_confidence = 0.4
)
synthetic_decision
#>   ID1 ID2 better_id p_sample_1 p_sample_2 p_chosen decision_confidence
#> 1   A   B         B        0.3        0.7      0.7                 0.4
build_bt_data(synthetic_decision)
#> # A tibble: 1 × 3
#>   object1 object2 result
#>   <chr>   <chr>    <dbl>
#> 1 A       B            0

p_sample_1 refers to ID1, and p_sample_2 to ID2, as presented. Compare the probability assigned to the same ID when auditing reversals; do not compare position columns without remapping IDs. Equal probabilities still require an explicit selected winner. decision_confidence is a separate provider field, not a substitute for either probability. These values have no demonstrated calibration for educational judgments and do not become BT weights or extra comparisons. The compatibility record describes the observed live results and their limits.

Provider and result schemas

The live wrappers accept the fixed-pair rows directly. Batch request builders convert the same rows to provider request records, and their matching parsers convert downloaded output back to package results. Do not send a request table built for one provider to another provider’s submission function.

The normalized result bundle retains pair identifiers and adds provider metadata, visible content, optional thoughts, better_sample, better_id, and token counts. Row-wise live calls return a list whose valid results, unresolved failed_pairs, and attempt-level failed_attempts components must be inspected separately. Batch parsers may omit unsuccessful rows from their successful result table, so preserve the provider output/error file and job registry for audit and recovery.

samples (ID, text)
  -> pairs (ID1, text1, ID2, text2)
     -> live call ---------------------> normalized results
     -> provider request + batch output -> provider parser -> normalized results

Modeling and adaptive branches

For frequentist Bradley–Terry or Elo models, start from rows with ID1, ID2, and a valid better_id and call build_bt_data() or build_elo_data(). Invalid or missing winners are not valid outcomes; inspect failures before modeling rather than silently treating them as ties.

data("example_writing_pairs", package = "pairwiseLLM")

bt_data <- build_bt_data(example_writing_pairs)
elo_data <- build_elo_data(example_writing_pairs)
bayes_data <- build_btl_results_data(example_writing_pairs)

names(bt_data)
#> [1] "object1" "object2" "result"
names(elo_data)
#> [1] "winner" "loser"
names(bayes_data)
#>  [1] "pair_uid"      "unordered_key" "ordered_key"   "A_id"         
#>  [5] "B_id"          "better_id"     "winner_pos"    "phase"        
#>  [9] "iter"          "received_at"   "backend"       "model"

Standalone Bayesian BTL uses the stricter canonical schema returned by build_btl_results_data(); see Standalone Bayesian BTL with CmdStan.

Adaptive ranking is a separate branch. adaptive_rank() starts from raw items and records each attempt in an adaptive state with canonical step, round, link-stage, and item logs. A normalized fixed-pair result table is not a drop-in replacement for an adaptive session. Use adaptive_results_history() only when you need the committed adaptive outcomes in build_bt_data() format.

Failure checks at schema boundaries

Reject missing, blank, or duplicate sample IDs before pairing.
Preserve the original pair IDs and provider custom_id through batch submission and parsing.
Model only outcomes whose better_id matches one member of the pair.
Treat thoughts and raw responses as sensitive submitted/returned text when deciding retention.
Validate persisted adaptive sessions with validate_session_dir() rather than editing .rds artifacts or coercing their schemas by hand.

See Getting Started with pairwiseLLM for the basic fixed-pair workflow, Provider Controls and Recovery for live and batch failures, and Guide: Adaptive Pairing for adaptive state and logs.

Citation

Mercer, S. H. (2026). Data schemas and prompt management [R package vignette]. Comprehensive R Archive Network. https://doi.org/10.32614/CRAN.package.pairwiseLLM