Live decision execution, persistence and recovery contracts

Collect and resume OpenAI Decisions and TypeSafe Jev comparisons through submit_llm_pairs(), submit_openai_pairs_live() or submit_typesafe_pairs_live(). Each pair occurrence is one live request; these endpoints do not use provider-managed Batch jobs.

Execution controls

The bulk wrappers accept execution_control, a uniquely named list. Unknown names, invalid scalars and unsupported provider options fail before HTTP. parallel = FALSE or workers = 1 selects sequential execution. parallel = TRUE, workers > 1 uses optional promises and later through httr2's experimental public asynchronous API. Missing packages cause an informative error before requests; no packages are installed automatically. Workers bound simultaneous in-flight requests, not R processes. One private event loop and curl pool belong to each session and are cleaned up on exit. The coordinator explicitly implements retries and admission; asynchronous httr2 retry/throttle policies are not assumed to apply.

Supported controls and defaults are:

requests_per_minute = NULL

Optional positive integer maximum transmission reservations in a rolling 60-second window.

tokens_per_minute = NULL

Optional positive integer maximum estimated token reservations in that window. Requires token_estimates.

token_estimates = NULL

Positive integer scalar or vector in input row order, including an output allowance. A reservation must fit the token limit. No tokenizer or missing provider usage is guessed. This bounds reservations, not actual usage when estimates are too small.

max_attempts = 3L

Persistent total send-reservation budget per occurrence, including initial, transient and ambiguous-recovery sends. It is fixed in the saved manifest; restarting does not replenish it.

timeout_seconds = 120

Positive finite timeout per transmission.

retry_ambiguous = TRUE

Allow at most one automatic reissue of an interrupted occurrence across all restarts, also within max_attempts. FALSE leaves ambiguous work unresolved. Toggling it never replenishes the consumed allowance.

lock_timeout_ms = 10000

Nonnegative finite milliseconds to wait for an exclusive saved-run lock before failing without provider calls.

Rate limits are caller configuration, not assumed provider quotas. Recent reservations, retry deadlines and shared provider cooldowns survive restart. Limits apply to this saved run; other sessions sharing an account can consume the same provider quota. Limits and timeout may change on resume without changing scientific identity; prior reservations remain counted.

Retries and ambiguous requests

Bulk OpenAI retries HTTP 408, 409, 429 and all 5xx statuses; TypeSafe retries 408, 429 and all 5xx, including 529. Recognized OpenAI insufficient-quota or billing-limit errors are terminal. Timeouts receive bounded transient retries; other transport failures and permanent HTTP errors remain unresolved. Exponential delays start at 0.5 seconds, double up to 30 seconds and add uniform jitter of up to 0.25 seconds. Valid Retry-After seconds or HTTP dates are a minimum delay, with shared cooldowns for 429, 503 and 529. Invalid headers use backoff. Single-pair retry defaults are unchanged.

A send reservation is committed before transmission. An interruption before transmission may therefore conservatively consume an attempt. Reservations without a committed response have an unknown provider outcome. Recovery first checks local committed evidence. The currently supported endpoint contracts provide no documented remote result lookup/idempotency mechanism: reconciliation is recorded as unsupported, without speculative retrieval requests. One automatic ambiguous reissue is permitted by default, across all restarts. Exhaustion leaves the occurrence in failed_pairs for explicit operator action; an operator can deliberately start a separate run containing only unresolved source pairs. Preserve the old journal rather than editing it or resetting budgets. Timeout retries can also incur duplicate charges. Neither locking nor checkpoints establish exactly-once provider execution.

Malformed/refused decisions, incomplete probabilities, invalid confidence and choice/probability inconsistencies are recorded without automatic retry and never become completed evidence. Only normalized valid results enter modeling.

Request identity

All four input ID/text columns must be nonmissing character columns. Supplied pair_uid values must be unique and nonempty. Otherwise deterministic scientific-fingerprint IDs plus occurrence suffixes distinguish repeats and reversed presentations. Supply explicit IDs when scheduled occurrences need independent labels. IDs are never matched by row position alone.

The saved workset must have the same occurrence membership, scientific payload and provider wire controls. Matching occurrences can be reordered. Adding or removing occurrences, or changing texts, orientation, trait, instructions, choices, model/provider/endpoint or OpenAI cache controls requires a separate save path. Omitted decision prompts select the exact decision_cj_v1 wording; custom instructions follow decision_request. Generative templates fail.

Every current request is rebuilt and validated even when its result is already saved. The shared scientific validator checks body, ordered evidence hashes, approved question core, provenance and fingerprint. Serialization and attached raw UTF-8 bytes are checked before every transmission, including retries. scientific_payload_sha256 hashes the serialized scientific body; wire_body_sha256 hashes the exact transmitted JSON body, including supported cache controls. These digests exclude authentication headers and timeout. Digests detect identity/integrity mismatches, not malicious modification with recomputed checksums. Requested and provider-reported resolved models remain separate. Moving aliases may change weights; saved results retain the originally reported resolution, and no availability or alias-resolution calls are made.

Persistence and output

A decision save_path names a CSV export and requires optional filelock. The dedicated <save_path>.lock is locked before checkpoints are read and held through all asynchronous calls, retries and writes. The OS lock releases on exit/process termination; its file is retained. Only one writer is admitted.

<save_path>.checkpoint/ is the authoritative lossless RDS journal. Its immutable checksummed manifest and sequenced records are written through same-directory temporary files, flushed/closed, then renamed to new final names. Each response is validated and committed immediately, independently of other in-flight work. Completed committed occurrences are never automatically submitted again. Uncommitted temporary files are ignored; corrupt committed files, missing interior records and unsupported schemas stop execution before HTTP. CSV alone cannot establish completion and is not imported as a resume journal.

CSV contains valid results with stable scalar columns, including all probability and provenance columns. Numeric values use 17 significant digits to round-trip R doubles with a correctly rounded decimal reader. Some readers (including base numeric conversion on macOS ARM in qualification) can change the last bit. For exact numeric CSV recovery, read columns as character and decode probability cells with jsonlite::parse_json(); use RDS for exact native recovery. Read IDs as character to preserve leading zeros. A missing or torn CSV export is rebuilt from RDS on resume. Journal envelopes are RDS lists with a SHA-256 and serialized raw data; unserialize(readRDS(record)$data) exposes the versioned event and its state/attempt records. Treat these as trusted local files, not a public cross-version interchange format.

The returned list retains results, failed_pairs and failed_attempts. Results preserve decision_results fields, exact parsed R double probabilities, full probabilities_json, independent confidence, usage and model provenance. Missing usage/resolution stays missing; no price or cost estimate is fabricated. Additional diagnostics include payload digests, controls, local attempt_id, attempt_count, ambiguous_reissues, possible_duplicate_charge, elapsed_seconds since admission, and recovery_status. Failed attempts retain unique local IDs, reason (initial, transient_retry, ambiguous_recovery), timestamps, available original/new provider request IDs, provenance, and reconciliation status. Provider IDs unknown at interruption remain missing. Successful outcomes and failed attempts remain separate; history is not re-added on every restart. Probabilities never become BT/adaptive weights, fractional wins or extra comparisons.

No durability is promised without save_path. Saved-run guarantees concern process interruption on a filesystem supporting OS locks and atomic rename; they are not power-loss/fsync guarantees. Network/cloud-synchronized storage and cross-machine locking require filesystem guarantees beyond this package.

Privacy

Samples go to the selected provider. Normal messages and scalar checkpoints omit API keys and sample text. Provenance retains complete custom instructions and trait definitions, which can themselves be sensitive. CSV excludes input text columns and arbitrary caller metadata. include_raw = TRUE explicitly retains parsed provider responses in RDS and may contain sensitive content; FALSE omits them from new persisted records, including failures. Existing opted-in records are not erased. Increasing raw retention on resume cannot recreate responses previously omitted and never triggers resubmission. In-memory failure objects may contain raw responses even with include_raw = FALSE. Explicitly saving or serializing the complete returned object (for example with saveRDS()) can persist sensitive text. Successful normalized results can also retain source text columns. The default journal/export privacy policy does not sanitize caller-created stores. Keep journals and returned diagnostics private.

See also

submit_llm_pairs(), decision_request, decision_results

Examples

if (FALSE) { # \dontrun{
# Requires provider credentials; incurs usage charges.
pairs <- data.frame(ID1 = c("A", "B"), ID2 = c("B", "A"),
                    text1 = c("Synthetic A.", "Synthetic B."),
                    text2 = c("Synthetic B.", "Synthetic A."),
                    pair_uid = c("forward", "reverse"))
result <- submit_llm_pairs(
  pairs, model = "jev-1.13.0", backend = "typesafe",
  trait_name = "Organization", trait_description = "Logical sequencing.",
  parallel = TRUE, workers = 4, save_path = "decisions.csv",
  execution_control = list(requests_per_minute = 60,
                           retry_ambiguous = FALSE)
)
# Repeat the same call to resume; preserve its .checkpoint directory.
result$failed_pairs
result$failed_attempts
} # }