Evidential Experiments API
The Experiments API connects your application to the experiments you design in Evidential. Your application asks Evidential which arm each participant is in and, for bandit experiments, reports the outcome it observes.
Authentication
Every request needs a data source API key in the X-API-Key header. Create and manage keys in Evidential under
Settings, in the Data sources section. Keys begin with xat.
curl https://api.evidential.dev/v1/experiments/<experiment_id> \
--header 'X-API-Key: <your API key>'
Choose your endpoints
Assignment works the same way for every experiment type: your application asks Evidential which arm a participant belongs to. What differs is how outcomes reach Evidential:
- Pull: Evidential reads outcomes from your data source, and your application reports none.
- Push: your application reports each outcome by calling Record an outcome.
The endpoints you call depend on the experiment type:
- Preassigned A/B (pull): assignments are created when the experiment is designed. List or export them all, or look up one participant's at runtime.
- Online A/B (pull): get or create an assignment as each participant arrives, or assign with filters if the experiment uses server-side filtering.
- Multi-armed bandit, MAB (push): get or create an assignment, then record its outcome.
- MAB with a data warehouse outcome (pull,
mab_online_dwh): get or create an assignment. Evidential reads each outcome from the column bound to the experiment. - Contextual multi-armed bandit, CMAB (push): assign with context values, then record its outcome.
To keep your own systems in sync with Evidential, list the experiments on a data source and get an experiment's design.
Errors
Branch on the HTTP status code, not on the text of the error message: status codes are stable, but message text may
change. 2xx codes mean success, 4xx codes usually mean a problem with your request, and 5xx codes mean a
server-side error that may be retried.
Retries
Retry only transient failures: network errors and 5xx responses. Use exponential backoff with jitter, and cap the
number of attempts. Don't retry 4xx responses; they fail again until you change the request, or until the
experiment, data source, or assignment changes. Assignment requests are safe to retry, because they return the existing
assignment when one was already created.
Experiments
Look up the experiments on a data source and their designs.
Assignments
Get or create participants' arm assignments, and list or export an experiment's assignments.
- Get or create a participant's assignmentGET
/v1/experiments/{experiment_id}/assignments/{participant_id} - Get or create an Online A/B assignment with server-side filteringPOST
/v1/experiments/{experiment_id}/assignments/{participant_id}/assign_with_filters - Get or create a CMAB assignment for a participantPOST
/v1/experiments/{experiment_id}/assignments/{participant_id}/assign_cmab - List an experiment's assignmentsGET
/v1/experiments/{experiment_id}/assignments - Export an experiment's assignments as CSVGET
/v1/experiments/{experiment_id}/assignments/csv
Outcomes
Report the outcomes of bandit experiments that use push integration.