Skip to content
Navigation

Get an experiment's design

GET
https://api.evidential.dev/v1/experiments/{experiment_id}

Authorization

  • DatasourceApiKey
    API key in header
    required

    X-API-Key: <token>

Path parameters

  • experiment_id
    string
    required

    The ID of the experiment to fetch.

Returns

200

Successful Response

JSON
  • experiment_id
    string
    required

    Server-generated ID of the experiment.

  • datasource_id
    string
    required
  • state
    assigned
    or
    abandoned
    or
    committed
    required

    Current state of this experiment.

  • stopped_assignments_at
    required

    The date and time when assignments were stopped. Null if the experiment is still accepting assignments. This date can change if the experiment's end date is modified.

    Any of the following:

    • string
    • null
  • stopped_assignments_reason
    required

    Any of the following:

    • StopAssignmentReason
    • null
  • design_spec
    DesignSpec
    required

    The type of assignment and experiment design.

    One of the following:

    • AnyBanditDesignSpec
      cmab_online

      One of the following:

      • CMABExperimentSpec
        cmab_online
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
          • alpha_init
            optional

            Initial alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta_init
            optional

            Initial beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu_init
            optional

            Initial mean parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • sigma_init
            optional

            Initial standard deviation parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • alpha
            optional

            Updated alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta
            optional

            Updated beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu
            optional

            Updated mean vector for the Normal prior.

            Any of the following:

            • array<number>
            • null
          • covariance
            optional

            Updated covariance matrix for the Normal prior.

            Any of the following:

            • array<array<number>>
            • null
        • experiment_type
          optional
          string
          default: "cmab_online"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • contexts
          optional

          Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

          Any of the following:

          • array<unknown>
            • context_name
              string
              required
              maxLength: 100
            • context_id
              optional

              Unique identifier for the context. Do not set this when you create a new context.

              Any of the following:

              • string
              • null
            • context_description
              optional

              Any of the following:

              • string
              • null
            • value_type
              optional
              binary
              or
              real-valued

              Type of value the context can take.

              default: "binary"
          • null
        • prior_type
          optional
          beta
          or
          normal

          The type of prior distribution for the arms.

          default: "beta"
        • reward_type
          optional
          binary
          or
          real-valued

          The type of outcome observed from the experiment.

          default: "binary"
        • enable_autofail
          optional
          boolean

          When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

          default: false
        • autofail_window
          optional
          integer

          The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

          minimum: 1 · default: 24
        • autofail_outcome_value
          optional
          number

          The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

          default: 0
      • MABExperimentSpec
        mab_online
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
          • alpha_init
            optional

            Initial alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta_init
            optional

            Initial beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu_init
            optional

            Initial mean parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • sigma_init
            optional

            Initial standard deviation parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • alpha
            optional

            Updated alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta
            optional

            Updated beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu
            optional

            Updated mean vector for the Normal prior.

            Any of the following:

            • array<number>
            • null
          • covariance
            optional

            Updated covariance matrix for the Normal prior.

            Any of the following:

            • array<array<number>>
            • null
        • experiment_type
          optional
          string
          default: "mab_online"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • contexts
          optional

          Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

          Any of the following:

          • array<unknown>
            • context_name
              string
              required
              maxLength: 100
            • context_id
              optional

              Unique identifier for the context. Do not set this when you create a new context.

              Any of the following:

              • string
              • null
            • context_description
              optional

              Any of the following:

              • string
              • null
            • value_type
              optional
              binary
              or
              real-valued

              Type of value the context can take.

              default: "binary"
          • null
        • prior_type
          optional
          beta
          or
          normal

          The type of prior distribution for the arms.

          default: "beta"
        • reward_type
          optional
          binary
          or
          real-valued

          The type of outcome observed from the experiment.

          default: "binary"
        • enable_autofail
          optional
          boolean

          When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

          default: false
        • autofail_window
          optional
          integer

          The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

          minimum: 1 · default: 24
        • autofail_outcome_value
          optional
          number

          The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

          default: 0
      • MABDwhExperimentSpec
        mab_online_dwh
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
          • alpha_init
            optional

            Initial alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta_init
            optional

            Initial beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu_init
            optional

            Initial mean parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • sigma_init
            optional

            Initial standard deviation parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • alpha
            optional

            Updated alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta
            optional

            Updated beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu
            optional

            Updated mean vector for the Normal prior.

            Any of the following:

            • array<number>
            • null
          • covariance
            optional

            Updated covariance matrix for the Normal prior.

            Any of the following:

            • array<array<number>>
            • null
        • table_name
          string
          required

          Datasource table used to resolve target field metadata.

          maxLength: 100
        • primary_key
          string
          required

          Column name in table_name that uniquely identifies each participant.

          maxLength: 300 · minLength: 1
        • target_field_name
          string
          required

          Column name in table_name whose values this bandit optimises. Its data type is resolved from the DWH at create-time and used to validate subsequent outcome reports.

          maxLength: 300 · minLength: 1
        • experiment_type
          optional
          string
          default: "mab_online_dwh"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • contexts
          optional

          Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

          Any of the following:

          • array<unknown>
            • context_name
              string
              required
              maxLength: 100
            • context_id
              optional

              Unique identifier for the context. Do not set this when you create a new context.

              Any of the following:

              • string
              • null
            • context_description
              optional

              Any of the following:

              • string
              • null
            • value_type
              optional
              binary
              or
              real-valued

              Type of value the context can take.

              default: "binary"
          • null
        • prior_type
          optional
          beta
          or
          normal

          The type of prior distribution for the arms.

          default: "beta"
        • reward_type
          optional
          binary
          or
          real-valued

          The type of outcome observed from the experiment.

          default: "binary"
        • enable_autofail
          optional
          boolean

          When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

          default: false
        • autofail_window
          optional
          integer

          The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

          minimum: 1 · default: 24
        • autofail_outcome_value
          optional
          number

          The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

          default: 0
    • AnyFrequentistDesignSpec
      freq_online

      One of the following:

      • OnlineFrequentistExperimentSpec
        freq_online
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
        • table_name
          string
          required

          Data source table used to resolve participant field metadata.

          maxLength: 100
        • primary_key
          string
          required

          Column name in table_name that uniquely identifies each participant.

          maxLength: 300 · minLength: 1
        • strata
          array of
          object
          required

          Optional fields to use for stratified assignment.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
        • metrics
          array of
          object
          required

          Primary and optional secondary metrics to target.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
          • metric_pct_change
            optional

            The smallest percent change, relative to metric_baseline, that you want to be able to detect. Cannot be set together with metric_target.

            Any of the following:

            • number
            • null
          • metric_target
            optional

            The absolute value you want to be able to detect. Cannot be set together with metric_pct_change.

            Any of the following:

            • number
            • null
          • icc
            optional

            Intracluster correlation coefficient for cluster-randomized designs.

            Any of the following:

            • number
            • null
          • avg_cluster_size
            optional

            Average number of participants per cluster.

            Any of the following:

            • number
            • null
          • cv
            optional

            Coefficient of variation in cluster sizes (0 = equal sizes).

            Any of the following:

            • number
            • null
          • metric_type
            optional

            Inferred from the data warehouse column type.

            Any of the following:

            • MetricType
            • null
          • metric_baseline
            optional

            Mean of the tracked metric.

            Any of the following:

            • number
            • null
          • metric_stddev
            optional

            Standard deviation is set only for MetricType.NUMERIC metrics. Must be set for numeric metrics when available_n > 0.

            Any of the following:

            • number
            • null
          • available_nonnull_n
            optional

            The number of participants who meet the filtering criteria and have a non-null value for this metric.

            Any of the following:

            • integer
            • null
          • available_n
            optional

            The number of participants who meet the filtering criteria, whether or not this metric's value is NULL. Note: assignments are made from the available_n population. When available_n is greater than available_nonnull_n, some assigned participants may have a missing value for this metric.

            Any of the following:

            • integer
            • null
          • use_one_time_metric
            optional
            boolean

            Enable one-time metric mode: only assign participants with null values for this metric. Only allowed on the primary (first) metric of an individually-randomized preassigned experiment.

            default: false
        • filters
          array of
          object
          required

          Optional filters that narrow the eligible audience to the participants you want in the experiment.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
          • relation
            includes
            or
            excludes
            or
            between
            required

            Defines operators for filtering values.

            INCLUDES matches when the value matches any of the provided values, including null if explicitly specified. This is equivalent to NOT(EXCLUDES(values)).

            EXCLUDES matches when the value does not match any of the provided values, including null if explicitly specified. If null is not explicitly excluded, we include nulls in the result.

            BETWEEN matches when the value is between the two provided bounds (inclusive). Either bound can be null for an open-ended range. A null value matches only if a 3rd value of null is provided. BETWEEN applies to numeric, date and timestamp values.

          • value
            FilterValueTypes
            required

            Any of the following:

            • array<unknown>

              Any of the following:

              • integer
              • null
            • array<unknown>

              Any of the following:

              • number
              • null
            • array<any of>

              Any of the following:

              • string
              • null
            • array<any of>

              Any of the following:

              • boolean
              • null
        • experiment_type
          optional
          string
          default: "freq_online"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • desired_n
          optional

          Desired number of individual participants. Required when creating individual-randomized preassigned experiments. For cluster-randomized preassigned experiment creation, use desired_n_clusters to control sampling; desired_n remains available for power calculations but is superseded by desired_n_clusters when both are set. When set, the power check also returns the minimum detectable effect for this size, along with the minimum sample size.

          Any of the following:

          • integer
          • null
        • desired_ns
          optional

          Optional list of desired individual participant sample sizes. The power check returns the minimum detectable effect for each size (MetricPowerAnalysis.mde_curve), letting clients plot an MDE-vs-sample-size power curve from a single request. Superseded by desired_ns_clusters when both are set. Ignored when creating an experiment.

          Any of the following:

          • array<integer>
          • null
        • power
          optional
          number

          The chance of detecting a real effect when one truly exists (1 - the false negative rate).

          maximum: 1 · minimum: 0 · default: 0.8
        • alpha
          optional
          number

          The chance of a false positive: concluding there is an effect when there is none.

          maximum: 1 · minimum: 0 · default: 0.05
        • fstat_thresh
          optional
          number

          The p-value threshold for the balance check. When the balance check's p-value is above this threshold, we treat the assignment as "balanced".

          maximum: 1 · minimum: 0 · default: 0.6
      • PreassignedFrequentistExperimentSpec
        freq_preassigned
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
        • table_name
          string
          required

          Data source table used to resolve participant field metadata.

          maxLength: 100
        • primary_key
          string
          required

          Column name in table_name that uniquely identifies each participant.

          maxLength: 300 · minLength: 1
        • strata
          array of
          object
          required

          Optional fields to use for stratified assignment.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
        • metrics
          array of
          object
          required

          Primary and optional secondary metrics to target.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
          • metric_pct_change
            optional

            The smallest percent change, relative to metric_baseline, that you want to be able to detect. Cannot be set together with metric_target.

            Any of the following:

            • number
            • null
          • metric_target
            optional

            The absolute value you want to be able to detect. Cannot be set together with metric_pct_change.

            Any of the following:

            • number
            • null
          • icc
            optional

            Intracluster correlation coefficient for cluster-randomized designs.

            Any of the following:

            • number
            • null
          • avg_cluster_size
            optional

            Average number of participants per cluster.

            Any of the following:

            • number
            • null
          • cv
            optional

            Coefficient of variation in cluster sizes (0 = equal sizes).

            Any of the following:

            • number
            • null
          • metric_type
            optional

            Inferred from the data warehouse column type.

            Any of the following:

            • MetricType
            • null
          • metric_baseline
            optional

            Mean of the tracked metric.

            Any of the following:

            • number
            • null
          • metric_stddev
            optional

            Standard deviation is set only for MetricType.NUMERIC metrics. Must be set for numeric metrics when available_n > 0.

            Any of the following:

            • number
            • null
          • available_nonnull_n
            optional

            The number of participants who meet the filtering criteria and have a non-null value for this metric.

            Any of the following:

            • integer
            • null
          • available_n
            optional

            The number of participants who meet the filtering criteria, whether or not this metric's value is NULL. Note: assignments are made from the available_n population. When available_n is greater than available_nonnull_n, some assigned participants may have a missing value for this metric.

            Any of the following:

            • integer
            • null
          • use_one_time_metric
            optional
            boolean

            Enable one-time metric mode: only assign participants with null values for this metric. Only allowed on the primary (first) metric of an individually-randomized preassigned experiment.

            default: false
        • filters
          array of
          object
          required

          Optional filters that narrow the eligible audience to the participants you want in the experiment.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
          • relation
            includes
            or
            excludes
            or
            between
            required

            Defines operators for filtering values.

            INCLUDES matches when the value matches any of the provided values, including null if explicitly specified. This is equivalent to NOT(EXCLUDES(values)).

            EXCLUDES matches when the value does not match any of the provided values, including null if explicitly specified. If null is not explicitly excluded, we include nulls in the result.

            BETWEEN matches when the value is between the two provided bounds (inclusive). Either bound can be null for an open-ended range. A null value matches only if a 3rd value of null is provided. BETWEEN applies to numeric, date and timestamp values.

          • value
            FilterValueTypes
            required

            Any of the following:

            • array<unknown>

              Any of the following:

              • integer
              • null
            • array<unknown>

              Any of the following:

              • number
              • null
            • array<any of>

              Any of the following:

              • string
              • null
            • array<any of>

              Any of the following:

              • boolean
              • null
        • experiment_type
          optional
          string
          default: "freq_preassigned"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • desired_n
          optional

          Desired number of individual participants. Required when creating individual-randomized preassigned experiments. For cluster-randomized preassigned experiment creation, use desired_n_clusters to control sampling; desired_n remains available for power calculations but is superseded by desired_n_clusters when both are set. When set, the power check also returns the minimum detectable effect for this size, along with the minimum sample size.

          Any of the following:

          • integer
          • null
        • desired_ns
          optional

          Optional list of desired individual participant sample sizes. The power check returns the minimum detectable effect for each size (MetricPowerAnalysis.mde_curve), letting clients plot an MDE-vs-sample-size power curve from a single request. Superseded by desired_ns_clusters when both are set. Ignored when creating an experiment.

          Any of the following:

          • array<integer>
          • null
        • power
          optional
          number

          The chance of detecting a real effect when one truly exists (1 - the false negative rate).

          maximum: 1 · minimum: 0 · default: 0.8
        • alpha
          optional
          number

          The chance of a false positive: concluding there is an effect when there is none.

          maximum: 1 · minimum: 0 · default: 0.05
        • fstat_thresh
          optional
          number

          The p-value threshold for the balance check. When the balance check's p-value is above this threshold, we treat the assignment as "balanced".

          maximum: 1 · minimum: 0 · default: 0.6
        • cluster_key
          optional

          Column name in table_name that identifies clusters for a cluster-randomized design. When set, per-metric icc, avg_cluster_size, and cv are either supplied on each metric or computed from this column at power_check time. When None, the design is assumed to be individual-randomized.

          Any of the following:

          • string
          • null
        • desired_n_clusters
          optional

          Desired number of clusters to sample when creating a cluster-randomized preassigned experiment. Required and only valid when cluster_key is set. All eligible participants in each sampled cluster are included. In power checks, the minimum detectable effect is computed for this many clusters, converted to a per-metric sample size using each metric's avg_cluster_size; takes precedence over desired_n.

          Any of the following:

          • integer
          • null
        • desired_ns_clusters
          optional

          Optional list of desired cluster counts. Only valid when cluster_key is set. The power check returns the minimum detectable effect for each count (MetricPowerAnalysis.mde_curve), converted to a per-metric sample size using each metric's avg_cluster_size; takes precedence over desired_ns. Ignored when creating an experiment.

          Any of the following:

          • array<integer>
          • null
    • AnyFrequentistDesignSpec
      freq_preassigned

      One of the following:

      • OnlineFrequentistExperimentSpec
        freq_online
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
        • table_name
          string
          required

          Data source table used to resolve participant field metadata.

          maxLength: 100
        • primary_key
          string
          required

          Column name in table_name that uniquely identifies each participant.

          maxLength: 300 · minLength: 1
        • strata
          array of
          object
          required

          Optional fields to use for stratified assignment.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
        • metrics
          array of
          object
          required

          Primary and optional secondary metrics to target.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
          • metric_pct_change
            optional

            The smallest percent change, relative to metric_baseline, that you want to be able to detect. Cannot be set together with metric_target.

            Any of the following:

            • number
            • null
          • metric_target
            optional

            The absolute value you want to be able to detect. Cannot be set together with metric_pct_change.

            Any of the following:

            • number
            • null
          • icc
            optional

            Intracluster correlation coefficient for cluster-randomized designs.

            Any of the following:

            • number
            • null
          • avg_cluster_size
            optional

            Average number of participants per cluster.

            Any of the following:

            • number
            • null
          • cv
            optional

            Coefficient of variation in cluster sizes (0 = equal sizes).

            Any of the following:

            • number
            • null
          • metric_type
            optional

            Inferred from the data warehouse column type.

            Any of the following:

            • MetricType
            • null
          • metric_baseline
            optional

            Mean of the tracked metric.

            Any of the following:

            • number
            • null
          • metric_stddev
            optional

            Standard deviation is set only for MetricType.NUMERIC metrics. Must be set for numeric metrics when available_n > 0.

            Any of the following:

            • number
            • null
          • available_nonnull_n
            optional

            The number of participants who meet the filtering criteria and have a non-null value for this metric.

            Any of the following:

            • integer
            • null
          • available_n
            optional

            The number of participants who meet the filtering criteria, whether or not this metric's value is NULL. Note: assignments are made from the available_n population. When available_n is greater than available_nonnull_n, some assigned participants may have a missing value for this metric.

            Any of the following:

            • integer
            • null
          • use_one_time_metric
            optional
            boolean

            Enable one-time metric mode: only assign participants with null values for this metric. Only allowed on the primary (first) metric of an individually-randomized preassigned experiment.

            default: false
        • filters
          array of
          object
          required

          Optional filters that narrow the eligible audience to the participants you want in the experiment.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
          • relation
            includes
            or
            excludes
            or
            between
            required

            Defines operators for filtering values.

            INCLUDES matches when the value matches any of the provided values, including null if explicitly specified. This is equivalent to NOT(EXCLUDES(values)).

            EXCLUDES matches when the value does not match any of the provided values, including null if explicitly specified. If null is not explicitly excluded, we include nulls in the result.

            BETWEEN matches when the value is between the two provided bounds (inclusive). Either bound can be null for an open-ended range. A null value matches only if a 3rd value of null is provided. BETWEEN applies to numeric, date and timestamp values.

          • value
            FilterValueTypes
            required

            Any of the following:

            • array<unknown>

              Any of the following:

              • integer
              • null
            • array<unknown>

              Any of the following:

              • number
              • null
            • array<any of>

              Any of the following:

              • string
              • null
            • array<any of>

              Any of the following:

              • boolean
              • null
        • experiment_type
          optional
          string
          default: "freq_online"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • desired_n
          optional

          Desired number of individual participants. Required when creating individual-randomized preassigned experiments. For cluster-randomized preassigned experiment creation, use desired_n_clusters to control sampling; desired_n remains available for power calculations but is superseded by desired_n_clusters when both are set. When set, the power check also returns the minimum detectable effect for this size, along with the minimum sample size.

          Any of the following:

          • integer
          • null
        • desired_ns
          optional

          Optional list of desired individual participant sample sizes. The power check returns the minimum detectable effect for each size (MetricPowerAnalysis.mde_curve), letting clients plot an MDE-vs-sample-size power curve from a single request. Superseded by desired_ns_clusters when both are set. Ignored when creating an experiment.

          Any of the following:

          • array<integer>
          • null
        • power
          optional
          number

          The chance of detecting a real effect when one truly exists (1 - the false negative rate).

          maximum: 1 · minimum: 0 · default: 0.8
        • alpha
          optional
          number

          The chance of a false positive: concluding there is an effect when there is none.

          maximum: 1 · minimum: 0 · default: 0.05
        • fstat_thresh
          optional
          number

          The p-value threshold for the balance check. When the balance check's p-value is above this threshold, we treat the assignment as "balanced".

          maximum: 1 · minimum: 0 · default: 0.6
      • PreassignedFrequentistExperimentSpec
        freq_preassigned
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
        • table_name
          string
          required

          Data source table used to resolve participant field metadata.

          maxLength: 100
        • primary_key
          string
          required

          Column name in table_name that uniquely identifies each participant.

          maxLength: 300 · minLength: 1
        • strata
          array of
          object
          required

          Optional fields to use for stratified assignment.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
        • metrics
          array of
          object
          required

          Primary and optional secondary metrics to target.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
          • metric_pct_change
            optional

            The smallest percent change, relative to metric_baseline, that you want to be able to detect. Cannot be set together with metric_target.

            Any of the following:

            • number
            • null
          • metric_target
            optional

            The absolute value you want to be able to detect. Cannot be set together with metric_pct_change.

            Any of the following:

            • number
            • null
          • icc
            optional

            Intracluster correlation coefficient for cluster-randomized designs.

            Any of the following:

            • number
            • null
          • avg_cluster_size
            optional

            Average number of participants per cluster.

            Any of the following:

            • number
            • null
          • cv
            optional

            Coefficient of variation in cluster sizes (0 = equal sizes).

            Any of the following:

            • number
            • null
          • metric_type
            optional

            Inferred from the data warehouse column type.

            Any of the following:

            • MetricType
            • null
          • metric_baseline
            optional

            Mean of the tracked metric.

            Any of the following:

            • number
            • null
          • metric_stddev
            optional

            Standard deviation is set only for MetricType.NUMERIC metrics. Must be set for numeric metrics when available_n > 0.

            Any of the following:

            • number
            • null
          • available_nonnull_n
            optional

            The number of participants who meet the filtering criteria and have a non-null value for this metric.

            Any of the following:

            • integer
            • null
          • available_n
            optional

            The number of participants who meet the filtering criteria, whether or not this metric's value is NULL. Note: assignments are made from the available_n population. When available_n is greater than available_nonnull_n, some assigned participants may have a missing value for this metric.

            Any of the following:

            • integer
            • null
          • use_one_time_metric
            optional
            boolean

            Enable one-time metric mode: only assign participants with null values for this metric. Only allowed on the primary (first) metric of an individually-randomized preassigned experiment.

            default: false
        • filters
          array of
          object
          required

          Optional filters that narrow the eligible audience to the participants you want in the experiment.

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
          • relation
            includes
            or
            excludes
            or
            between
            required

            Defines operators for filtering values.

            INCLUDES matches when the value matches any of the provided values, including null if explicitly specified. This is equivalent to NOT(EXCLUDES(values)).

            EXCLUDES matches when the value does not match any of the provided values, including null if explicitly specified. If null is not explicitly excluded, we include nulls in the result.

            BETWEEN matches when the value is between the two provided bounds (inclusive). Either bound can be null for an open-ended range. A null value matches only if a 3rd value of null is provided. BETWEEN applies to numeric, date and timestamp values.

          • value
            FilterValueTypes
            required

            Any of the following:

            • array<unknown>

              Any of the following:

              • integer
              • null
            • array<unknown>

              Any of the following:

              • number
              • null
            • array<any of>

              Any of the following:

              • string
              • null
            • array<any of>

              Any of the following:

              • boolean
              • null
        • experiment_type
          optional
          string
          default: "freq_preassigned"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • desired_n
          optional

          Desired number of individual participants. Required when creating individual-randomized preassigned experiments. For cluster-randomized preassigned experiment creation, use desired_n_clusters to control sampling; desired_n remains available for power calculations but is superseded by desired_n_clusters when both are set. When set, the power check also returns the minimum detectable effect for this size, along with the minimum sample size.

          Any of the following:

          • integer
          • null
        • desired_ns
          optional

          Optional list of desired individual participant sample sizes. The power check returns the minimum detectable effect for each size (MetricPowerAnalysis.mde_curve), letting clients plot an MDE-vs-sample-size power curve from a single request. Superseded by desired_ns_clusters when both are set. Ignored when creating an experiment.

          Any of the following:

          • array<integer>
          • null
        • power
          optional
          number

          The chance of detecting a real effect when one truly exists (1 - the false negative rate).

          maximum: 1 · minimum: 0 · default: 0.8
        • alpha
          optional
          number

          The chance of a false positive: concluding there is an effect when there is none.

          maximum: 1 · minimum: 0 · default: 0.05
        • fstat_thresh
          optional
          number

          The p-value threshold for the balance check. When the balance check's p-value is above this threshold, we treat the assignment as "balanced".

          maximum: 1 · minimum: 0 · default: 0.6
        • cluster_key
          optional

          Column name in table_name that identifies clusters for a cluster-randomized design. When set, per-metric icc, avg_cluster_size, and cv are either supplied on each metric or computed from this column at power_check time. When None, the design is assumed to be individual-randomized.

          Any of the following:

          • string
          • null
        • desired_n_clusters
          optional

          Desired number of clusters to sample when creating a cluster-randomized preassigned experiment. Required and only valid when cluster_key is set. All eligible participants in each sampled cluster are included. In power checks, the minimum detectable effect is computed for this many clusters, converted to a per-metric sample size using each metric's avg_cluster_size; takes precedence over desired_n.

          Any of the following:

          • integer
          • null
        • desired_ns_clusters
          optional

          Optional list of desired cluster counts. Only valid when cluster_key is set. The power check returns the minimum detectable effect for each count (MetricPowerAnalysis.mde_curve), converted to a per-metric sample size using each metric's avg_cluster_size; takes precedence over desired_ns. Ignored when creating an experiment.

          Any of the following:

          • array<integer>
          • null
    • AnyBanditDesignSpec
      mab_online

      One of the following:

      • CMABExperimentSpec
        cmab_online
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
          • alpha_init
            optional

            Initial alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta_init
            optional

            Initial beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu_init
            optional

            Initial mean parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • sigma_init
            optional

            Initial standard deviation parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • alpha
            optional

            Updated alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta
            optional

            Updated beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu
            optional

            Updated mean vector for the Normal prior.

            Any of the following:

            • array<number>
            • null
          • covariance
            optional

            Updated covariance matrix for the Normal prior.

            Any of the following:

            • array<array<number>>
            • null
        • experiment_type
          optional
          string
          default: "cmab_online"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • contexts
          optional

          Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

          Any of the following:

          • array<unknown>
            • context_name
              string
              required
              maxLength: 100
            • context_id
              optional

              Unique identifier for the context. Do not set this when you create a new context.

              Any of the following:

              • string
              • null
            • context_description
              optional

              Any of the following:

              • string
              • null
            • value_type
              optional
              binary
              or
              real-valued

              Type of value the context can take.

              default: "binary"
          • null
        • prior_type
          optional
          beta
          or
          normal

          The type of prior distribution for the arms.

          default: "beta"
        • reward_type
          optional
          binary
          or
          real-valued

          The type of outcome observed from the experiment.

          default: "binary"
        • enable_autofail
          optional
          boolean

          When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

          default: false
        • autofail_window
          optional
          integer

          The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

          minimum: 1 · default: 24
        • autofail_outcome_value
          optional
          number

          The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

          default: 0
      • MABExperimentSpec
        mab_online
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
          • alpha_init
            optional

            Initial alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta_init
            optional

            Initial beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu_init
            optional

            Initial mean parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • sigma_init
            optional

            Initial standard deviation parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • alpha
            optional

            Updated alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta
            optional

            Updated beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu
            optional

            Updated mean vector for the Normal prior.

            Any of the following:

            • array<number>
            • null
          • covariance
            optional

            Updated covariance matrix for the Normal prior.

            Any of the following:

            • array<array<number>>
            • null
        • experiment_type
          optional
          string
          default: "mab_online"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • contexts
          optional

          Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

          Any of the following:

          • array<unknown>
            • context_name
              string
              required
              maxLength: 100
            • context_id
              optional

              Unique identifier for the context. Do not set this when you create a new context.

              Any of the following:

              • string
              • null
            • context_description
              optional

              Any of the following:

              • string
              • null
            • value_type
              optional
              binary
              or
              real-valued

              Type of value the context can take.

              default: "binary"
          • null
        • prior_type
          optional
          beta
          or
          normal

          The type of prior distribution for the arms.

          default: "beta"
        • reward_type
          optional
          binary
          or
          real-valued

          The type of outcome observed from the experiment.

          default: "binary"
        • enable_autofail
          optional
          boolean

          When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

          default: false
        • autofail_window
          optional
          integer

          The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

          minimum: 1 · default: 24
        • autofail_outcome_value
          optional
          number

          The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

          default: 0
      • MABDwhExperimentSpec
        mab_online_dwh
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
          • alpha_init
            optional

            Initial alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta_init
            optional

            Initial beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu_init
            optional

            Initial mean parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • sigma_init
            optional

            Initial standard deviation parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • alpha
            optional

            Updated alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta
            optional

            Updated beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu
            optional

            Updated mean vector for the Normal prior.

            Any of the following:

            • array<number>
            • null
          • covariance
            optional

            Updated covariance matrix for the Normal prior.

            Any of the following:

            • array<array<number>>
            • null
        • table_name
          string
          required

          Datasource table used to resolve target field metadata.

          maxLength: 100
        • primary_key
          string
          required

          Column name in table_name that uniquely identifies each participant.

          maxLength: 300 · minLength: 1
        • target_field_name
          string
          required

          Column name in table_name whose values this bandit optimises. Its data type is resolved from the DWH at create-time and used to validate subsequent outcome reports.

          maxLength: 300 · minLength: 1
        • experiment_type
          optional
          string
          default: "mab_online_dwh"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • contexts
          optional

          Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

          Any of the following:

          • array<unknown>
            • context_name
              string
              required
              maxLength: 100
            • context_id
              optional

              Unique identifier for the context. Do not set this when you create a new context.

              Any of the following:

              • string
              • null
            • context_description
              optional

              Any of the following:

              • string
              • null
            • value_type
              optional
              binary
              or
              real-valued

              Type of value the context can take.

              default: "binary"
          • null
        • prior_type
          optional
          beta
          or
          normal

          The type of prior distribution for the arms.

          default: "beta"
        • reward_type
          optional
          binary
          or
          real-valued

          The type of outcome observed from the experiment.

          default: "binary"
        • enable_autofail
          optional
          boolean

          When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

          default: false
        • autofail_window
          optional
          integer

          The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

          minimum: 1 · default: 24
        • autofail_outcome_value
          optional
          number

          The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

          default: 0
    • AnyBanditDesignSpec
      mab_online_dwh

      One of the following:

      • CMABExperimentSpec
        cmab_online
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
          • alpha_init
            optional

            Initial alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta_init
            optional

            Initial beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu_init
            optional

            Initial mean parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • sigma_init
            optional

            Initial standard deviation parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • alpha
            optional

            Updated alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta
            optional

            Updated beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu
            optional

            Updated mean vector for the Normal prior.

            Any of the following:

            • array<number>
            • null
          • covariance
            optional

            Updated covariance matrix for the Normal prior.

            Any of the following:

            • array<array<number>>
            • null
        • experiment_type
          optional
          string
          default: "cmab_online"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • contexts
          optional

          Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

          Any of the following:

          • array<unknown>
            • context_name
              string
              required
              maxLength: 100
            • context_id
              optional

              Unique identifier for the context. Do not set this when you create a new context.

              Any of the following:

              • string
              • null
            • context_description
              optional

              Any of the following:

              • string
              • null
            • value_type
              optional
              binary
              or
              real-valued

              Type of value the context can take.

              default: "binary"
          • null
        • prior_type
          optional
          beta
          or
          normal

          The type of prior distribution for the arms.

          default: "beta"
        • reward_type
          optional
          binary
          or
          real-valued

          The type of outcome observed from the experiment.

          default: "binary"
        • enable_autofail
          optional
          boolean

          When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

          default: false
        • autofail_window
          optional
          integer

          The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

          minimum: 1 · default: 24
        • autofail_outcome_value
          optional
          number

          The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

          default: 0
      • MABExperimentSpec
        mab_online
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
          • alpha_init
            optional

            Initial alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta_init
            optional

            Initial beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu_init
            optional

            Initial mean parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • sigma_init
            optional

            Initial standard deviation parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • alpha
            optional

            Updated alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta
            optional

            Updated beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu
            optional

            Updated mean vector for the Normal prior.

            Any of the following:

            • array<number>
            • null
          • covariance
            optional

            Updated covariance matrix for the Normal prior.

            Any of the following:

            • array<array<number>>
            • null
        • experiment_type
          optional
          string
          default: "mab_online"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • contexts
          optional

          Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

          Any of the following:

          • array<unknown>
            • context_name
              string
              required
              maxLength: 100
            • context_id
              optional

              Unique identifier for the context. Do not set this when you create a new context.

              Any of the following:

              • string
              • null
            • context_description
              optional

              Any of the following:

              • string
              • null
            • value_type
              optional
              binary
              or
              real-valued

              Type of value the context can take.

              default: "binary"
          • null
        • prior_type
          optional
          beta
          or
          normal

          The type of prior distribution for the arms.

          default: "beta"
        • reward_type
          optional
          binary
          or
          real-valued

          The type of outcome observed from the experiment.

          default: "binary"
        • enable_autofail
          optional
          boolean

          When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

          default: false
        • autofail_window
          optional
          integer

          The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

          minimum: 1 · default: 24
        • autofail_outcome_value
          optional
          number

          The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

          default: 0
      • MABDwhExperimentSpec
        mab_online_dwh
        • experiment_name
          string
          required
          maxLength: 100
        • description
          string
          required
          maxLength: 2000
        • start_date
          string
          required
          format: date-time
        • end_date
          string
          required
          format: date-time
        • arms
          array of
          object
          required
          • arm_name
            string
            required
            maxLength: 100
          • arm_id
            optional

            Unique ID of the arm. Leave unset when creating an arm.

            Any of the following:

            • string
            • null
          • arm_description
            optional

            Any of the following:

            • string
            • null
          • arm_weight
            optional

            Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

            Any of the following:

            • ArmWeight
            • null
          • alpha_init
            optional

            Initial alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta_init
            optional

            Initial beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu_init
            optional

            Initial mean parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • sigma_init
            optional

            Initial standard deviation parameter for the Normal prior.

            Any of the following:

            • number
            • null
          • alpha
            optional

            Updated alpha parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • beta
            optional

            Updated beta parameter for the Beta prior.

            Any of the following:

            • number
            • null
          • mu
            optional

            Updated mean vector for the Normal prior.

            Any of the following:

            • array<number>
            • null
          • covariance
            optional

            Updated covariance matrix for the Normal prior.

            Any of the following:

            • array<array<number>>
            • null
        • table_name
          string
          required

          Datasource table used to resolve target field metadata.

          maxLength: 100
        • primary_key
          string
          required

          Column name in table_name that uniquely identifies each participant.

          maxLength: 300 · minLength: 1
        • target_field_name
          string
          required

          Column name in table_name whose values this bandit optimises. Its data type is resolved from the DWH at create-time and used to validate subsequent outcome reports.

          maxLength: 300 · minLength: 1
        • experiment_type
          optional
          string
          default: "mab_online_dwh"
        • design_url
          optional

          Optional URL to a more detailed experiment design doc.

          Any of the following:

          • string
          • null
        • contexts
          optional

          Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

          Any of the following:

          • array<unknown>
            • context_name
              string
              required
              maxLength: 100
            • context_id
              optional

              Unique identifier for the context. Do not set this when you create a new context.

              Any of the following:

              • string
              • null
            • context_description
              optional

              Any of the following:

              • string
              • null
            • value_type
              optional
              binary
              or
              real-valued

              Type of value the context can take.

              default: "binary"
          • null
        • prior_type
          optional
          beta
          or
          normal

          The type of prior distribution for the arms.

          default: "beta"
        • reward_type
          optional
          binary
          or
          real-valued

          The type of outcome observed from the experiment.

          default: "binary"
        • enable_autofail
          optional
          boolean

          When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

          default: false
        • autofail_window
          optional
          integer

          The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

          minimum: 1 · default: 24
        • autofail_outcome_value
          optional
          number

          The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

          default: 0
  • power_analyses
    required

    Any of the following:

    • PowerResponse
      • analyses
        array of
        object
        required
        • metric_spec
          object
          required

          Defines a metric to measure in an experiment with its baseline stats.

          Adds the fields only the server can derive from the data, so request and response metrics stay separate types (and separate OpenAPI schemas).

          • field_name
            string
            required
            maxLength: 300 · minLength: 1
          • metric_pct_change
            optional

            Percent change target relative to the metric_baseline.

            Any of the following:

            • number
            • null
          • metric_target
            optional

            Absolute target value = metric_baseline * (1 + metric_pct_change).

            Any of the following:

            • number
            • null
          • icc
            optional

            Intracluster correlation coefficient for cluster-randomized designs.

            Any of the following:

            • number
            • null
          • avg_cluster_size
            optional

            Average number of participants per cluster.

            Any of the following:

            • number
            • null
          • cv
            optional

            Coefficient of variation in cluster sizes (0 = equal sizes).

            Any of the following:

            • number
            • null
          • metric_type
            optional

            Inferred from the data warehouse column type.

            Any of the following:

            • MetricType
            • null
          • metric_baseline
            optional

            Mean of the tracked metric.

            Any of the following:

            • number
            • null
          • metric_stddev
            optional

            Standard deviation is set only for MetricType.NUMERIC metrics. Must be set for numeric metrics when available_n > 0.

            Any of the following:

            • number
            • null
          • available_nonnull_n
            optional

            The number of participants who meet the filtering criteria and have a non-null value for this metric.

            Any of the following:

            • integer
            • null
          • available_n
            optional

            The number of participants who meet the filtering criteria, whether or not this metric's value is NULL. Note: assignments are made from the available_n population. When available_n is greater than available_nonnull_n, some assigned participants may have a missing value for this metric.

            Any of the following:

            • integer
            • null
          • use_one_time_metric
            optional
            boolean

            Enable one-time metric mode: only assign participants with null values for this metric. Only allowed on the primary (first) metric of an individually-randomized preassigned experiment.

            default: false
          • is_one_time_eligible
            optional
            boolean

            Set by the power check: True if the metric is eligible for one-time mode (it is the primary metric and its column has both null and non-null values).

            default: false
        • target_n
          optional

          Minimum sample size needed to meet the design specs.

          Any of the following:

          • integer
          • null
        • sufficient_n
          optional

          Whether there are enough available participants to sample to reach target_n.

          Any of the following:

          • boolean
          • null
        • target_possible
          optional

          When the sample size is too small to reach the desired metric_target, this is the target that is possible given available_n, in the improvement (positive) direction from the baseline. It is the absolute counterpart of pct_change_possible. None when the sample size is large enough to detect the desired change, or (for BINARY metrics) when no value above the baseline is a large enough change to reach the requested power.

          Any of the following:

          • number
          • null
        • pct_change_possible
          optional

          When the sample size is too small to reach the desired metric_pct_change, this is the percent change that is possible given available_n. It is the relative counterpart of target_possible. None whenever target_possible is None.

          Any of the following:

          • number
          • null
        • target_possible_lower
          optional

          The detectable metric value below the baseline. (target_possible is the metric value above the baseline.) Both detectable metric values above and below the baseline use the chosen confidence and power, and are always reported together, regardless of the direction of the requested change. For numeric metrics, this is the same distance below the baseline as target_possible is above it. However, for binary metrics, an increase and a decrease that are equally detectable can sit at different distances from the baseline. None when the sample size is large enough to detect the desired change, or (for BINARY metrics) when no value below the baseline is a large enough change to reach the requested power.

          Any of the following:

          • number
          • null
        • pct_change_possible_lower
          optional

          The relative counterpart of target_possible_lower, as a percent change from the baseline (negative for a decrease). A value of -0.10 means a 10% decrease. None whenever target_possible_lower is None.

          Any of the following:

          • number
          • null
        • pct_change_with_desired_n
          optional

          The minimum detectable effect (MDE) achievable for the desired sample size at the chosen confidence and power, reported in the direction of the requested change (negative when the requested change is a decrease from the baseline). Present only when design_spec.desired_n or design_spec.desired_n_clusters is set (frequentist design specs). When desired_n_clusters is set, the desired sample size is desired_n_clusters times this metric's avg_cluster_size.

          Any of the following:

          • number
          • null
        • mde_curve
          optional

          The MDE for each requested sample size, in the same order as design_spec.desired_ns (or desired_ns_clusters, which takes precedence). Present only when one of those is set. Each point is computed best-effort: a size where the calculation fails yields a null pct_change instead of failing the request.

          Any of the following:

          • array<unknown>
            • desired_n
              integer
              required

              Sample size in individual participants used for this point. For cluster-randomized designs this is desired_n_clusters times the metric's avg_cluster_size.

            • desired_n_clusters
              optional

              The requested number of clusters for this point. None for individual-randomized requests.

              Any of the following:

              • integer
              • null
            • pct_change
              optional

              The minimum detectable effect (MDE) at this sample size, as a percent change relative to metric_baseline. None when the power calculation fails at this size (e.g. too small to solve).

              Any of the following:

              • number
              • null
          • null
        • msg
          optional

          A human-friendly summary of the power analysis results.

          Any of the following:

          • MetricPowerAnalysisMessage
          • null
        • num_clusters_total
          optional

          Total number of clusters needed across all arms.

          Any of the following:

          • integer
          • null
        • clusters_per_arm
          optional

          Number of clusters needed for each arm (one entry per arm).

          Any of the following:

          • array<integer>
          • null
        • n_per_arm
          optional

          Number of participants for each arm (one entry per arm).

          Any of the following:

          • array<integer>
          • null
        • design_effect
          optional

          Design effect (DEFF), the clustering penalty multiplier.

          Any of the following:

          • number
          • null
        • effective_sample_size
          optional

          Effective sample size accounting for clustering (total_n / DEFF).

          Any of the following:

          • integer
          • null
    • null
  • assign_summary
    required

    Any of the following:

    • AssignSummary
      • sample_size
        integer
        required

        The number of participants across all arms in total.

      • balance_check
        optional

        Balance test results if available. Online experiments do not have balance checks.

        Any of the following:

        • BalanceCheck
        • null
      • arm_sizes
        optional

        For each arm, the number of participants (and optionally clusters) assigned.

        Any of the following:

        • array<unknown>
          • arm
            object
            required

            Describes an experiment treatment arm.

            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
          • size
            optional
            integer
            default: 0
          • cluster_count
            optional

            The number of clusters assigned to this arm. Set only on cluster-randomized experiments.

            Any of the following:

            • integer
            • null
        • null
    • null
  • spreadsheet_full_url
    optional

    Full URL of the datasource spreadsheet, including the launched experiment's outcomes tab.

    Any of the following:

    • string
    • null
  • webhooks
    optional
    array of
    string

    List of webhook IDs associated with this experiment. These webhooks are triggered when the experiment is committed.

    default: []
  • decision
    optional
    string

    Record any decision(s) made because of this experiment. Will you launch it, and if so when? Regardless of positive or negative results, how do any learnings inform next steps or future hypotheses?

    default: ""
  • impact
    optional
    high
    or
    medium
    or
    low
    or
    negative
    or
    unclear
    or

    Given the results across your tracked metrics and any other observed effects seen elsewhere, record an overall summary here. Do they agree or reject your hypotheses? Beyond the metrics tracked, did variations affect scalability, cost, feedback, or other aspects?

    default: ""

400

The request is invalid. Check that the request body matches the required structure and that all required fields and headers are present.

403

You did not provide a valid API key, or your API key does not have permission for this operation.

Tip: Check that the API key in the X-API-Key header has access to the data source or experiment.

404

The resource was not found, or you do not have access to it.

Tip: Check that the API key in the X-API-Key header has access to the data source or experiment.

422

Validation Error
JSON
  • detail
    optional
    array of
    object
    • loc
      array of
      required

      Any of the following:

      • string
      • integer
    • msg
      string
      required
    • type
      string
      required
    • input
      optional
      unknown
    • ctx
      optional
      object

Type to search…

↑↓ navigate↵ selectEsc close