Skip to content
Navigation

List experiments on a data source

GET
https://api.evidential.dev/v1/experiments

Authorization

  • DatasourceApiKey
    API key in header
    required

    X-API-Key: <token>

Header parameters

  • Datasource-ID
    string
    required

    The ID of the datasource to operate on.

    example: "testing"

Returns

200

Successful Response

JSON
  • items
    array of
    object
    required
    • experiment_id
      string
      required

      Server-generated ID of the experiment.

    • datasource_id
      string
      required
    • state
      assigned
      or
      abandoned
      or
      committed
      required

      Current state of this experiment.

    • stopped_assignments_at
      required

      The date and time when assignments were stopped. Null if the experiment is still accepting assignments. This date can change if the experiment's end date is modified.

      Any of the following:

      • string
      • null
    • stopped_assignments_reason
      required

      Any of the following:

      • StopAssignmentReason
      • null
    • design_spec
      DesignSpec
      required

      The type of assignment and experiment design.

      One of the following:

      • AnyBanditDesignSpec
        cmab_online

        One of the following:

        • CMABExperimentSpec
          cmab_online
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
            • alpha_init
              optional

              Initial alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta_init
              optional

              Initial beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu_init
              optional

              Initial mean parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • sigma_init
              optional

              Initial standard deviation parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • alpha
              optional

              Updated alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta
              optional

              Updated beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu
              optional

              Updated mean vector for the Normal prior.

              Any of the following:

              • array<number>
              • null
            • covariance
              optional

              Updated covariance matrix for the Normal prior.

              Any of the following:

              • array<array<number>>
              • null
          • experiment_type
            optional
            string
            default: "cmab_online"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • contexts
            optional

            Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

            Any of the following:

            • array<unknown>
              • context_name
                string
                required
                maxLength: 100
              • context_id
                optional

                Unique identifier for the context. Do not set this when you create a new context.

                Any of the following:

                • string
                • null
              • context_description
                optional

                Any of the following:

                • string
                • null
              • value_type
                optional
                binary
                or
                real-valued

                Type of value the context can take.

                default: "binary"
            • null
          • prior_type
            optional
            beta
            or
            normal

            The type of prior distribution for the arms.

            default: "beta"
          • reward_type
            optional
            binary
            or
            real-valued

            The type of outcome observed from the experiment.

            default: "binary"
          • enable_autofail
            optional
            boolean

            When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

            default: false
          • autofail_window
            optional
            integer

            The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

            minimum: 1 · default: 24
          • autofail_outcome_value
            optional
            number

            The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

            default: 0
        • MABExperimentSpec
          mab_online
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
            • alpha_init
              optional

              Initial alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta_init
              optional

              Initial beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu_init
              optional

              Initial mean parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • sigma_init
              optional

              Initial standard deviation parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • alpha
              optional

              Updated alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta
              optional

              Updated beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu
              optional

              Updated mean vector for the Normal prior.

              Any of the following:

              • array<number>
              • null
            • covariance
              optional

              Updated covariance matrix for the Normal prior.

              Any of the following:

              • array<array<number>>
              • null
          • experiment_type
            optional
            string
            default: "mab_online"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • contexts
            optional

            Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

            Any of the following:

            • array<unknown>
              • context_name
                string
                required
                maxLength: 100
              • context_id
                optional

                Unique identifier for the context. Do not set this when you create a new context.

                Any of the following:

                • string
                • null
              • context_description
                optional

                Any of the following:

                • string
                • null
              • value_type
                optional
                binary
                or
                real-valued

                Type of value the context can take.

                default: "binary"
            • null
          • prior_type
            optional
            beta
            or
            normal

            The type of prior distribution for the arms.

            default: "beta"
          • reward_type
            optional
            binary
            or
            real-valued

            The type of outcome observed from the experiment.

            default: "binary"
          • enable_autofail
            optional
            boolean

            When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

            default: false
          • autofail_window
            optional
            integer

            The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

            minimum: 1 · default: 24
          • autofail_outcome_value
            optional
            number

            The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

            default: 0
        • MABDwhExperimentSpec
          mab_online_dwh
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
            • alpha_init
              optional

              Initial alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta_init
              optional

              Initial beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu_init
              optional

              Initial mean parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • sigma_init
              optional

              Initial standard deviation parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • alpha
              optional

              Updated alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta
              optional

              Updated beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu
              optional

              Updated mean vector for the Normal prior.

              Any of the following:

              • array<number>
              • null
            • covariance
              optional

              Updated covariance matrix for the Normal prior.

              Any of the following:

              • array<array<number>>
              • null
          • table_name
            string
            required

            Datasource table used to resolve target field metadata.

            maxLength: 100
          • primary_key
            string
            required

            Column name in table_name that uniquely identifies each participant.

            maxLength: 300 · minLength: 1
          • target_field_name
            string
            required

            Column name in table_name whose values this bandit optimises. Its data type is resolved from the DWH at create-time and used to validate subsequent outcome reports.

            maxLength: 300 · minLength: 1
          • experiment_type
            optional
            string
            default: "mab_online_dwh"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • contexts
            optional

            Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

            Any of the following:

            • array<unknown>
              • context_name
                string
                required
                maxLength: 100
              • context_id
                optional

                Unique identifier for the context. Do not set this when you create a new context.

                Any of the following:

                • string
                • null
              • context_description
                optional

                Any of the following:

                • string
                • null
              • value_type
                optional
                binary
                or
                real-valued

                Type of value the context can take.

                default: "binary"
            • null
          • prior_type
            optional
            beta
            or
            normal

            The type of prior distribution for the arms.

            default: "beta"
          • reward_type
            optional
            binary
            or
            real-valued

            The type of outcome observed from the experiment.

            default: "binary"
          • enable_autofail
            optional
            boolean

            When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

            default: false
          • autofail_window
            optional
            integer

            The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

            minimum: 1 · default: 24
          • autofail_outcome_value
            optional
            number

            The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

            default: 0
      • AnyFrequentistDesignSpec
        freq_online

        One of the following:

        • OnlineFrequentistExperimentSpec
          freq_online
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
          • table_name
            string
            required

            Data source table used to resolve participant field metadata.

            maxLength: 100
          • primary_key
            string
            required

            Column name in table_name that uniquely identifies each participant.

            maxLength: 300 · minLength: 1
          • strata
            array of
            object
            required

            Optional fields to use for stratified assignment.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
          • metrics
            array of
            object
            required

            Primary and optional secondary metrics to target.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
            • metric_pct_change
              optional

              The smallest percent change, relative to metric_baseline, that you want to be able to detect. Cannot be set together with metric_target.

              Any of the following:

              • number
              • null
            • metric_target
              optional

              The absolute value you want to be able to detect. Cannot be set together with metric_pct_change.

              Any of the following:

              • number
              • null
            • icc
              optional

              Intracluster correlation coefficient for cluster-randomized designs.

              Any of the following:

              • number
              • null
            • avg_cluster_size
              optional

              Average number of participants per cluster.

              Any of the following:

              • number
              • null
            • cv
              optional

              Coefficient of variation in cluster sizes (0 = equal sizes).

              Any of the following:

              • number
              • null
            • metric_type
              optional

              Inferred from the data warehouse column type.

              Any of the following:

              • MetricType
              • null
            • metric_baseline
              optional

              Mean of the tracked metric.

              Any of the following:

              • number
              • null
            • metric_stddev
              optional

              Standard deviation is set only for MetricType.NUMERIC metrics. Must be set for numeric metrics when available_n > 0.

              Any of the following:

              • number
              • null
            • available_nonnull_n
              optional

              The number of participants who meet the filtering criteria and have a non-null value for this metric.

              Any of the following:

              • integer
              • null
            • available_n
              optional

              The number of participants who meet the filtering criteria, whether or not this metric's value is NULL. Note: assignments are made from the available_n population. When available_n is greater than available_nonnull_n, some assigned participants may have a missing value for this metric.

              Any of the following:

              • integer
              • null
            • use_one_time_metric
              optional
              boolean

              Enable one-time metric mode: only assign participants with null values for this metric. Only allowed on the primary (first) metric of an individually-randomized preassigned experiment.

              default: false
          • filters
            array of
            object
            required

            Optional filters that narrow the eligible audience to the participants you want in the experiment.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
            • relation
              includes
              or
              excludes
              or
              between
              required

              Defines operators for filtering values.

              INCLUDES matches when the value matches any of the provided values, including null if explicitly specified. This is equivalent to NOT(EXCLUDES(values)).

              EXCLUDES matches when the value does not match any of the provided values, including null if explicitly specified. If null is not explicitly excluded, we include nulls in the result.

              BETWEEN matches when the value is between the two provided bounds (inclusive). Either bound can be null for an open-ended range. A null value matches only if a 3rd value of null is provided. BETWEEN applies to numeric, date and timestamp values.

            • value
              FilterValueTypes
              required

              Any of the following:

              • array<unknown>

                Any of the following:

                • integer
                • null
              • array<unknown>

                Any of the following:

                • number
                • null
              • array<any of>

                Any of the following:

                • string
                • null
              • array<any of>

                Any of the following:

                • boolean
                • null
          • experiment_type
            optional
            string
            default: "freq_online"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • desired_n
            optional

            Desired number of individual participants. Required when creating individual-randomized preassigned experiments. For cluster-randomized preassigned experiment creation, use desired_n_clusters to control sampling; desired_n remains available for power calculations but is superseded by desired_n_clusters when both are set. When set, the power check also returns the minimum detectable effect for this size, along with the minimum sample size.

            Any of the following:

            • integer
            • null
          • desired_ns
            optional

            Optional list of desired individual participant sample sizes. The power check returns the minimum detectable effect for each size (MetricPowerAnalysis.mde_curve), letting clients plot an MDE-vs-sample-size power curve from a single request. Superseded by desired_ns_clusters when both are set. Ignored when creating an experiment.

            Any of the following:

            • array<integer>
            • null
          • power
            optional
            number

            The chance of detecting a real effect when one truly exists (1 - the false negative rate).

            maximum: 1 · minimum: 0 · default: 0.8
          • alpha
            optional
            number

            The chance of a false positive: concluding there is an effect when there is none.

            maximum: 1 · minimum: 0 · default: 0.05
          • fstat_thresh
            optional
            number

            The p-value threshold for the balance check. When the balance check's p-value is above this threshold, we treat the assignment as "balanced".

            maximum: 1 · minimum: 0 · default: 0.6
        • PreassignedFrequentistExperimentSpec
          freq_preassigned
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
          • table_name
            string
            required

            Data source table used to resolve participant field metadata.

            maxLength: 100
          • primary_key
            string
            required

            Column name in table_name that uniquely identifies each participant.

            maxLength: 300 · minLength: 1
          • strata
            array of
            object
            required

            Optional fields to use for stratified assignment.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
          • metrics
            array of
            object
            required

            Primary and optional secondary metrics to target.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
            • metric_pct_change
              optional

              The smallest percent change, relative to metric_baseline, that you want to be able to detect. Cannot be set together with metric_target.

              Any of the following:

              • number
              • null
            • metric_target
              optional

              The absolute value you want to be able to detect. Cannot be set together with metric_pct_change.

              Any of the following:

              • number
              • null
            • icc
              optional

              Intracluster correlation coefficient for cluster-randomized designs.

              Any of the following:

              • number
              • null
            • avg_cluster_size
              optional

              Average number of participants per cluster.

              Any of the following:

              • number
              • null
            • cv
              optional

              Coefficient of variation in cluster sizes (0 = equal sizes).

              Any of the following:

              • number
              • null
            • metric_type
              optional

              Inferred from the data warehouse column type.

              Any of the following:

              • MetricType
              • null
            • metric_baseline
              optional

              Mean of the tracked metric.

              Any of the following:

              • number
              • null
            • metric_stddev
              optional

              Standard deviation is set only for MetricType.NUMERIC metrics. Must be set for numeric metrics when available_n > 0.

              Any of the following:

              • number
              • null
            • available_nonnull_n
              optional

              The number of participants who meet the filtering criteria and have a non-null value for this metric.

              Any of the following:

              • integer
              • null
            • available_n
              optional

              The number of participants who meet the filtering criteria, whether or not this metric's value is NULL. Note: assignments are made from the available_n population. When available_n is greater than available_nonnull_n, some assigned participants may have a missing value for this metric.

              Any of the following:

              • integer
              • null
            • use_one_time_metric
              optional
              boolean

              Enable one-time metric mode: only assign participants with null values for this metric. Only allowed on the primary (first) metric of an individually-randomized preassigned experiment.

              default: false
          • filters
            array of
            object
            required

            Optional filters that narrow the eligible audience to the participants you want in the experiment.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
            • relation
              includes
              or
              excludes
              or
              between
              required

              Defines operators for filtering values.

              INCLUDES matches when the value matches any of the provided values, including null if explicitly specified. This is equivalent to NOT(EXCLUDES(values)).

              EXCLUDES matches when the value does not match any of the provided values, including null if explicitly specified. If null is not explicitly excluded, we include nulls in the result.

              BETWEEN matches when the value is between the two provided bounds (inclusive). Either bound can be null for an open-ended range. A null value matches only if a 3rd value of null is provided. BETWEEN applies to numeric, date and timestamp values.

            • value
              FilterValueTypes
              required

              Any of the following:

              • array<unknown>

                Any of the following:

                • integer
                • null
              • array<unknown>

                Any of the following:

                • number
                • null
              • array<any of>

                Any of the following:

                • string
                • null
              • array<any of>

                Any of the following:

                • boolean
                • null
          • experiment_type
            optional
            string
            default: "freq_preassigned"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • desired_n
            optional

            Desired number of individual participants. Required when creating individual-randomized preassigned experiments. For cluster-randomized preassigned experiment creation, use desired_n_clusters to control sampling; desired_n remains available for power calculations but is superseded by desired_n_clusters when both are set. When set, the power check also returns the minimum detectable effect for this size, along with the minimum sample size.

            Any of the following:

            • integer
            • null
          • desired_ns
            optional

            Optional list of desired individual participant sample sizes. The power check returns the minimum detectable effect for each size (MetricPowerAnalysis.mde_curve), letting clients plot an MDE-vs-sample-size power curve from a single request. Superseded by desired_ns_clusters when both are set. Ignored when creating an experiment.

            Any of the following:

            • array<integer>
            • null
          • power
            optional
            number

            The chance of detecting a real effect when one truly exists (1 - the false negative rate).

            maximum: 1 · minimum: 0 · default: 0.8
          • alpha
            optional
            number

            The chance of a false positive: concluding there is an effect when there is none.

            maximum: 1 · minimum: 0 · default: 0.05
          • fstat_thresh
            optional
            number

            The p-value threshold for the balance check. When the balance check's p-value is above this threshold, we treat the assignment as "balanced".

            maximum: 1 · minimum: 0 · default: 0.6
          • cluster_key
            optional

            Column name in table_name that identifies clusters for a cluster-randomized design. When set, per-metric icc, avg_cluster_size, and cv are either supplied on each metric or computed from this column at power_check time. When None, the design is assumed to be individual-randomized.

            Any of the following:

            • string
            • null
          • desired_n_clusters
            optional

            Desired number of clusters to sample when creating a cluster-randomized preassigned experiment. Required and only valid when cluster_key is set. All eligible participants in each sampled cluster are included. In power checks, the minimum detectable effect is computed for this many clusters, converted to a per-metric sample size using each metric's avg_cluster_size; takes precedence over desired_n.

            Any of the following:

            • integer
            • null
          • desired_ns_clusters
            optional

            Optional list of desired cluster counts. Only valid when cluster_key is set. The power check returns the minimum detectable effect for each count (MetricPowerAnalysis.mde_curve), converted to a per-metric sample size using each metric's avg_cluster_size; takes precedence over desired_ns. Ignored when creating an experiment.

            Any of the following:

            • array<integer>
            • null
      • AnyFrequentistDesignSpec
        freq_preassigned

        One of the following:

        • OnlineFrequentistExperimentSpec
          freq_online
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
          • table_name
            string
            required

            Data source table used to resolve participant field metadata.

            maxLength: 100
          • primary_key
            string
            required

            Column name in table_name that uniquely identifies each participant.

            maxLength: 300 · minLength: 1
          • strata
            array of
            object
            required

            Optional fields to use for stratified assignment.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
          • metrics
            array of
            object
            required

            Primary and optional secondary metrics to target.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
            • metric_pct_change
              optional

              The smallest percent change, relative to metric_baseline, that you want to be able to detect. Cannot be set together with metric_target.

              Any of the following:

              • number
              • null
            • metric_target
              optional

              The absolute value you want to be able to detect. Cannot be set together with metric_pct_change.

              Any of the following:

              • number
              • null
            • icc
              optional

              Intracluster correlation coefficient for cluster-randomized designs.

              Any of the following:

              • number
              • null
            • avg_cluster_size
              optional

              Average number of participants per cluster.

              Any of the following:

              • number
              • null
            • cv
              optional

              Coefficient of variation in cluster sizes (0 = equal sizes).

              Any of the following:

              • number
              • null
            • metric_type
              optional

              Inferred from the data warehouse column type.

              Any of the following:

              • MetricType
              • null
            • metric_baseline
              optional

              Mean of the tracked metric.

              Any of the following:

              • number
              • null
            • metric_stddev
              optional

              Standard deviation is set only for MetricType.NUMERIC metrics. Must be set for numeric metrics when available_n > 0.

              Any of the following:

              • number
              • null
            • available_nonnull_n
              optional

              The number of participants who meet the filtering criteria and have a non-null value for this metric.

              Any of the following:

              • integer
              • null
            • available_n
              optional

              The number of participants who meet the filtering criteria, whether or not this metric's value is NULL. Note: assignments are made from the available_n population. When available_n is greater than available_nonnull_n, some assigned participants may have a missing value for this metric.

              Any of the following:

              • integer
              • null
            • use_one_time_metric
              optional
              boolean

              Enable one-time metric mode: only assign participants with null values for this metric. Only allowed on the primary (first) metric of an individually-randomized preassigned experiment.

              default: false
          • filters
            array of
            object
            required

            Optional filters that narrow the eligible audience to the participants you want in the experiment.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
            • relation
              includes
              or
              excludes
              or
              between
              required

              Defines operators for filtering values.

              INCLUDES matches when the value matches any of the provided values, including null if explicitly specified. This is equivalent to NOT(EXCLUDES(values)).

              EXCLUDES matches when the value does not match any of the provided values, including null if explicitly specified. If null is not explicitly excluded, we include nulls in the result.

              BETWEEN matches when the value is between the two provided bounds (inclusive). Either bound can be null for an open-ended range. A null value matches only if a 3rd value of null is provided. BETWEEN applies to numeric, date and timestamp values.

            • value
              FilterValueTypes
              required

              Any of the following:

              • array<unknown>

                Any of the following:

                • integer
                • null
              • array<unknown>

                Any of the following:

                • number
                • null
              • array<any of>

                Any of the following:

                • string
                • null
              • array<any of>

                Any of the following:

                • boolean
                • null
          • experiment_type
            optional
            string
            default: "freq_online"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • desired_n
            optional

            Desired number of individual participants. Required when creating individual-randomized preassigned experiments. For cluster-randomized preassigned experiment creation, use desired_n_clusters to control sampling; desired_n remains available for power calculations but is superseded by desired_n_clusters when both are set. When set, the power check also returns the minimum detectable effect for this size, along with the minimum sample size.

            Any of the following:

            • integer
            • null
          • desired_ns
            optional

            Optional list of desired individual participant sample sizes. The power check returns the minimum detectable effect for each size (MetricPowerAnalysis.mde_curve), letting clients plot an MDE-vs-sample-size power curve from a single request. Superseded by desired_ns_clusters when both are set. Ignored when creating an experiment.

            Any of the following:

            • array<integer>
            • null
          • power
            optional
            number

            The chance of detecting a real effect when one truly exists (1 - the false negative rate).

            maximum: 1 · minimum: 0 · default: 0.8
          • alpha
            optional
            number

            The chance of a false positive: concluding there is an effect when there is none.

            maximum: 1 · minimum: 0 · default: 0.05
          • fstat_thresh
            optional
            number

            The p-value threshold for the balance check. When the balance check's p-value is above this threshold, we treat the assignment as "balanced".

            maximum: 1 · minimum: 0 · default: 0.6
        • PreassignedFrequentistExperimentSpec
          freq_preassigned
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
          • table_name
            string
            required

            Data source table used to resolve participant field metadata.

            maxLength: 100
          • primary_key
            string
            required

            Column name in table_name that uniquely identifies each participant.

            maxLength: 300 · minLength: 1
          • strata
            array of
            object
            required

            Optional fields to use for stratified assignment.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
          • metrics
            array of
            object
            required

            Primary and optional secondary metrics to target.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
            • metric_pct_change
              optional

              The smallest percent change, relative to metric_baseline, that you want to be able to detect. Cannot be set together with metric_target.

              Any of the following:

              • number
              • null
            • metric_target
              optional

              The absolute value you want to be able to detect. Cannot be set together with metric_pct_change.

              Any of the following:

              • number
              • null
            • icc
              optional

              Intracluster correlation coefficient for cluster-randomized designs.

              Any of the following:

              • number
              • null
            • avg_cluster_size
              optional

              Average number of participants per cluster.

              Any of the following:

              • number
              • null
            • cv
              optional

              Coefficient of variation in cluster sizes (0 = equal sizes).

              Any of the following:

              • number
              • null
            • metric_type
              optional

              Inferred from the data warehouse column type.

              Any of the following:

              • MetricType
              • null
            • metric_baseline
              optional

              Mean of the tracked metric.

              Any of the following:

              • number
              • null
            • metric_stddev
              optional

              Standard deviation is set only for MetricType.NUMERIC metrics. Must be set for numeric metrics when available_n > 0.

              Any of the following:

              • number
              • null
            • available_nonnull_n
              optional

              The number of participants who meet the filtering criteria and have a non-null value for this metric.

              Any of the following:

              • integer
              • null
            • available_n
              optional

              The number of participants who meet the filtering criteria, whether or not this metric's value is NULL. Note: assignments are made from the available_n population. When available_n is greater than available_nonnull_n, some assigned participants may have a missing value for this metric.

              Any of the following:

              • integer
              • null
            • use_one_time_metric
              optional
              boolean

              Enable one-time metric mode: only assign participants with null values for this metric. Only allowed on the primary (first) metric of an individually-randomized preassigned experiment.

              default: false
          • filters
            array of
            object
            required

            Optional filters that narrow the eligible audience to the participants you want in the experiment.

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
            • relation
              includes
              or
              excludes
              or
              between
              required

              Defines operators for filtering values.

              INCLUDES matches when the value matches any of the provided values, including null if explicitly specified. This is equivalent to NOT(EXCLUDES(values)).

              EXCLUDES matches when the value does not match any of the provided values, including null if explicitly specified. If null is not explicitly excluded, we include nulls in the result.

              BETWEEN matches when the value is between the two provided bounds (inclusive). Either bound can be null for an open-ended range. A null value matches only if a 3rd value of null is provided. BETWEEN applies to numeric, date and timestamp values.

            • value
              FilterValueTypes
              required

              Any of the following:

              • array<unknown>

                Any of the following:

                • integer
                • null
              • array<unknown>

                Any of the following:

                • number
                • null
              • array<any of>

                Any of the following:

                • string
                • null
              • array<any of>

                Any of the following:

                • boolean
                • null
          • experiment_type
            optional
            string
            default: "freq_preassigned"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • desired_n
            optional

            Desired number of individual participants. Required when creating individual-randomized preassigned experiments. For cluster-randomized preassigned experiment creation, use desired_n_clusters to control sampling; desired_n remains available for power calculations but is superseded by desired_n_clusters when both are set. When set, the power check also returns the minimum detectable effect for this size, along with the minimum sample size.

            Any of the following:

            • integer
            • null
          • desired_ns
            optional

            Optional list of desired individual participant sample sizes. The power check returns the minimum detectable effect for each size (MetricPowerAnalysis.mde_curve), letting clients plot an MDE-vs-sample-size power curve from a single request. Superseded by desired_ns_clusters when both are set. Ignored when creating an experiment.

            Any of the following:

            • array<integer>
            • null
          • power
            optional
            number

            The chance of detecting a real effect when one truly exists (1 - the false negative rate).

            maximum: 1 · minimum: 0 · default: 0.8
          • alpha
            optional
            number

            The chance of a false positive: concluding there is an effect when there is none.

            maximum: 1 · minimum: 0 · default: 0.05
          • fstat_thresh
            optional
            number

            The p-value threshold for the balance check. When the balance check's p-value is above this threshold, we treat the assignment as "balanced".

            maximum: 1 · minimum: 0 · default: 0.6
          • cluster_key
            optional

            Column name in table_name that identifies clusters for a cluster-randomized design. When set, per-metric icc, avg_cluster_size, and cv are either supplied on each metric or computed from this column at power_check time. When None, the design is assumed to be individual-randomized.

            Any of the following:

            • string
            • null
          • desired_n_clusters
            optional

            Desired number of clusters to sample when creating a cluster-randomized preassigned experiment. Required and only valid when cluster_key is set. All eligible participants in each sampled cluster are included. In power checks, the minimum detectable effect is computed for this many clusters, converted to a per-metric sample size using each metric's avg_cluster_size; takes precedence over desired_n.

            Any of the following:

            • integer
            • null
          • desired_ns_clusters
            optional

            Optional list of desired cluster counts. Only valid when cluster_key is set. The power check returns the minimum detectable effect for each count (MetricPowerAnalysis.mde_curve), converted to a per-metric sample size using each metric's avg_cluster_size; takes precedence over desired_ns. Ignored when creating an experiment.

            Any of the following:

            • array<integer>
            • null
      • AnyBanditDesignSpec
        mab_online

        One of the following:

        • CMABExperimentSpec
          cmab_online
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
            • alpha_init
              optional

              Initial alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta_init
              optional

              Initial beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu_init
              optional

              Initial mean parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • sigma_init
              optional

              Initial standard deviation parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • alpha
              optional

              Updated alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta
              optional

              Updated beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu
              optional

              Updated mean vector for the Normal prior.

              Any of the following:

              • array<number>
              • null
            • covariance
              optional

              Updated covariance matrix for the Normal prior.

              Any of the following:

              • array<array<number>>
              • null
          • experiment_type
            optional
            string
            default: "cmab_online"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • contexts
            optional

            Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

            Any of the following:

            • array<unknown>
              • context_name
                string
                required
                maxLength: 100
              • context_id
                optional

                Unique identifier for the context. Do not set this when you create a new context.

                Any of the following:

                • string
                • null
              • context_description
                optional

                Any of the following:

                • string
                • null
              • value_type
                optional
                binary
                or
                real-valued

                Type of value the context can take.

                default: "binary"
            • null
          • prior_type
            optional
            beta
            or
            normal

            The type of prior distribution for the arms.

            default: "beta"
          • reward_type
            optional
            binary
            or
            real-valued

            The type of outcome observed from the experiment.

            default: "binary"
          • enable_autofail
            optional
            boolean

            When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

            default: false
          • autofail_window
            optional
            integer

            The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

            minimum: 1 · default: 24
          • autofail_outcome_value
            optional
            number

            The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

            default: 0
        • MABExperimentSpec
          mab_online
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
            • alpha_init
              optional

              Initial alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta_init
              optional

              Initial beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu_init
              optional

              Initial mean parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • sigma_init
              optional

              Initial standard deviation parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • alpha
              optional

              Updated alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta
              optional

              Updated beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu
              optional

              Updated mean vector for the Normal prior.

              Any of the following:

              • array<number>
              • null
            • covariance
              optional

              Updated covariance matrix for the Normal prior.

              Any of the following:

              • array<array<number>>
              • null
          • experiment_type
            optional
            string
            default: "mab_online"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • contexts
            optional

            Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

            Any of the following:

            • array<unknown>
              • context_name
                string
                required
                maxLength: 100
              • context_id
                optional

                Unique identifier for the context. Do not set this when you create a new context.

                Any of the following:

                • string
                • null
              • context_description
                optional

                Any of the following:

                • string
                • null
              • value_type
                optional
                binary
                or
                real-valued

                Type of value the context can take.

                default: "binary"
            • null
          • prior_type
            optional
            beta
            or
            normal

            The type of prior distribution for the arms.

            default: "beta"
          • reward_type
            optional
            binary
            or
            real-valued

            The type of outcome observed from the experiment.

            default: "binary"
          • enable_autofail
            optional
            boolean

            When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

            default: false
          • autofail_window
            optional
            integer

            The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

            minimum: 1 · default: 24
          • autofail_outcome_value
            optional
            number

            The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

            default: 0
        • MABDwhExperimentSpec
          mab_online_dwh
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
            • alpha_init
              optional

              Initial alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta_init
              optional

              Initial beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu_init
              optional

              Initial mean parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • sigma_init
              optional

              Initial standard deviation parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • alpha
              optional

              Updated alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta
              optional

              Updated beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu
              optional

              Updated mean vector for the Normal prior.

              Any of the following:

              • array<number>
              • null
            • covariance
              optional

              Updated covariance matrix for the Normal prior.

              Any of the following:

              • array<array<number>>
              • null
          • table_name
            string
            required

            Datasource table used to resolve target field metadata.

            maxLength: 100
          • primary_key
            string
            required

            Column name in table_name that uniquely identifies each participant.

            maxLength: 300 · minLength: 1
          • target_field_name
            string
            required

            Column name in table_name whose values this bandit optimises. Its data type is resolved from the DWH at create-time and used to validate subsequent outcome reports.

            maxLength: 300 · minLength: 1
          • experiment_type
            optional
            string
            default: "mab_online_dwh"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • contexts
            optional

            Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

            Any of the following:

            • array<unknown>
              • context_name
                string
                required
                maxLength: 100
              • context_id
                optional

                Unique identifier for the context. Do not set this when you create a new context.

                Any of the following:

                • string
                • null
              • context_description
                optional

                Any of the following:

                • string
                • null
              • value_type
                optional
                binary
                or
                real-valued

                Type of value the context can take.

                default: "binary"
            • null
          • prior_type
            optional
            beta
            or
            normal

            The type of prior distribution for the arms.

            default: "beta"
          • reward_type
            optional
            binary
            or
            real-valued

            The type of outcome observed from the experiment.

            default: "binary"
          • enable_autofail
            optional
            boolean

            When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

            default: false
          • autofail_window
            optional
            integer

            The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

            minimum: 1 · default: 24
          • autofail_outcome_value
            optional
            number

            The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

            default: 0
      • AnyBanditDesignSpec
        mab_online_dwh

        One of the following:

        • CMABExperimentSpec
          cmab_online
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
            • alpha_init
              optional

              Initial alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta_init
              optional

              Initial beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu_init
              optional

              Initial mean parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • sigma_init
              optional

              Initial standard deviation parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • alpha
              optional

              Updated alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta
              optional

              Updated beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu
              optional

              Updated mean vector for the Normal prior.

              Any of the following:

              • array<number>
              • null
            • covariance
              optional

              Updated covariance matrix for the Normal prior.

              Any of the following:

              • array<array<number>>
              • null
          • experiment_type
            optional
            string
            default: "cmab_online"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • contexts
            optional

            Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

            Any of the following:

            • array<unknown>
              • context_name
                string
                required
                maxLength: 100
              • context_id
                optional

                Unique identifier for the context. Do not set this when you create a new context.

                Any of the following:

                • string
                • null
              • context_description
                optional

                Any of the following:

                • string
                • null
              • value_type
                optional
                binary
                or
                real-valued

                Type of value the context can take.

                default: "binary"
            • null
          • prior_type
            optional
            beta
            or
            normal

            The type of prior distribution for the arms.

            default: "beta"
          • reward_type
            optional
            binary
            or
            real-valued

            The type of outcome observed from the experiment.

            default: "binary"
          • enable_autofail
            optional
            boolean

            When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

            default: false
          • autofail_window
            optional
            integer

            The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

            minimum: 1 · default: 24
          • autofail_outcome_value
            optional
            number

            The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

            default: 0
        • MABExperimentSpec
          mab_online
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
            • alpha_init
              optional

              Initial alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta_init
              optional

              Initial beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu_init
              optional

              Initial mean parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • sigma_init
              optional

              Initial standard deviation parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • alpha
              optional

              Updated alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta
              optional

              Updated beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu
              optional

              Updated mean vector for the Normal prior.

              Any of the following:

              • array<number>
              • null
            • covariance
              optional

              Updated covariance matrix for the Normal prior.

              Any of the following:

              • array<array<number>>
              • null
          • experiment_type
            optional
            string
            default: "mab_online"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • contexts
            optional

            Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

            Any of the following:

            • array<unknown>
              • context_name
                string
                required
                maxLength: 100
              • context_id
                optional

                Unique identifier for the context. Do not set this when you create a new context.

                Any of the following:

                • string
                • null
              • context_description
                optional

                Any of the following:

                • string
                • null
              • value_type
                optional
                binary
                or
                real-valued

                Type of value the context can take.

                default: "binary"
            • null
          • prior_type
            optional
            beta
            or
            normal

            The type of prior distribution for the arms.

            default: "beta"
          • reward_type
            optional
            binary
            or
            real-valued

            The type of outcome observed from the experiment.

            default: "binary"
          • enable_autofail
            optional
            boolean

            When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

            default: false
          • autofail_window
            optional
            integer

            The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

            minimum: 1 · default: 24
          • autofail_outcome_value
            optional
            number

            The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

            default: 0
        • MABDwhExperimentSpec
          mab_online_dwh
          • experiment_name
            string
            required
            maxLength: 100
          • description
            string
            required
            maxLength: 2000
          • start_date
            string
            required
            format: date-time
          • end_date
            string
            required
            format: date-time
          • arms
            array of
            object
            required
            • arm_name
              string
              required
              maxLength: 100
            • arm_id
              optional

              Unique ID of the arm. Leave unset when creating an arm.

              Any of the following:

              • string
              • null
            • arm_description
              optional

              Any of the following:

              • string
              • null
            • arm_weight
              optional

              Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

              Any of the following:

              • ArmWeight
              • null
            • alpha_init
              optional

              Initial alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta_init
              optional

              Initial beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu_init
              optional

              Initial mean parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • sigma_init
              optional

              Initial standard deviation parameter for the Normal prior.

              Any of the following:

              • number
              • null
            • alpha
              optional

              Updated alpha parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • beta
              optional

              Updated beta parameter for the Beta prior.

              Any of the following:

              • number
              • null
            • mu
              optional

              Updated mean vector for the Normal prior.

              Any of the following:

              • array<number>
              • null
            • covariance
              optional

              Updated covariance matrix for the Normal prior.

              Any of the following:

              • array<array<number>>
              • null
          • table_name
            string
            required

            Datasource table used to resolve target field metadata.

            maxLength: 100
          • primary_key
            string
            required

            Column name in table_name that uniquely identifies each participant.

            maxLength: 300 · minLength: 1
          • target_field_name
            string
            required

            Column name in table_name whose values this bandit optimises. Its data type is resolved from the DWH at create-time and used to validate subsequent outcome reports.

            maxLength: 300 · minLength: 1
          • experiment_type
            optional
            string
            default: "mab_online_dwh"
          • design_url
            optional

            Optional URL to a more detailed experiment design doc.

            Any of the following:

            • string
            • null
          • contexts
            optional

            Optional list of contexts that can be used to condition the assignment. Required for Contextual Multi-armed Bandit (CMAB) experiments.

            Any of the following:

            • array<unknown>
              • context_name
                string
                required
                maxLength: 100
              • context_id
                optional

                Unique identifier for the context. Do not set this when you create a new context.

                Any of the following:

                • string
                • null
              • context_description
                optional

                Any of the following:

                • string
                • null
              • value_type
                optional
                binary
                or
                real-valued

                Type of value the context can take.

                default: "binary"
            • null
          • prior_type
            optional
            beta
            or
            normal

            The type of prior distribution for the arms.

            default: "beta"
          • reward_type
            optional
            binary
            or
            real-valued

            The type of outcome observed from the experiment.

            default: "binary"
          • enable_autofail
            optional
            boolean

            When true, Evidential records a failure for participants whose outcome does not arrive (via either push or pull) within a specified time window.

            default: false
          • autofail_window
            optional
            integer

            The time window in hours after which a participant is considered to have failed if no outcome arrives (via either push or pull). Default is 24 hours. Required if enable_autofail is true.

            minimum: 1 · default: 24
          • autofail_outcome_value
            optional
            number

            The outcome value to assign to participants who are automatically failed. Required if enable_autofail is true.

            default: 0
    • power_analyses
      required

      Any of the following:

      • PowerResponse
        • analyses
          array of
          object
          required
          • metric_spec
            object
            required

            Defines a metric to measure in an experiment with its baseline stats.

            Adds the fields only the server can derive from the data, so request and response metrics stay separate types (and separate OpenAPI schemas).

            • field_name
              string
              required
              maxLength: 300 · minLength: 1
            • metric_pct_change
              optional

              Percent change target relative to the metric_baseline.

              Any of the following:

              • number
              • null
            • metric_target
              optional

              Absolute target value = metric_baseline * (1 + metric_pct_change).

              Any of the following:

              • number
              • null
            • icc
              optional

              Intracluster correlation coefficient for cluster-randomized designs.

              Any of the following:

              • number
              • null
            • avg_cluster_size
              optional

              Average number of participants per cluster.

              Any of the following:

              • number
              • null
            • cv
              optional

              Coefficient of variation in cluster sizes (0 = equal sizes).

              Any of the following:

              • number
              • null
            • metric_type
              optional

              Inferred from the data warehouse column type.

              Any of the following:

              • MetricType
              • null
            • metric_baseline
              optional

              Mean of the tracked metric.

              Any of the following:

              • number
              • null
            • metric_stddev
              optional

              Standard deviation is set only for MetricType.NUMERIC metrics. Must be set for numeric metrics when available_n > 0.

              Any of the following:

              • number
              • null
            • available_nonnull_n
              optional

              The number of participants who meet the filtering criteria and have a non-null value for this metric.

              Any of the following:

              • integer
              • null
            • available_n
              optional

              The number of participants who meet the filtering criteria, whether or not this metric's value is NULL. Note: assignments are made from the available_n population. When available_n is greater than available_nonnull_n, some assigned participants may have a missing value for this metric.

              Any of the following:

              • integer
              • null
            • use_one_time_metric
              optional
              boolean

              Enable one-time metric mode: only assign participants with null values for this metric. Only allowed on the primary (first) metric of an individually-randomized preassigned experiment.

              default: false
            • is_one_time_eligible
              optional
              boolean

              Set by the power check: True if the metric is eligible for one-time mode (it is the primary metric and its column has both null and non-null values).

              default: false
          • target_n
            optional

            Minimum sample size needed to meet the design specs.

            Any of the following:

            • integer
            • null
          • sufficient_n
            optional

            Whether there are enough available participants to sample to reach target_n.

            Any of the following:

            • boolean
            • null
          • target_possible
            optional

            When the sample size is too small to reach the desired metric_target, this is the target that is possible given available_n, in the improvement (positive) direction from the baseline. It is the absolute counterpart of pct_change_possible. None when the sample size is large enough to detect the desired change, or (for BINARY metrics) when no value above the baseline is a large enough change to reach the requested power.

            Any of the following:

            • number
            • null
          • pct_change_possible
            optional

            When the sample size is too small to reach the desired metric_pct_change, this is the percent change that is possible given available_n. It is the relative counterpart of target_possible. None whenever target_possible is None.

            Any of the following:

            • number
            • null
          • target_possible_lower
            optional

            The detectable metric value below the baseline. (target_possible is the metric value above the baseline.) Both detectable metric values above and below the baseline use the chosen confidence and power, and are always reported together, regardless of the direction of the requested change. For numeric metrics, this is the same distance below the baseline as target_possible is above it. However, for binary metrics, an increase and a decrease that are equally detectable can sit at different distances from the baseline. None when the sample size is large enough to detect the desired change, or (for BINARY metrics) when no value below the baseline is a large enough change to reach the requested power.

            Any of the following:

            • number
            • null
          • pct_change_possible_lower
            optional

            The relative counterpart of target_possible_lower, as a percent change from the baseline (negative for a decrease). A value of -0.10 means a 10% decrease. None whenever target_possible_lower is None.

            Any of the following:

            • number
            • null
          • pct_change_with_desired_n
            optional

            The minimum detectable effect (MDE) achievable for the desired sample size at the chosen confidence and power, reported in the direction of the requested change (negative when the requested change is a decrease from the baseline). Present only when design_spec.desired_n or design_spec.desired_n_clusters is set (frequentist design specs). When desired_n_clusters is set, the desired sample size is desired_n_clusters times this metric's avg_cluster_size.

            Any of the following:

            • number
            • null
          • mde_curve
            optional

            The MDE for each requested sample size, in the same order as design_spec.desired_ns (or desired_ns_clusters, which takes precedence). Present only when one of those is set. Each point is computed best-effort: a size where the calculation fails yields a null pct_change instead of failing the request.

            Any of the following:

            • array<unknown>
              • desired_n
                integer
                required

                Sample size in individual participants used for this point. For cluster-randomized designs this is desired_n_clusters times the metric's avg_cluster_size.

              • desired_n_clusters
                optional

                The requested number of clusters for this point. None for individual-randomized requests.

                Any of the following:

                • integer
                • null
              • pct_change
                optional

                The minimum detectable effect (MDE) at this sample size, as a percent change relative to metric_baseline. None when the power calculation fails at this size (e.g. too small to solve).

                Any of the following:

                • number
                • null
            • null
          • msg
            optional

            A human-friendly summary of the power analysis results.

            Any of the following:

            • MetricPowerAnalysisMessage
            • null
          • num_clusters_total
            optional

            Total number of clusters needed across all arms.

            Any of the following:

            • integer
            • null
          • clusters_per_arm
            optional

            Number of clusters needed for each arm (one entry per arm).

            Any of the following:

            • array<integer>
            • null
          • n_per_arm
            optional

            Number of participants for each arm (one entry per arm).

            Any of the following:

            • array<integer>
            • null
          • design_effect
            optional

            Design effect (DEFF), the clustering penalty multiplier.

            Any of the following:

            • number
            • null
          • effective_sample_size
            optional

            Effective sample size accounting for clustering (total_n / DEFF).

            Any of the following:

            • integer
            • null
      • null
    • assign_summary
      required

      Any of the following:

      • AssignSummary
        • sample_size
          integer
          required

          The number of participants across all arms in total.

        • balance_check
          optional

          Balance test results if available. Online experiments do not have balance checks.

          Any of the following:

          • BalanceCheck
          • null
        • arm_sizes
          optional

          For each arm, the number of participants (and optionally clusters) assigned.

          Any of the following:

          • array<unknown>
            • arm
              object
              required

              Describes an experiment treatment arm.

              • arm_name
                string
                required
                maxLength: 100
              • arm_id
                optional

                Unique ID of the arm. Leave unset when creating an arm.

                Any of the following:

                • string
                • null
              • arm_description
                optional

                Any of the following:

                • string
                • null
              • arm_weight
                optional

                Optional weight for this arm, used for unequal allocation. The weight must be greater than 0 and less than 100. If you set a weight, all arms must have weights, and they must sum to 100.

                Any of the following:

                • ArmWeight
                • null
            • size
              optional
              integer
              default: 0
            • cluster_count
              optional

              The number of clusters assigned to this arm. Set only on cluster-randomized experiments.

              Any of the following:

              • integer
              • null
          • null
      • null
    • spreadsheet_full_url
      optional

      Full URL of the datasource spreadsheet, including the launched experiment's outcomes tab.

      Any of the following:

      • string
      • null
    • webhooks
      optional
      array of
      string

      List of webhook IDs associated with this experiment. These webhooks are triggered when the experiment is committed.

      default: []
    • decision
      optional
      string

      Record any decision(s) made because of this experiment. Will you launch it, and if so when? Regardless of positive or negative results, how do any learnings inform next steps or future hypotheses?

      default: ""
    • impact
      optional
      high
      or
      medium
      or
      low
      or
      negative
      or
      unclear
      or

      Given the results across your tracked metrics and any other observed effects seen elsewhere, record an overall summary here. Do they agree or reject your hypotheses? Beyond the metrics tracked, did variations affect scalability, cost, feedback, or other aspects?

      default: ""

400

The request is invalid. Check that the request body matches the required structure and that all required fields and headers are present.

403

You did not provide a valid API key, or your API key does not have permission for this operation.

Tip: Check that the API key in the X-API-Key header has access to the data source or experiment.

404

The resource was not found, or you do not have access to it.

Tip: Check that the API key in the X-API-Key header has access to the data source or experiment.

422

Validation Error
JSON
  • detail
    optional
    array of
    object
    • loc
      array of
      required

      Any of the following:

      • string
      • integer
    • msg
      string
      required
    • type
      string
      required
    • input
      optional
      unknown
    • ctx
      optional
      object

Type to search…

↑↓ navigate↵ selectEsc close