Increase serving traffic and promote a challenger

When you need it

Use a daily rollout to introduce a registered pipeline version gradually. The default policy increases challenger traffic by 10 percentage points after at least 24 hours at each stage. After a passing observation at 100% traffic, the scheduled job automatically promotes that version to the champion alias.

flowchart LR
    A["Challenger stage"] --> C["daily_rollout evaluates evidence"]
    C --> D["ADVANCE after 24 h: add 10 points"]
    D --> A
    C --> E["HOLD: keep traffic and stage clock"]
    E --> A
    C --> F["ROLLBACK on FAIL, even before 24 h"]
    C --> G["COMPLETE: passing 100% stage, promote champion"]

The diagram uses default step/timing settings. ADVANCE and COMPLETE need the full stage time; ROLLBACK happens as soon as fresh evidence fails. A rollback ends that rollout; a hold is evaluated again with better evidence.

This workflow requires an existing champion, a staged challenger with a saved passing comparison, and two compatible serving packages. Train the challenger with promotion_policy: manual_approval so training leaves the current champion in place. The rollout owns the later automatic promotion. Both packages must use the serving-compatible Skyulf runtime used by the controller.

Automatic promotion supports one fitted pipeline per version. It rejects model sets before initialization. Spark-native models and preprocessing that needs other request rows are outside the existing endpoint certificate. Model-set traffic composition is available at the lower-level endpoint API, but the generated daily job requires the single-pipeline approval workflow.

Choose the right release mechanism

If a batch scorer resolves champion, it can pick up the promoted version on a later run. That is separate from serving traffic and does not rewrite prior prediction rows by itself. An online store is needed only when the saved model requires online feature lookup; raw-input serving can use rollout without one.

The sequence is: train and stage a challenger, create the two-version endpoint, initialize durable rollout state, configure the generated controller job, test one invocation, then unpause its schedule. The snippets below form that sequence and require existing model versions and permissions; they do not train models.

What to edit

Use serving observations for telemetry and label collection. Their evidence feeds this controller; their schedules do not replace the rollout stage clock.

The command

Prepare an isolated endpoint

Use a new A/B endpoint name. Keep an existing SQL function's pinned endpoint separate; changing A/B traffic would change that function's prediction contract.

from dataclasses import replace
from databricks.sdk import WorkspaceClient
from skyulf.integrations.databricks.serving import (
    PinnedEndpointSpec,
    build_rollout_endpoint,
    prepare_pinned_endpoint,
    rollout_endpoint_ready,
)

workspace = WorkspaceClient()
registry_options = {"tracking_uri": "databricks", "registry_uri": "databricks-uc"}
champion_spec = PinnedEndpointSpec(
    endpoint_name="risk-gradual-release",
    model_name="models.risk.customer_risk",
    model_version="7",
    logging_catalog="ops",
    logging_schema="serving",
    logging_table_prefix="risk_gradual_release",
)
challenger_spec = replace(champion_spec, model_version="8")
plan = build_rollout_endpoint(
    prepare_pinned_endpoint(champion_spec, **registry_options),
    prepare_pinned_endpoint(challenger_spec, **registry_options),
)
print(plan.config)

# Run creation once. The native create operation rejects an existing name.
created = workspace.api_client.do(
    "POST", "/api/2.0/serving-endpoints", body=plan.config
)

Creation is asynchronous. In a later cell, inspect the same endpoint:

endpoint = workspace.api_client.do(
    "GET", f"/api/2.0/serving-endpoints/{champion_spec.endpoint_name}"
)
ready = rollout_endpoint_ready(endpoint, plan, challenger_percentage=0)
print(ready)

Continue only when ready is true. Both model versions must be loaded, routing must be exactly 100/0, and the configured inference logging must be intact. Do not repeat creation after a timeout without inspecting the endpoint first.

Initialize durable rollout state

Create an explicit MLflow run for this rollout. Use the comparison digest and quality settings from the challenger's saved training result. The approval settings below are examples; their metric, threshold and improvement must match that saved comparison.

Set the traffic step and timing in your generated project's config/serving.yml before initialization, then read that same file in the initialization cell:

rollout:
  # Keep the other generated rollout fields.
  traffic_step_percent: 10
  stage_minimum_hours: 24.0
  evidence_max_age_hours: 24.0

traffic_step_percent is an integer from 1 to 100; both hour values must be positive and finite. stage_minimum_hours applies to every stage, including the final 100% stage. evidence_max_age_hours limits how old the observation and its window end may be; it is independent of the stage's length. For an isolated test, stage_minimum_hours: 0.1 means six minutes; keep the 24-hour default for normal use. Shorter stages still require enough requests and request-keyed labels under health. They do not change the daily job schedule: invoke the job manually after each test interval or adjust its schedule while preserving one writer. Each invocation advances at most one step.

from pathlib import Path

import yaml
from mlflow.tracking import MlflowClient
from skyulf.integrations.databricks.data.admission import SingleWriterAdmission
from skyulf.integrations.databricks.serving import (
    DailyRolloutPolicy,
    MLflowRolloutStore,
    build_rollout_promotion,
    initialize_rollout,
)

registry = MlflowClient(**registry_options)
approval = {
    "model_name": champion_spec.model_name,
    "metric": "heldout_rmse",
    "min_improvement": 0.0,
    "quality_threshold": 5.0,
    "quality_gates": None,
    "max_rows": 10000,
    "max_input_mb": 64,
    **registry_options,
}
comparison_sha256 = "REPLACE_WITH_SAVED_COMPARISON_SHA256"
promotion = build_rollout_promotion(
    plan, approval, auto_promote=True, comparison_sha256=comparison_sha256
)
experiment_id = "REPLACE_WITH_EXISTING_EXPERIMENT_ID"
run = registry.create_run(experiment_id)
store = MLflowRolloutStore(registry, run.info.run_id)
# Use the deployed file's absolute /Workspace/.../config/serving.yml path.
serving_config_path = "/Workspace/REPLACE_WITH_PROJECT_PATH/config/serving.yml"
rollout_settings = yaml.safe_load(Path(serving_config_path).read_text())["rollout"]
result = initialize_rollout(
    workspace,
    plan,
    store=store,
    policy=DailyRolloutPolicy.from_config(rollout_settings),
    admission=SingleWriterAdmission(),
    rollout_id=run.info.run_id,
    promotion=promotion,
)
print(result)
print("rollout run_id:", run.info.run_id)

The job checks these YAML values against the saved policy before reconciling the endpoint. To change a policy, initialize a new rollout on a settled 100/0 endpoint and configure its new run_id; editing YAML alone cannot shorten an active rollout. Missing timing fields use the 10-point/24-hour defaults. For an older run initialized with custom SDK values, add matching fields to its YAML.

SingleWriterAdmission is an explicit ownership contract, not a distributed lock. Restrict endpoint and alias mutation permissions to the controller's identity, serialize all its writers, and keep max_concurrent_runs: 1. Another job or UI writer must not modify those resources concurrently. Applications with multiple writers must supply a shared admission provider to the SDK operations. There is no native atomic compare-and-swap for an endpoint configuration.

Enable the daily Bundle job

Generate the project with include_daily_rollout: "true" in the template init configuration. This adds resources/daily_rollout.job.yml, its notebook, config/serving.yml, and serving dependencies. To add it to an existing project, follow project upgrades. The job is initially PAUSED; its configuration is initially disabled.

Fill rollout in config/serving.yml:

• Set run_id to the initialized rollout run and copy the comparison digest and exact approval settings used above.
• Supply representative smoke_records matching the admitted request schema. The job calls both named served versions directly, including the version with zero routed traffic.
• Configure minimum request and label counts, label coverage, maximum error rate and the latency limit (p99) in health.
• Configure monitoring for the concrete challenger version and endpoint. source_table and prediction_table select its inference payload table. Use execution_engine: spark, an existing reference_namespace, and a label table with a result-availability timestamp.
• monitoring.environment labels the saved observation only. The generated daily_rollout job passes bundle_target: ${bundle.target}, so evidence records the target that ran it; the YAML value (syst) is used only when that parameter is absent. It does not select tables or change rollout decisions.
• Set enabled: true and exclusive_writer: true only after establishing the ownership contract. Deploy, test one run, and then unpause the schedule.

These fields are an excerpt of that generated block, not a replacement for the required approval, endpoint and monitoring identities:

rollout:
  enabled: true
  exclusive_writer: true
  run_id: "COPY_THE_INITIALIZED_ROLLOUT_RUN_ID"
  traffic_step_percent: 10
  stage_minimum_hours: 24.0
  evidence_max_age_hours: 24.0
  comparison_sha256: "COPY_THE_SAVED_COMPARISON_DIGEST"
  smoke_records:
    - income: 42000.0
      age: 38.0
  health:
    minimum_requests: 100
    maximum_error_rate: 0.01
    maximum_latency_ms: 1000.0
    minimum_labeled_rows: 30
    minimum_label_coverage: 0.8

Replace the smoke fields with the actual request schema; an online feature model may require keys and additional source fields instead. The three policy fields must match initialization; the health thresholds configure the job's observation. Preserve both deliberately.

From the configured generated project:

databricks bundle validate --strict -t test --profile YOUR_PROFILE
databricks bundle deploy -t test --profile YOUR_PROFILE
databricks bundle run daily_rollout -t test --profile YOUR_PROFILE

Open the resulting job task's Output, then compare the reported percentage and endpoint revision with the native endpoint configuration. Inspect the saved observation and reason before unpausing the daily schedule. Missing observations, missing latency, incomplete logs and immature labels are holds, not healthy passes.

Serving labels join on databricks_request_id and request_row_index; retain these identifiers when collecting outcomes. The adapter uses the actual served model identity from telemetry and evaluates the current stage's time interval. It does not recompute historical predictions. Late or missing outcomes hold the rollout until enough mature labels are available. Quality thresholds come from the saved comparison; the job does not accept a configurable PASS flag.

The job records the complete observation and verifies its MLflow artifact before changing traffic. Its output includes status, state.challenger_percentage, the decision and an observation digest. A successful final transition also includes the guarded promotion receipt.

What you should see

Decisions and timing

A missed scheduled run does not cause several increases at once. Stage time starts when the new routing configuration is verified as ready. Endpoint traffic and registry aliases are separate: reaching 100% does not itself change an alias. The final approval also checks that the original champion, challenger, comparison digest and quality policy still match.

health.maximum_latency_ms bounds p99 model execution time, the duration at or below which approximately 99% of timed captured requests fall. A single unusually slow request can exceed the threshold without failing the stage. For example, p99 of 80 ms passes a 100 ms limit even if the maximum is 900 ms; p99 of 120 ms fails the same limit and requests rollback. This measures model execution, not total client/network response time.

If a saved summary has no latency_p99_ms, or it is null, the check uses latency_max_ms. Recorded zero is valid. An invalid non-null p99, or no usable latency in either field, holds the stage. Error-rate, minimum request and label counts, quality, stage duration and incomplete-log checks also apply.

For example, a verified 10% stage with insufficient mature labels stays at 10%. A confirmed health regression returns traffic to 0% challenger / 100% retained champion and ends that rollout. This traffic rollback does not promote the challenger or restore a registry alias from an unrelated promotion. At 100%, the challenger still needs its final complete passing stage before becoming champion. A day passing on the calendar is not sufficient evidence.

The evidence verdict and controller action are different fields:

A label shortfall today does not reset the clock. The next invocation can advance once the complete stage has enough fresh, mature evidence; it never catches up by applying several missed daily increases. A terminal ROLLED_BACK run stays at 0% on subsequent invocations and cannot resume.

Too few requests, unavailable or inconsistent request counts, missing or invalid latency, stale/future observations, logging errors, too few mature labels and insufficient label coverage all hold. Logging errors mean capture is incomplete, not a measured prediction regression. Once usable evidence exists, an excessive error rate, excessive latency or failed quality gate yields FAIL. The controller first verifies evidence identity and freshness; stale or mismatched failure evidence cannot trigger rollback.

Feature updates during a rollout

New feature-bound candidates save their enriched training-time holdout at lifecycle/data/holdout.parquet in the candidate's MLflow run. The saved comparison binds its hash, owner run, schema, row and memory limits, split membership and model identity. Lifecycle training reuses its prepared holdout; SDK training records the same bounded artifact when needed.

Final approval reloads those verified inputs and evaluates the fitted pipeline. Feature table versions may advance while the rollout runs: current feature values do not replace historical holdout values. The original source and feature table identities must still match. Missing or altered artifacts, changed split metadata, insufficient read budgets or a dropped and recreated table block approval. Retain the candidate run and its artifacts throughout the rollout and approval.

A candidate without this saved holdout uses strict snapshot reconstruction. If a feature table has advanced, approval explains that no frozen holdout is available and remains blocked. Train a new candidate with the updated runtime; do not edit comparison artifacts to bypass that check.

Online requests still use current published features and enforce their missing value and freshness policy. Historical holdout success does not replace fresh serving health, quality or final-stage observations. A shared feature failure can affect both versions; returning traffic to the champion does not repair that data. See online features for the serving freshness contract and supported request shape.

Common errors and fixes

Retry, stop and recovery

Before an endpoint update, the controller persists the exact intended request. After a timeout or interruption, rerun the job to reconcile that request. A reconciliation run does not immediately apply another traffic increase. The stage clock advances only after the expected endpoint revision and routes are verified.

New PREPARED receipts distinguish two submission states:

The controller persists OUTCOME_UNKNOWN immediately before sending the request. A crash between those operations is therefore ambiguous even if the endpoint still has its old configuration. Repeated invocations hold that receipt; elapsed time alone does not prove that an update cannot still apply.

Inspect the saved intent and endpoint using the same clients and store as initialization:

receipt = store.load()
request = receipt.get("request", {})
endpoint = workspace.api_client.do(
    "GET", f"/api/2.0/serving-endpoints/{receipt['state']['endpoint_name']}"
)
print({
    "receipt_status": receipt["status"],
    "submission_state": request.get("submission_state", "OUTCOME_UNKNOWN"),
    "prior_revision": request.get("prior_config_version"),
    "target_revision": request.get("target_config_version"),
    "current_revision": endpoint.get("config", {}).get("config_version"),
    "pending_config": endpoint.get("pending_config"),
    "endpoint_state": endpoint.get("state"),
})

If the exact requested configuration has settled, rerun daily_rollout to reconcile it. A pending update must settle before proceeding. An unexpected endpoint ID, revision, model or routing setting stops automation rather than overwriting another writer's change.

There is no force-abort API for an ambiguous receipt. If the request cannot be resolved, pause the job, establish exclusive ownership and inspect the native endpoint operation with your workspace administrator. Retain the old run as evidence. Once no old update can still apply, explicitly restore an admitted, ready 100/0 endpoint and initialize a new rollout run using the sequence above; update rollout.run_id before resuming. Do not edit receipt artifacts or tags, reset the submission marker, or blindly resend an uncertain request.

Pause the scheduled job to stop further decisions. A verified regression rollback is terminal for that rollout; starting again requires a new explicit rollout. After a successful promotion, both concrete served entities remain available. Retiring the previous model or changing SQL pins is a separate operator action.

To restart after rollback, investigate the saved failure, correct its cause, and follow the explicit endpoint/initialization sequence with a new rollout run and an admitted model pair. Recheck the current champion, comparison and ownership; do not reuse or edit the terminal receipt to reset its state. Pausing a schedule only prevents new controller runs; it does not itself roll traffic back. A shared feature outage still needs its own repair.