Deployment

Bases: object

The Lightning AI Deployment.

Allows to fully control a deployment, including retrieving the status, making new release and switching machine types, etc..

Parameters:

• name (Optional[str]) – The name or the id of the deployment.

• teamspace (Union[str, Teamspace, None]) – The teamspace in which you want to deploy. Accepts a bare name or an owner/teamspace slug.

• org (Union[str, Organization, None]) – The name of the organization owning the :param`teamspace` in case it is owned by an org. Deprecated — pass the owner as part of teamspace instead, e.g. teamspace="owner/teamspace".

• user (Union[str, User, None]) – The name of the user owning the :param`teamspace` in case it is owned directly by a user instead of an org. Deprecated — pass the owner as part of teamspace instead, e.g. teamspace="owner/teamspace".

• name (str | None)

• teamspace (str | Teamspace | None)

• org (str | Organization | None)

• user (str | User | None)

delete_env(key)

Delete one directly configured literal Deployment environment variable.

Parameters:

• key (str) – Name of the literal environment variable to delete.

• key (str)

Raises:

ValueError – If the name is invalid, the Deployment does not exist, or the literal variable is not directly configured.

Return type:

None

delete_secret(env_name)

Remove a teamspace secret reference from the Deployment.

Parameters:

• env_name (str) – Injected environment variable name of the secret reference to remove.

• env_name (str)

Raises:

ValueError – If the name is invalid, the Deployment does not exist, or no secret is injected under that name.

Return type:

None

download_logs(*, job_id=None, timestamps=False)

Download a deployment replica’s stored logs to a file and return the written path.

Return type:

str

Parameters:

• job_id (str | None)

• timestamps (bool)

export_request_captures(*, start, end, output_dir, paths=None, include_all_paths=False, status_codes=None, strict=False, request_timeout=60, overwrite=False, remote_path=None)

Export captured request telemetry to local artifacts.

The export is always written to output_dir. If remote_path is provided, the same artifacts are also uploaded to that remote destination. This currently supports lightning_storage targets only.

Parameters:

• start (Union[datetime, str]) – Start of the time range to export captures for.

• end (Union[datetime, str]) – End of the time range to export captures for.

• output_dir (Union[Path, str]) – Local directory where exported artifacts will be written.

• paths (Optional[List[str]]) – Optional list of request paths to filter exports by.

• include_all_paths (bool) – If True, export captures for all request paths.

• status_codes (Optional[List[int]]) – Optional list of HTTP status codes to filter exports by.

• strict (bool) – If True, raise an error on any export failure instead of skipping.

• request_timeout (Union[float, tuple]) – Timeout in seconds (or (connect, read) tuple) for each request.

• overwrite (bool) – If True, overwrite existing files in output_dir.

• remote_path (Optional[str]) – Optional remote destination path for uploading exported artifacts.

• start (datetime | str)

• end (datetime | str)

• output_dir (Path | str)

• paths (List[str] | None)

• include_all_paths (bool)

• status_codes (List[int] | None)

• strict (bool)

• request_timeout (float | tuple)

• overwrite (bool)

• remote_path (str | None)

Returns:

Summary of the export operation.

Return type:

RequestCaptureExportResult

Raises:

RuntimeError – If the deployment has not been created yet.

set_env(new_env, partial=True)

Set directly configured literal Deployment environment variables.

Parameters:

• new_env (Dict[str, str]) – Literal environment variable names and values.

• partial (bool) – Preserve unrelated literal variables when True. Secret references are preserved unless shadowed by a new literal in either mode.

• new_env (Dict[str, str])

• partial (bool)

Raises:

ValueError – If a name is invalid or the Deployment does not exist.

Return type:

None

set_secret(secret_name, env_name=None)

Reference a teamspace secret as a Deployment environment variable.

Parameters:

• secret_name (str) – Name of the teamspace secret to inject.

• env_name (Optional[str]) – Environment variable name to inject the secret under. Defaults to the secret name.

• secret_name (str)

• env_name (str | None)

Raises:

ValueError – If a name is invalid or the Deployment does not exist.

Return type:

None

start(studio=None, cloud=None, machine=None, image=None, autoscale=None, ports=None, release_strategy=None, entrypoint=None, command=None, commands=None, env=None, spot=None, replicas=None, health_check=None, auth=None, custom_domain=None, cloudspace_id=None, quantity=None, include_credentials=None, from_onboarding=None, from_litserve=None, max_runtime=None, path_mappings=None, model=None, hf_token_secret=None, base_image_variant=None, tensor_parallel_size=None, max_model_len=None, gpu_memory_utilization=None, quantization=None, dtype=None, extra_vllm_args=None, enable_weight_reload=None, acknowledged_warnings=None)

The Lightning AI Deployment.

This method creates the first release of the deployment. If the deployment already exists, it sets the deployment to the running state.

Parameters:

• name – The name of the deployment.

• cloud (Union[CloudProvider, str, None]) – Cloud provider or cloud account to run replicas on.

• machine (Optional[Machine]) – The machine used by the deployment replicas.

• autoscale (Optional[AutoScaleConfig]) – The list of the metrics to autoscale on.

• ports (Union[float, List[float], None]) – The ports to reach your replica services.

• image (Optional[str]) – The environement used by the deployment. Currentely, only docker images.

• release_strategy (Optional[ReleaseStrategy]) – The release strategy to use when changing core deployment specs.

• entrypoint (Optional[str]) – The docker container entrypoint.

• command (Optional[str]) – The docker container command.

• env (Union[List[Union[Env, Secret]], Dict[str, str], None]) – The environment variables or secrets to use.

• spot (Optional[bool]) – Whether to use spot instances for the replicas.

• replicas (Optional[int]) – The number of replicas to deploy with.

• health_check (Union[HttpHealthCheck, ExecHealthCheck, None]) – The health check config to know whether your service is ready to receive traffic.

• auth (Union[BasicAuth, TokenAuth, None]) – The auth config to protect your services. Only Basic and Token supported.

• custom_domain (Optional[str]) – Whether your service would be referenced under a custom domain.

• cloudspace_id (Optional[str]) – Connect deployment to a Studio.

• quantity (Optional[int]) – The number of machines per replica to deploy.

• include_credentials (Optional[bool]) – Whether to include the environment variables for the SDK to authenticate

• from_onboarding (Optional[bool]) – Whether the deployment is from onboarding.

• max_runtime (Optional[int]) – the duration (in seconds) for which to allocate the machine. Irrelevant for most machines, required for some of the top-end machines on GCP. If in doubt, set it. Won’t have an effect on machines not requiring it. Defaults to 3h

• path_mappings (Optional[Dict[str, str]]) – Maps container paths to data-connection paths in the form {"<CONTAINER_PATH>": "<CONNECTION_NAME>:<PATH>"} or {"<CONTAINER_PATH>": "<CONNECTION_NAME>"} for the root of a connection. Only applicable when deploying docker containers.

• studio (str | Studio | None)

• cloud (CloudProvider | str | None)

• machine (Machine | None)

• image (str | None)

• autoscale (AutoScaleConfig | None)

• ports (float | List[float] | None)

• release_strategy (ReleaseStrategy | None)

• entrypoint (str | None)

• command (str | None)

• commands (List[str] | None)

• env (List[Env | Secret] | Dict[str, str] | None)

• spot (bool | None)

• replicas (int | None)

• health_check (HttpHealthCheck | ExecHealthCheck | None)

• auth (BasicAuth | TokenAuth | None)

• custom_domain (str | None)

• cloudspace_id (str | None)

• quantity (int | None)

• include_credentials (bool | None)

• from_onboarding (bool | None)

• from_litserve (bool | None)

• max_runtime (int | None)

• path_mappings (Dict[str, str] | None)

• model (str | None)

• hf_token_secret (str | None)

• base_image_variant (str | None)

• tensor_parallel_size (int | None)

• max_model_len (int | None)

• gpu_memory_utilization (float | None)

• quantization (str | None)

• dtype (str | None)

• extra_vllm_args (List[str] | None)

• enable_weight_reload (bool | None)

• acknowledged_warnings (List[str] | None)

Return type:

None

stop()

All the deployment replicas will be stopped and all their traffic blocked.

Return type:

None

update(machine=None, cloud=None, image=None, entrypoint=None, command=None, commands=None, env=None, spot=None, health_check=None, min_replicas=None, max_replicas=None, name=None, ports=None, release_strategy=None, replicas=None, auth=None, custom_domain=None, quantity=None, include_credentials=None, max_runtime=None, path_mappings=None)

Update the deployment configuration.

Changes to machine, image, entrypoint, command/commands, env, spot, cloud, or health_check create a new release. All other arguments (replica counts, ports, auth, etc.) are applied in-place without a new release.

Parameters:

• machine (Optional[Machine]) – New machine type for the replicas.

• cloud (Union[CloudProvider, str, None]) – Cloud provider or cloud account to run replicas on.

• image (Optional[str]) – New docker image for the replicas.

• entrypoint (Optional[str]) – New container entrypoint.

• command (Optional[str]) – New container command string.

• commands (Optional[List[str]]) – List of command strings, joined into a single command. Mutually exclusive with command.

• env (Union[List[Union[Env, Secret]], Dict[str, str], None]) – Environment variables or secrets to inject into replicas.

• spot (Optional[bool]) – Whether to use spot/interruptible instances.

• health_check (Union[HttpHealthCheck, ExecHealthCheck, None]) – Readiness probe configuration.

• min_replicas (Optional[int]) – New minimum replica count for autoscaling.

• max_replicas (Optional[int]) – New maximum replica count for autoscaling.

• name (Optional[str]) – New display name for the deployment.

• ports (Optional[List[float]]) – Exposed port numbers.

• release_strategy (Optional[ReleaseStrategy]) – How to roll out the new release (e.g. blue/green).

• replicas (Optional[int]) – Fixed number of replicas (overrides autoscaling).

• auth (Optional[Auth]) – Authentication configuration for the deployment endpoint.

• custom_domain (Optional[str]) – Custom domain to attach to the deployment.

• quantity (Optional[int]) – Number of machines per replica.

• include_credentials (Optional[bool]) – Whether to inject SDK authentication env vars.

• max_runtime (Optional[int]) – Maximum machine allocation duration in seconds.

• path_mappings (Optional[Dict[str, str]]) – Container-path → data-connection-path mappings for docker jobs.

• machine (Machine | None)

• cloud (CloudProvider | str | None)

• image (str | None)

• entrypoint (str | None)

• command (str | None)

• commands (List[str] | None)

• env (List[Env | Secret] | Dict[str, str] | None)

• spot (bool | None)

• health_check (HttpHealthCheck | ExecHealthCheck | None)

• min_replicas (int | None)

• max_replicas (int | None)

• name (str | None)

• ports (List[float] | None)

• release_strategy (ReleaseStrategy | None)

• replicas (int | None)

• auth (Auth | None)

• custom_domain (str | None)

• quantity (int | None)

• include_credentials (bool | None)

• max_runtime (int | None)

• path_mappings (Dict[str, str] | None)

Return type:

None

property auth: Auth | None

The authentification configuration of the deployment.

Returns:

The authentication configuration, or None if not started.

Return type:

Optional[Auth]

property autoscale: AutoScaleConfig | None

The autoscaling configuration of the deployment.

Returns:

The autoscaling configuration, or None if not started.

Return type:

Optional[AutoScaleConfig]

property cloud_account: str | None

The cloud_account of the replicas.

property deleting_replicas: int | None

The number of deleting replicas.

property env: List[Env | Secret] | None

The env configuration of the deployment.

Returns:

The environment variables and secrets, or None if not started.

Return type:

Optional[List[Union[Secret, Env]]]

property failing_replicas: int | None

The number of failing replicas.

property health_check: HttpHealthCheck | ExecHealthCheck | None

The health check to validate the replicas are ready to receive traffic.

Returns:

The readiness probe configuration, or None if not started.

Return type:

Optional[Union[HttpHealthCheck, ExecHealthCheck]]

property id: str | None

The deployment’s unique identifier, or None before it is created.

property include_credentials: bool | None

The number of machines per replica.

property max_replicas: int | None

The maximum number of replicas.

Returns:

The autoscaling maximum replica count, or None if not started.

Return type:

Optional[int]

property min_replicas: int | None

The minimum number of replicas.

Returns:

The autoscaling minimum replica count, or None if not started.

Return type:

Optional[int]

property pending_replicas: int | None

The number of pending replicas.

Returns:

The count of replicas currently pending, or None if not started.

Return type:

Optional[int]

property ports: List[int] | None

The exposed ports on which you can reach your deployment.

Returns:

List of exposed port numbers, or None if not started.

Return type:

Optional[List[int]]

property quantity: str | None

The number of machines per replica.

property release_id: str | None

The release id of the deployment.

property release_strategy: ReleaseStrategy | None

The release strategy of the deployment.

Returns:

The current release strategy, or None if not started.

Return type:

Optional[ReleaseStrategy]

property replicas: int | None

The default number of replicas the release starts with.

Returns:

The replica count, or None if the deployment has not been started.

Return type:

Optional[int]

property running_replicas: int | None

The number of failing replicas.

Returns:

The count of replicas currently running and ready, or None if not started.

Return type:

Optional[int]

property teamspace: Teamspace

The teamspace of the deployment.

property urls: List[str] | None

The urls to reach the deployment.

Returns:

The list of URLs for the deployment, or None if not started.

Return type:

Optional[List[str]]

property user: User | None

The user of the deployment.

Bases: object

The Env describes an environnement variable.

Parameters:

• name (str)

• value (str)

property env_var_name: str

The container environment variable name this entry occupies.

Bases: object

The Secret describes a protected environnement variable.

Parameters:

• name (str) – Name of the Lightning secret to read the value from.

• env_name (Optional[str]) – Environment variable name to inject the secret under. Defaults to name.

• name (str)

• env_name (str | None)

property env_var_name: str

The container environment variable name this entry occupies.

Bases: Auth

The BasicAuth describes the basic auth mechanism where a username and password are required to authenticate.

Parameters:

• username (str)

• password (str)

Bases: Auth

The TokenAuth describes the token auth mechanism where a token is required to authenticate.

Parameters:

token (str)

Bases: object

The base class for release strategy.

Bases: ReleaseStrategy

The RollingUpdateReleaseStrategy describes the rolling update strategy.

Parameters:

• max_surge (int) – The max_surge argument controls the maximum number of additional replicas that can be created during a rolling update. It specifies the number above the desired replica count that can be temporarily created. During an update, Lightning creates new replicas to replace the old ones, and the max_surge argument ensures that the total number of pods does not exceed a certain limit.

• max_unavailable (int) – The max_unavailable argument determines the maximum number of replicas that can be unavailable during a rolling update. It specifies the maximum number that can be simultaneously removed from service during the update progresses. By default, Lightning terminates one replica at a time while creating new replicas, ensuring that the desired replica count is maintained.

• max_surge (int)

• max_unavailable (int)

Bases: object

Bases: HealthCheck

The ExecHealthCheck determines whether your service is ready using exec command in the container.

Parameters:

• command (str) – The command to be executed within your container to decide whether the service is ready.

• timeout_seconds (int) – The total number of time to wait before declaring the service as unhealthy.

• initial_delay_seconds (int) – The amount of time to wait before starting to execute the command.

• failure_threshold (int) – The total number of retries to do befor declaring the service as unhealthy.

• interval_seconds (int) – The amount of time between retries.

• command (str)

• initial_delay_seconds (int)

• failure_threshold (int)

• interval_seconds (int)

• timeout_seconds (int)

Bases: HealthCheck

The HttpHealthCheck determines whether your service is ready using http request.

Parameters:

• path (str) – The server path to hit to check whether the server is healthy

• port (float) – The server port to hit to check whether the server is healthy

• timeout_seconds (int) – The total number of time to wait before declaring the service as unhealthy.

• initial_delay_seconds (int) – The amount of time to wait before starting to execute the command.

• failure_threshold (int) – The total number of retries to do befor declaring the service as unhealthy.

• interval_seconds (int) – The amount of time between retries.

• path (str)

• port (float)

• initial_delay_seconds (int)

• failure_threshold (int)

• interval_seconds (int)

• timeout_seconds (int)

Bases: object

The AutoScalingMetric determines the metric used to decide whether we should autoscale.

Parameters:

• name (Literal['GPU', 'CPU', 'RPM']) – The name of the metric used to decide whether we should autoscale.

• target (float) – The metric threshold to decide whether we should autoscale.

• name (Literal['GPU', 'CPU', 'RPM'])

• target (float)

Bases: object

The AutoScaleConfig determines how to autoscale your deployment.

Parameters:

• min_replicas (Optional[int]) – The minimum number of replicas. When set to 0, the replicas will stop when there is no traffic left.

• max_replicas (Optional[int]) – The maximum number of replicas.

• metric (Optional[Literal['GPU', 'CPU', 'RPM']]) – The metric used to decide whether we should autoscale.

• threshold (Optional[float]) – The metric threshold to decide whether we should autoscale.

• target_metrics (Optional[List[AutoScalingMetric]]) – Multiple target metrics to autoscale the deployment.

• idle_threshold_seconds (Optional[str]) – The amount of time to wait before stopping a replica after the latest seen request.

• min_replicas (int | None)

• max_replicas (int | None)

• metric (Literal['GPU', 'CPU', 'RPM'] | None)

• threshold (float | None)

• target_metrics (List[AutoScalingMetric] | None)

• idle_threshold_seconds (str | None)

• scale_down_cooldown_seconds (str | None)

• scale_up_cooldown_seconds (str | None)

Bases: object

Parameters:

• output_dir (Path)

• manifest_path (Path)

• csv_path (Path)

• jsonl_path (Path)

• row_count (int)

• captured_count (int)

• uncaptured_count (int)

• missing_content_count (int)

• content_error_count (int)

• uploaded_artifacts (Dict[str, str] | None)