Inference & Cloud¶

Production inference pipeline and S3 infrastructure. Implements Phase F (inference) plus the cloud storage layer for model I/O, output auditing, and the model registry.

src.inference ¶

Bridge Classification Inference Script.

Loads a trained Sparse U-Net model, processes raw LAS/LAZ file(s), and outputs classified LAS/LAZ file(s) with ASPRS standard codes.

Workflow:

Load Model (once).
For each input file: a. Load LAS file. b. Voxelize points (keep track of which point belongs to which voxel). c. Run Model Inference. d. Map Voxel Labels to Original Points. e. Save LAS file.

Requires an NVIDIA GPU (spconv-cu120).

Example - single file, masked mode (bridge deck overlaid on original lidar):

python src/inference.py         --input ./data/ml-data/testing/02050206/bridge_10598181.laz         --output ./data/ml-data/prediction.laz         --model ./experiments/bridge-base-v0/checkpoints/best.ckpt

Example - single file, raw mode (all model labels):

python src/inference.py         --input bridge.laz --output pred.laz --model model.ckpt --mode raw

Example - batch via pairs file

python src/inference.py         --pairs-file ./pairs.tsv         --model ./experiments/bridge-base-v0/checkpoints/best.ckpt         --mode masked

Where pairs.tsv has one tab-separated line per file: input.laz<TAB>output.laz

apply_bridge_mask ¶

apply_bridge_mask(original_classification: ndarray, point_labels_model: ndarray) -> ndarray

Apply binary bridge deck mask: only reclassify model class 2 -> ASPRS 17.

All other points retain their original LiDAR classification unchanged.

Parameters:

Name	Type	Description	Default
`original_classification`	`ndarray`	np.ndarray of original ASPRS codes from the LAS file.	required
`point_labels_model`	`ndarray`	np.ndarray of model class predictions (0-3) per point.	required

Returns:

Type	Description
`ndarray`	np.ndarray of merged classification codes (uint8).

load_las ¶

load_las(filepath: str) -> tuple

Read LAS file and return XYZ + Intensity.

save_las ¶

save_las(output_path: str | Path, original_arrays: ndarray, labels: ndarray) -> None

Save the classified point cloud.

load_model ¶

load_model(checkpoint_path: str, device: device) -> SparseUNet

Load a trained SparseUNet model from a checkpoint file.

Handles both Lightning checkpoints (strips 'model.' prefix from keys) and raw state dict checkpoints.

Parameters:

Name	Type	Description	Default
`checkpoint_path`	`str`	Path to .ckpt or .pth checkpoint file.	required
`device`	`device`	Target torch device (cuda or cpu).	required

Returns:

Type	Description
`SparseUNet`	SparseUNet model in eval mode on the specified device.

run_inference ¶

run_inference(model: Module, input_path: str, output_path: str, voxel_size: float = 0.1, device: device = device('cuda'), mode: str = 'masked') -> InferenceResult

Run inference on a single LAS/LAZ file and save the classified result.

Parameters:

Name	Type	Description	Default
`model`	`Module`	Pre-loaded SparseUNet model (already on device, in eval mode).	required
`input_path`	`str`	Path to input LAS/LAZ file.	required
`output_path`	`str`	Path to write classified LAS/LAZ file.	required
`voxel_size`	`float`	Voxel size in meters (must match training). Default: 0.1.	`0.1`
`device`	`device`	Device the model is on. Default: cpu.	`device('cuda')`
`mode`	`str`	Output mode - 'masked' (bridge deck only, default), 'raw' (all model labels), or 'both' (save raw to output_path and masked alongside it).	`'masked'`

Returns:

Type	Description
`InferenceResult`	InferenceResult.SUCCESS if inference succeeded,
`InferenceResult`	InferenceResult.FAILED if the file failed,
`InferenceResult`	or InferenceResult.SKIPPED if intentionally skipped (e.g. too few points).

parse_pairs_file ¶

parse_pairs_file(filepath: str) -> list

Parse a TSV file of input/output path pairs.

Each line: input_pathoutput_path

Returns:

Type	Description
`list`	List of (input_path, output_path) tuples.

run_batch_inference ¶

run_batch_inference(model: SparseUNet, pairs: list, voxel_size: float = 0.1, device: device = device('cuda'), bridge_timeout: float = 150, mode: InferenceMode = MASKED) -> tuple

Run inference on multiple input/output file pairs.

Processes each pair sequentially, continuing on failure so one bad file does not prevent the rest from being processed.

A per-bridge wall-clock timeout (bridge_timeout seconds) is enforced via SIGALRM. If a bridge hangs (e.g. stuck PDAL read, GPU kernel), it is skipped and processing continues with the next bridge.

Parameters:

Name	Type	Description	Default
`model`	`SparseUNet`	Pre-loaded SparseUNet model (already on device, in eval mode).	required
`pairs`	`list`	List of (input_path, output_path) tuples.	required
`voxel_size`	`float`	Voxel size in meters. Default: 0.1.	`0.1`
`device`	`device`	Device the model is on. Default: cpu.	`device('cuda')`
`bridge_timeout`	`float`	Seconds before a hung bridge is skipped. Default: 150.	`150`
`mode`	`InferenceMode`	Output mode passed to run_inference. Default: 'masked'.	`MASKED`

Returns:

Type	Description
`tuple`	Tuple of (succeeded_count, failed_count, skipped_count).

CLI Arguments (`src/inference.py`)¶

Argument	Default	Description
`--input`	None	Input LAS/LAZ file path (single-file mode)
`--output`	None	Output LAS/LAZ file path (single-file mode)
`--pairs-file`	None	TSV file with input/output pairs (batch mode)
`--model`	(required)	Path to `.ckpt` checkpoint
`--voxel-size`	0.1	Voxel size (must match training)
`--bridge-timeout`	150	Seconds before a hung bridge is skipped (batch mode)
`--mode`	`masked`	Output mode: `masked`, `raw`, or `both`

Modes:

masked - bridge deck only (class 2 to ASPRS 17) overlaid on original classification
raw - all model classes replace original classification via MODEL_TO_LAS_MAP
both - saves _predicted (raw) and _bridge_masked (masked) files

Usage Examples¶

# Single file, masked mode (default)
python src/inference.py \
    --model ./experiments/my-model/checkpoints/epoch=35.ckpt \
    --input ./data/ml-data/testing/02050206/bridge_10598181.laz \
    --output ./data/ml-data/predictions/bridge_10598181_bridge_masked.laz

# Batch mode with pairs file (model loaded once, processes all pairs)
python src/inference.py \
    --pairs-file ./pairs.tsv \
    --model ./experiments/my-model/checkpoints/epoch=35.ckpt \
    --mode masked --bridge-timeout 150

# Both mode (saves _predicted and _bridge_masked side by side)
python src/inference.py \
    --model ./experiments/my-model/checkpoints/epoch=35.ckpt \
    --input ./bridge.laz --output ./bridge_predicted.laz \
    --mode both

src.s3_client ¶

Generic S3 operations - client factory, file I/O, URI parsing.

Pure S3 logic, no domain-specific code (e.g. no LAS/point cloud handling here).

create_s3_client ¶

create_s3_client(profile: Optional[str] = None) -> Any

Create a boto3 S3 client with standard retry config.

Parameters:

Name	Type	Description	Default
`profile`	`Optional[str]`	AWS profile name (None uses default credentials).	`None`

Returns:

Type	Description
`Any`	boto3 S3 client with adaptive retry (AWS_MAX_RETRIES attempts).

parse_s3_uri ¶

parse_s3_uri(uri: str) -> Tuple[str, str]

Split 's3://bucket/key' into (bucket, key).

object_exists ¶

object_exists(s3_client: Any, bucket: str, key: str) -> bool

Return True if the S3 object exists, False on 404; re-raises other errors.

download_file ¶

download_file(s3_client: Any, bucket: str, key: str, local_path: str) -> None

Download an S3 object to a local path, creating parent directories as needed.

upload_file ¶

upload_file(s3_client: Any, local_path: str, bucket: str, key: str) -> None

Upload a local file to S3.

upload_json ¶

upload_json(s3_client: Any, data: Any, bucket: str, key: str) -> None

Serialize data as JSON and upload to S3 via temp file.

upload_text ¶

upload_text(s3_client: Any, text: str, bucket: str, key: str) -> None

Upload a text string to S3 via temp file.

download_json ¶

download_json(s3_client: Any, bucket: str, key: str) -> Any

Download a JSON file from S3 and return parsed data.

stream_manifest_lines ¶

stream_manifest_lines(s3_client: Any, manifest_uri: str) -> Iterator[str]

Stream a manifest file from S3, yielding non-empty stripped lines.

Parameters:

Name	Type	Description	Default
`s3_client`	`Any`	boto3 S3 client.	required
`manifest_uri`	`str`	Full S3 URI (s3://bucket/key).	required

Yields:

Type	Description
`str`	Non-empty stripped line strings.

src.s3_paths ¶

Bridge-specific S3 path conventions for inference I/O.

Resolves manifest lines to full S3 keys by probing extensions, and computes output key naming by inference mode (raw/masked/both).

resolve_input_key ¶

resolve_input_key(s3_client: Any, bucket: str, input_prefix: str, manifest_line: str) -> str

Resolve a manifest line to a full S3 input key, probing extensions if needed.

If the manifest line already ends with .laz or .las, it is used directly. Otherwise each extension in PROBE_EXTENSIONS is tried via head_object.

Parameters:

Name	Type	Description	Default
`s3_client`	`Any`	boto3 S3 client.	required
`bucket`	`str`	S3 bucket name.	required
`input_prefix`	`str`	S3 prefix for input files.	required
`manifest_line`	`str`	e.g. '02050206/bridge_123' or '02050206/bridge_123.laz'	required

Returns:

Type	Description
`str`	Full S3 key string (e.g. 'prefix/02050206/bridge_123.laz').

Raises:

Type	Description
`FileNotFoundError`	If no matching object exists in S3.

resolve_extension ¶

resolve_extension(s3_client: Any, bucket: str, input_prefix: str, manifest_line: str) -> str

Determine the actual file extension for a manifest line by probing S3.

Returns the extension from the manifest line directly if it already has one. Otherwise probes S3 via resolve_input_key. Falls back to '.laz' if not found.

Parameters:

Name	Type	Description	Default
`s3_client`	`Any`	boto3 S3 client.	required
`bucket`	`str`	S3 bucket name.	required
`input_prefix`	`str`	S3 prefix for input files.	required
`manifest_line`	`str`	Manifest entry.	required

Returns:

Type	Description
`str`	Extension string (e.g. '.laz').

resolve_output_keys ¶

resolve_output_keys(output_prefix: str, manifest_line: str, ext: str, mode: InferenceMode) -> Dict[str, str]

Compute the expected S3 output key(s) for a manifest line.

Parameters:

Name	Type	Description	Default
`output_prefix`	`str`	S3 prefix for output files (no trailing slash).	required
`manifest_line`	`str`	Manifest entry used to derive HUC ID and stem (e.g. '02050206/bridge_123' or '02050206/bridge_123.laz').	required
`ext`	`str`	File extension to use for outputs (e.g. '.laz').	required
`mode`	`InferenceMode`	Inference mode - 'raw', 'masked', or 'both'.	required

Returns:

Type	Description
`Dict[str, str]`	Dict with keys: 'primary' - always present (the main output key) 'masked' - present only when mode='both'

Output key patterns: - raw: {output_prefix}/{huc_id}/{stem}_predicted{ext} - masked: {output_prefix}/{huc_id}/{stem}_bridge_masked{ext} - both: primary = {output_prefix}/{huc_id}/{stem}_predicted{ext} masked = {output_prefix}/{huc_id}/{stem}_bridge_masked{ext}

src.s3_audit ¶

S3 output audit for bridge inference runs.

Thread-pool orchestrator that checks whether expected output files exist in S3. Uses bridge path conventions from s3_paths and generic S3 ops from s3_client.

audit_s3_outputs ¶

audit_s3_outputs(profile: Optional[str], bucket: str, input_prefix: str, output_prefix: str, mode: InferenceMode, manifest_lines: List[str], workers: int = DEFAULT_AUDIT_WORKERS, progress_interval: int = 0) -> Tuple[int, List[str]]

Check S3 for expected inference outputs using a thread pool.

Creates per-thread S3 clients (boto3 clients are not thread-safe). For each manifest line, resolves the expected output key(s) and checks existence via head_object.

Parameters:

Name	Type	Description	Default
`profile`	`Optional[str]`	AWS profile name (None uses default credentials).	required
`bucket`	`str`	S3 bucket containing outputs.	required
`input_prefix`	`str`	S3 prefix for input files (for extension probing). Empty string skips probing and assumes .laz.	required
`output_prefix`	`str`	S3 prefix for output files.	required
`mode`	`InferenceMode`	Inference mode (determines expected output filenames).	required
`manifest_lines`	`List[str]`	List of manifest entries to check.	required
`workers`	`int`	Number of parallel threads.	`DEFAULT_AUDIT_WORKERS`
`progress_interval`	`int`	Print progress every N entries (0 = silent).	`0`

Returns:

Type	Description
`Tuple[int, List[str]]`	Tuple of (found_count, list of missing manifest lines).

src.model_registry ¶

Model registry I/O for the S3-backed model registry.

Shared load/upload operations for registry.json, used by register_model, promote_model, compare_experiments, and evaluate_model.

load_registry ¶

load_registry(s3_client: Any, bucket: str, registry_key: str) -> Dict[str, Any]

Download registry.json from S3, or return a skeleton if it doesn't exist.

Parameters:

Name	Type	Description	Default
`s3_client`	`Any`	Boto3 S3 client.	required
`bucket`	`str`	S3 bucket containing the registry.	required
`registry_key`	`str`	S3 key for registry.json.	required

Returns:

Type	Description
`Dict[str, Any]`	Registry dict with keys: schema_version, models, production.

upload_registry ¶

upload_registry(s3_client: Any, bucket: str, registry_key: str, registry: Dict[str, Any]) -> None

Write registry.json to a temp file and upload to S3.

Parameters:

Name	Type	Description	Default
`s3_client`	`Any`	Boto3 S3 client.	required
`bucket`	`str`	S3 bucket containing the registry.	required
`registry_key`	`str`	S3 key for registry.json.	required
`registry`	`Dict[str, Any]`	Registry dict to upload.	required

Inference & Cloud¶

src.inference ¶

apply_bridge_mask ¶

load_las ¶

save_las ¶

load_model ¶

run_inference ¶

parse_pairs_file ¶

run_batch_inference ¶

CLI Arguments (src/inference.py)¶

Usage Examples¶

src.s3_client ¶

create_s3_client ¶

parse_s3_uri ¶

object_exists ¶

download_file ¶

upload_file ¶

upload_json ¶

upload_text ¶

download_json ¶

stream_manifest_lines ¶

src.s3_paths ¶

resolve_input_key ¶

resolve_extension ¶

resolve_output_keys ¶

src.s3_audit ¶

audit_s3_outputs ¶

src.model_registry ¶

load_registry ¶

upload_registry ¶

CLI Arguments (`src/inference.py`)¶