Weak Supervision Pipeline¶

Deterministic weak labeling algorithm for generating silver training data from raw LiDAR. Implements Phases A (download & weak supervise) and B (preprocess) of the pipeline.

src.weak_supervision ¶

Weak supervision algorithm for bridge LiDAR classification.

Core algorithm for determining linear vs. complex bridges using RANSAC plane fitting, linearity checking, convex hull masking, and Z-distance heuristic labeling.

FailureReason ¶

Bases: Enum

Why a bridge failed weak supervision.

Most values are set by process_bridge() itself. TIMEOUT is set by the caller's subprocess wrapper - included here so the consumer can dispatch on all failure reasons with a single enum.

NO_POINTS `class-attribute` `instance-attribute` ¶

NO_POINTS = 'no_points'

No LiDAR points found in the EPT source for this bridge geometry.

RANSAC_INSUFFICIENT `class-attribute` `instance-attribute` ¶

RANSAC_INSUFFICIENT = 'ransac_insufficient'

Too few points (or unique points) to run RANSAC.

RANSAC_FAILED `class-attribute` `instance-attribute` ¶

RANSAC_FAILED = 'ransac_failed'

RANSAC fitting raised an exception.

RANSAC_LOW_INLIERS `class-attribute` `instance-attribute` ¶

RANSAC_LOW_INLIERS = 'ransac_low_inliers'

RANSAC converged but found fewer inliers than min_ransac_inliers.

HULL_FAILED `class-attribute` `instance-attribute` ¶

HULL_FAILED = 'hull_failed'

Convex hull of RANSAC inliers could not be computed.

HIGH_RMSE `class-attribute` `instance-attribute` ¶

HIGH_RMSE = 'high_rmse'

Inlier RMSE exceeded max_rmse threshold (poor plane fit).

CURVED `class-attribute` `instance-attribute` ¶

CURVED = 'curved'

Bridge skeleton exceeded the linearity deviation threshold (arch or sag).

EXCEPTION `class-attribute` `instance-attribute` ¶

EXCEPTION = 'exception'

Unhandled exception during processing.

TIMEOUT `class-attribute` `instance-attribute` ¶

TIMEOUT = 'timeout'

Bridge exceeded the wall-clock timeout (set by caller, not process_bridge).

BridgeProcessingConfig `dataclass` ¶

Centralized configuration for bridge processing pipeline. Contains all magic numbers, thresholds, and parameters.

__post_init__ ¶

__post_init__()

Set default values for mutable types.

from_dict `classmethod` ¶

from_dict(d: Dict[str, Any]) -> BridgeProcessingConfig

Create config from dictionary (for multiprocessing serialization).

to_dict ¶

to_dict() -> Dict[str, Any]

Convert config to dictionary (for multiprocessing serialization).

check_bridge_linearity ¶

check_bridge_linearity(xy_points: ndarray, z_points: ndarray, config: BridgeProcessingConfig, deviation_threshold: Optional[float] = None) -> Tuple[bool, float]

Slices the bridge into bins, finds the median Z for each bin (the 'skeleton'), and checks if that skeleton deviates from a straight line.

Parameters:

Name	Type	Description	Default
`xy_points`	`ndarray`	XY coordinates of points	required
`z_points`	`ndarray`	Z coordinates of points	required
`config`	`BridgeProcessingConfig`	BridgeProcessingConfig instance	required
`deviation_threshold`	`Optional[float]`	Optional threshold override (defaults to config.linearity_deviation_threshold)	`None`

Returns:

Type	Description
`Tuple[bool, float]`	Tuple of (is_curved: bool, max_deviation: float)

fit_ransac_from_arrays ¶

fit_ransac_from_arrays(arrays: ndarray, config: BridgeProcessingConfig) -> Dict[str, Any]

Run RANSAC plane fitting on in-memory point arrays (e.g. from LAZ). Used for visualization; does not run linearity or RMSE rejection.

Parameters:

Name	Type	Description	Default
`arrays`	`ndarray`	Structured array with fields X, Y, Z, Classification (PDAL-style).	required
`config`	`BridgeProcessingConfig`	BridgeProcessingConfig instance.	required

Returns:

Type	Description
`Dict[str, Any]`	On success: dict with success=True, x_center, y_center, X_local, Y_local, Z, coef_x, coef_y, intercept, inlier_mask_full, lateral_mask, dist_from_plane_all.
`Dict[str, Any]`	On failure: dict with success=False, error=str.

process_bridge ¶

process_bridge(ept_url: str, bridge_geometry: Any, config: BridgeProcessingConfig, buffer_meters: Optional[float] = None) -> Optional[Dict[str, Any]]

Process a single bridge with weak supervision rules.

Parameters:

Name	Type	Description	Default
`ept_url`	`str`	EPT URL for the lidar source	required
`bridge_geometry`	`Any`	Shapely geometry of the bridge	required
`config`	`BridgeProcessingConfig`	BridgeProcessingConfig instance with all parameters	required
`buffer_meters`	`Optional[float]`	Buffer size in meters (defaults to config.default_buffer_meters)	`None`

Returns:

Type	Description
`Optional[Dict[str, Any]]`	Dictionary with keys: 'success' (bool), 'arrays' (numpy structured array),
`Optional[Dict[str, Any]]`	'original_arrays' (numpy structured array), 'rmse' (float), 'deviation' (float),
`Optional[Dict[str, Any]]`	'error' (str). Returns None if processing failed.

src.preprocess_bridges ¶

Bridge Normalization Preprocessing Script.

Processes bridge lidar data from HUC-organized silver_training directory, normalizes coordinates and classifications per bridge, and saves normalized outputs while preserving HUC folder structure.

Example

python src/preprocess_bridges.py
python src/preprocess_bridges.py \
    --input-dir ./data/ml-data/silver_training \
    --output-dir ./data/ml-data/silver_training_normalized
python src/preprocess_bridges.py --skip-existing --workers 4

process_laz_file ¶

process_laz_file(filepath: Path, output_dir: Path, skip_existing: bool = False) -> Tuple[bool, Optional[str]]

Process a single LAZ file with normalization and class remapping.

Parameters:

Name	Type	Description	Default
`filepath`	`Path`	Path to input LAZ file	required
`output_dir`	`Path`	Output directory for .npy and .json files	required
`skip_existing`	`bool`	If True, skip files that already have output files	`False`

Returns:

Type	Description
`Tuple[bool, Optional[str]]`	Tuple of (success: bool, error_message: Optional[str])

process_laz_file_worker ¶

process_laz_file_worker(args: Tuple[Path, Path, bool]) -> Tuple[bool, Optional[str], Path, bool]

Worker function for multiprocessing that processes a single LAZ file.

Parameters:

Name	Type	Description	Default
`args`	`Tuple[Path, Path, bool]`	Tuple of (filepath, output_dir, skip_existing)	required

Returns:

Type	Description
`Tuple[bool, Optional[str], Path, bool]`	Tuple of (success: bool, error_message: Optional[str], filepath: Path, was_skipped: bool)

process_huc_folder ¶

process_huc_folder(huc_dir: Path, output_base_dir: Path, skip_existing: bool = False, show_progress: bool = True, num_workers: Optional[int] = None) -> Dict[str, Any]

Process all LAZ files in a HUC folder.

Parameters:

Name	Type	Description	Default
`huc_dir`	`Path`	Input HUC directory containing LAZ files	required
`output_base_dir`	`Path`	Base output directory (will create huc_id subfolder)	required
`skip_existing`	`bool`	If True, skip files that already have output files	`False`
`show_progress`	`bool`	If True, show progress bar (if tqdm available)	`True`
`num_workers`	`Optional[int]`	Number of parallel workers (None = use CPU count, 1 = sequential)	`None`

Returns:

Type	Description
`Dict[str, Any]`	Dictionary with processing statistics

main ¶

main() -> None

Main entry point with command-line argument parsing.

CLI Arguments¶

Argument	Default	Description
`--input-dir`	`./data/ml-data/silver_training`	Input directory with HUC-organized LAZ/LAS files
`--output-dir`	`./data/ml-data/silver_training_normalized`	Output directory for `.npy` and `.json`
`--skip-existing`	False	Skip if `.npy` + `.json` already exist
`--hucs`	all	Specific HUC IDs to process
`--workers`	CPU count	Parallel workers per HUC
`--no-progress`	False	Disable progress bars

Usage¶

# Process all HUCs
python src/preprocess_bridges.py

# Process specific HUCs with resume
python src/preprocess_bridges.py --hucs 02050206 03070101 --skip-existing

src.download_and_weak_supervise_hucs ¶

HUC-based Bridge Processing Pipeline.

Pipeline for processing bridge lidar data organized by Hydrologic Unit Code (HUC) regions. This script finds intersecting lidar sources, applies ground filtering and weak supervision rules to generate labeled training data for machine learning models.

Input directory structure:

hucs_dir/
    {huc_id}/
        osm_bridges_lidar_subset__{huc_id}.gpkg

Example

# Process all bridges in all HUCs with default settings
python src/download_and_weak_supervise_hucs.py

# Process bridges in specific HUC regions
python src/download_and_weak_supervise_hucs.py --hucs 01010001 01010002

# Resume processing (skip already processed)
python src/download_and_weak_supervise_hucs.py --skip-existing

# Custom directories and settings
python src/download_and_weak_supervise_hucs.py \
    --hucs-dir ./data/osm/hucs \
    --source-dir ./data/ml-data/source \
    --silver-dir ./data/ml-data/silver_training \
    --buffer 15.0 --workers 8

LidarSourceFinder ¶

Class for finding intersecting lidar sources. Loads lidar_resources.geojson and builds spatial index for fast queries.

init ¶

__init__(lidar_resources_path: str) -> None

Initialize the finder by loading lidar resources and building spatial index.

find_intersecting_sources ¶

find_intersecting_sources(bridge_geometry: Any, buffer_meters: float = 10) -> List[Dict[str, str]]

Find all lidar sources that intersect with the buffered bridge geometry.

DataManager ¶

Manages file I/O and directory organization.

get_paths ¶

get_paths(huc_id: str, osmid: str, source_name: str) -> Tuple[str, str]

Get file paths for source and silver training outputs.

Returns:

Type	Description
`Tuple[str, str]`	Tuple of (source_path, silver_path) as strings for pdal compatibility

ensure_directories ¶

ensure_directories(huc_id: str) -> None

Create output directories for a HUC if they don't exist.

file_exists ¶

file_exists(huc_id: str, osmid: str, source_name: str) -> bool

Check if both source and silver files already exist.

no_points_sentinel_path ¶

no_points_sentinel_path(huc_id: str, osmid: str, source_name: str) -> Path

Path to sentinel file indicating no lidar points were found for this bridge/source.

no_points_sentinel_exists ¶

no_points_sentinel_exists(huc_id: str, osmid: str, source_name: str) -> bool

Check if a no-points sentinel exists (skip on restart with --skip-existing).

write_no_points_sentinel ¶

write_no_points_sentinel(huc_id: str, osmid: str, source_name: str) -> None

Write sentinel so this (huc_id, osmid, source_name) is skipped on restart with --skip-existing.

timeout_sentinel_path ¶

timeout_sentinel_path(huc_id: str, osmid: str, source_name: str) -> Path

Path to sentinel file indicating this bridge timed out during processing.

timeout_sentinel_exists ¶

timeout_sentinel_exists(huc_id: str, osmid: str, source_name: str) -> bool

Check if a timeout sentinel exists (skip on restart with --skip-existing).

write_timeout_sentinel ¶

write_timeout_sentinel(huc_id: str, osmid: str, source_name: str) -> None

Write sentinel so this timed-out bridge is skipped on restart with --skip-existing.

save_files ¶

save_files(original_arrays: Any, modified_arrays: Any, huc_id: str, osmid: str, source_name: str, config: BridgeProcessingConfig) -> bool

Save both source and silver training files.

Parameters:

Name	Type	Description	Default
`original_arrays`	`Any`	Arrays after SMRF, before weak supervision classification	required
`modified_arrays`	`Any`	Arrays after weak supervision classification	required
`config`	`BridgeProcessingConfig`	BridgeProcessingConfig instance	required

Returns:

Type	Description
`bool`	True if successful, False otherwise

save_source_only ¶

save_source_only(original_arrays: Any, huc_id: str, osmid: str, source_name: str, config: BridgeProcessingConfig) -> bool

Save only the source training file (e.g. when silver is rejected).

Parameters:

Name	Type	Description	Default
`original_arrays`	`Any`	Point arrays to write (e.g. raw or SMRF output)	required
`huc_id`	`str`	HUC identifier	required
`osmid`	`str`	OSM bridge ID	required
`source_name`	`str`	Source name for filename	required
`config`	`BridgeProcessingConfig`	BridgeProcessingConfig instance	required

Returns:

Type	Description
`bool`	True if successful, False otherwise

BridgeProcessor ¶

Main orchestrator class for processing bridges across HUCs.

find_huc_files ¶

find_huc_files(huc_ids: Optional[List[str]] = None) -> List[Tuple[str, str]]

Find all HUC GPKG files.

Returns:

Type	Description
`List[Tuple[str, str]]`	List of (huc_id, file_path) tuples where file_path is a string

load_bridges ¶

load_bridges(gpkg_path: str, osm_ids: Optional[List[str]] = None) -> GeoDataFrame

Load bridges from GPKG file, optionally filtered by OSM IDs.

generate_tasks ¶

generate_tasks(huc_ids: Optional[List[str]] = None, osm_ids: Optional[List[str]] = None) -> List[TaskTuple]

Generate all (bridge, source) tasks for processing in parallel per HUC.

Returns:

Type	Description
`List[TaskTuple]`	List of task tuples for process_bridge_source function

process ¶

process(huc_ids: Optional[List[str]] = None, osm_ids: Optional[List[str]] = None, show_progress: bool = True, shuffle_seed: Optional[int] = None) -> None

Main processing method with parallel execution.

process_bridge_source ¶

process_bridge_source(args: TaskTuple) -> Dict[str, Any]

Process a single (bridge, source) pair. This function is called by worker processes.

Parameters:

Name	Type	Description	Default
`args`	`TaskTuple`	Tuple of (huc_id, osmid, bridge_geometry_wkt, source_url, source_name, buffer_meters, source_dir, silver_dir, config_dict)	required

Returns:

Type	Description
`Dict[str, Any]`	Dictionary with keys: 'success' (bool), 'huc_id' (str), 'osmid' (str),
`Dict[str, Any]`	'source_name' (str), 'error' (Optional[str]), 'rmse' (Optional[float]),
`Dict[str, Any]`	'deviation' (Optional[float]), 'skipped' (Optional[bool])

main ¶

main() -> None

Main entry point with command-line argument parsing.

CLI Arguments¶

Argument	Default	Description
`--hucs-dir`	`./data/osm/hucs`	Directory containing per-HUC GPKG files
`--source-dir`	`./data/ml-data/source`	Output directory for raw LAZ downloads
`--silver-dir`	`./data/ml-data/silver_training`	Output directory for weak-supervised LAZ
`--lidar-resources`	`./data/usgs_entwine/lidar_resources.geojson`	USGS EPT source index
`--hucs`	all	Space-separated HUC IDs to process
`--osm-ids`	all	Space-separated OSM IDs to process
`--buffer`	10.0	Bridge geometry buffer in meters
`--workers`	CPU count	Parallel worker processes
`--skip-existing`	False	Skip bridges already processed
`--bridge-timeout`	300	Seconds before a hung bridge is skipped
`--shuffle-seed`	None	Seed for task shuffle (reproducible order for debugging)
`--results-csv`	None	Write per-bridge results to this CSV path
`--log-dir`	`./logs`	Directory for processing logs
`--no-progress`	False	Disable tqdm progress bars

Usage¶

# Process all HUCs
python src/download_and_weak_supervise_hucs.py

# Process specific HUCs with resume
python src/download_and_weak_supervise_hucs.py \
    --hucs 02050206 03070101 \
    --skip-existing

# Custom output paths and timeout
python src/download_and_weak_supervise_hucs.py \
    --source-dir /data/source \
    --silver-dir /data/silver \
    --bridge-timeout 600 \
    --results-csv results.csv

src.download_and_weak_supervise_demo ¶

Bridge Weak Supervision Demo Script.

Standalone demo script that processes a small set of target bridges from a single lidar dataset. Uses the shared weak_supervision module for the core algorithm (RANSAC plane fitting, linearity checks, Z-distance classification).

This script is a simplified, single-dataset version of the full HUC-based pipeline (download_and_weak_supervise_hucs.py). It is useful for quickly testing the weak supervision algorithm on a handful of known bridges.

Example

python src/download_and_weak_supervise_demo.py

Weak Supervision Pipeline¶

src.weak_supervision ¶

FailureReason ¶

NO_POINTS class-attribute instance-attribute ¶

RANSAC_INSUFFICIENT class-attribute instance-attribute ¶

RANSAC_FAILED class-attribute instance-attribute ¶

RANSAC_LOW_INLIERS class-attribute instance-attribute ¶

HULL_FAILED class-attribute instance-attribute ¶

HIGH_RMSE class-attribute instance-attribute ¶

CURVED class-attribute instance-attribute ¶

EXCEPTION class-attribute instance-attribute ¶

TIMEOUT class-attribute instance-attribute ¶

BridgeProcessingConfig dataclass ¶

__post_init__ ¶

from_dict classmethod ¶

to_dict ¶

check_bridge_linearity ¶

fit_ransac_from_arrays ¶

process_bridge ¶

src.preprocess_bridges ¶

process_laz_file ¶

process_laz_file_worker ¶

process_huc_folder ¶

main ¶

CLI Arguments¶

Usage¶

src.download_and_weak_supervise_hucs ¶

LidarSourceFinder ¶

__init__ ¶

find_intersecting_sources ¶

DataManager ¶

get_paths ¶

ensure_directories ¶

file_exists ¶

no_points_sentinel_path ¶

no_points_sentinel_exists ¶

write_no_points_sentinel ¶

timeout_sentinel_path ¶

timeout_sentinel_exists ¶

write_timeout_sentinel ¶

save_files ¶

save_source_only ¶

BridgeProcessor ¶

find_huc_files ¶

load_bridges ¶

generate_tasks ¶

process ¶

process_bridge_source ¶

main ¶

CLI Arguments¶

Usage¶

src.download_and_weak_supervise_demo ¶

NO_POINTS `class-attribute` `instance-attribute` ¶

RANSAC_INSUFFICIENT `class-attribute` `instance-attribute` ¶

RANSAC_FAILED `class-attribute` `instance-attribute` ¶

RANSAC_LOW_INLIERS `class-attribute` `instance-attribute` ¶

HULL_FAILED `class-attribute` `instance-attribute` ¶

HIGH_RMSE `class-attribute` `instance-attribute` ¶

CURVED `class-attribute` `instance-attribute` ¶

EXCEPTION `class-attribute` `instance-attribute` ¶

TIMEOUT `class-attribute` `instance-attribute` ¶

BridgeProcessingConfig `dataclass` ¶

from_dict `classmethod` ¶

init ¶