Mining Perception: YOLO26n BDD Road-Camera Detector

A prototype object-detection model developed as part of a perception pipeline for autonomous mining trucks, loaders, and excavators.

The broader project explores two visual domains:

• Aerial/mining imagery: an initial single-class detection experiment.
• Vehicle-mounted road-camera imagery: a BDD100K-based experiment intended to better match an onboard camera viewpoint.

Checkpoint scope: The BDD checkpoint in this repository was fine-tuned independently from pretrained YOLO26n weights. It was not jointly or sequentially trained on both the aerial/mining dataset and BDD100K.

Intended Role

The detector is intended to support an experimental perception-to-planning pipeline:

Camera image
    ↓
Object detection
    ↓
Depth/LiDAR association and calibrated localization
    ↓
World-coordinate obstacle tracking
    ↓
Obstacle observations for a candidate-trajectory PPO planner

The model outputs image-space bounding boxes, class predictions, and confidence scores. It does not directly output:

• Metric obstacle distances.
• World-coordinate positions.
• Object velocities.
• Drivable-area segmentation.
• Collision-free paths.
• Steering or braking commands.

These require additional perception, calibration, tracking, and planning components.

Training Experiments

Experiment 1: Aerial/Mining Imagery

An initial mining-image experiment used images and LabelMe rectangle annotations supplied through a Modal volume.

The preparation process:

• Paired images with their annotation files.
• Audited source labels.
• Excluded images containing unapproved or ambiguous labels.
• Retained images annotated only with source label "1".
• Converted rectangles into YOLO-format bounding boxes.

The semantic meaning of source_label_1 was not established from the supplied annotations.

Numerical evaluation results for this experiment are not included here because they were not supplied in the available training logs.

The uploaded BDD checkpoint should not be described as containing this experiment’s learned weights.

Experiment 2: BDD Road-Camera Imagery

The uploaded detector was fine-tuned on a small subset of:

ShantyCam/bdd100k-det

Source revision:

d775466059d07da240a4137c8cec93deea11ceed

This dataset provides road-camera images and object annotations. The selected images were downloaded individually rather than downloading the entire dataset.

Split Method

This experiment used a custom development split, not the official complete BDD100K benchmark split.

Images sharing the same leading filename prefix were grouped to reduce the risk of related images appearing in both training and validation. This is a grouping heuristic, not a verified substitute for recording-session metadata.

The small validation subset should not be treated as a comprehensive estimate of real-world or mining-site performance.

Evaluation Results

The following results were reported when validating the saved best.pt checkpoint:

These are aggregate metrics as reported by Ultralytics.

Per-Class Validation Results

Only categories included in the supplied validation report are shown below. The checkpoint contains 13 class outputs, but the report lists nine categories represented in this evaluation.

A reported precision of 1.000 together with zero recall does not indicate reliable detection. Metrics for rare classes are particularly unstable in this small validation set.

Training Losses

The training log reported the following loss components:

Separate validation-loss values were present in the supplied logs and are therefore not reported. The evaluation results above are detection metrics, not loss values.

Inference Speed

The best-checkpoint validation log reported:

These measurements were collected during validation on an NVIDIA L4.

They do not include camera transport, depth association, tracking, planning, or control. They should not be interpreted as an end-to-end deployment latency guarantee or extrapolated directly to other hardware.

Usage

Install Ultralytics and the Hugging Face Hub client in an environment with a compatible PyTorch installation:

pip install ultralytics==8.4.179 huggingface_hub

Download and run the model:

from huggingface_hub import hf_hub_download
from ultralytics import YOLO

weights = hf_hub_download(
    repo_id="pi-hack-mining/bdd-yolo26n-road-detector",
    filename="bdd_yolo26n_best.pt",
)

model = YOLO(weights)

# Always inspect the actual checkpoint label mapping.
print(model.names)

results = model.predict(
    source="camera_frame.jpg",
    imgsz=640,
    conf=0.25,
)

results[0].save(filename="prediction.jpg")

If the uploaded checkpoint is named best.pt, replace the download filename accordingly.

For a private repository, authenticate with Hugging Face before downloading.

Only load checkpoints from sources you trust.

Relationship to the Six-Class Mining Taxonomy

The desired future mining taxonomy is:

CLASS_NAMES = [
    "vehicle",
    "digger",
    "car",
    "loader",
    "person",
    "stone",
]

This checkpoint is not a trained six-class mining detector.

Some BDD predictions can be mapped into a downstream planner vocabulary:

Other categories require an explicit application policy.

The current model has no verified dedicated capability for:

• Diggers/excavators.
• Loaders.
• Stones or rock obstacles.

Renaming output labels does not create those capabilities. A genuine six-class model requires appropriately labelled examples, training, and per-class evaluation.

PPO Planner Integration

The planned integration keeps the existing PPO observation structure:

• obs: 79 values.
• cands: 44 × 33 candidate features.
• mask: 44 candidate-validity values.

A separate adapter must:

1. Detect objects in RGB images.
2. Associate valid depth or LiDAR measurements.
3. Transform observations into world coordinates.
4. Track object positions and velocities over time.
5. Estimate obstacle footprints and uncertainty.
6. Update obstacle-dependent ray observations, candidate clearance, and safety checks.

The detector should initially run in shadow mode, without changing vehicle controls.

Preserving PPO input dimensions does not guarantee that a policy trained with ideal simulator observations will perform reliably with noisy or missing perception. That must be tested.

Safety and Intended Use

Suitable uses include:

• Research and educational experiments.
• Detector integration tests.
• Visualization and offline analysis.
• Shadow-mode evaluation in simulation.
• A starting point for additional fine-tuning.

This model is not validated for safety-critical autonomous vehicle operation.

Do not use it as the sole input for collision avoidance. Retain independent obstacle sensing, stale-data handling, conservative collision checks, and a validated stop mechanism.

Future Work

• Verify and improve source-label consistency.
• Expand and balance the training dataset.
• Add verified negative/background examples.
• Collect labelled Unity and real mining-camera imagery.
• Train the intended six-class detector.
• Evaluate on independent episodes, sites, and recording sessions.
• Add calibrated depth association and world-frame tracking.
• Evaluate PPO behavior under perception noise, latency, and missed detections.

Licensing and Data Provenance

Review the applicable Ultralytics model/software terms and the original BDD100K dataset terms before redistribution or commercial deployment.

The license metadata of a third-party dataset mirror should not be treated as sufficient permission for all underlying assets.

This repository does not claim to redistribute the original mining or BDD100K image datasets.