Technical project documentation

Inside the Driver Monitoring System.

A notebook-backed explanation of the datasets, three model pipelines, image preparation, detection merging, model results, deployment architecture, and people behind this academic computer vision project.

Run the Project Explore Models View Source
3trained model pipelines
13source labels across YOLO
2runtime deployment modes
Driver Monitoring System architecture The input image is preprocessed and sent to two browser YOLO detectors and one local CNN. YOLO detections pass through NMS while the CNN classification joins the final report separately. MULTI-MODEL INFERENCE ARCHITECTURE Input image JPG or PNG Preprocessing letterbox + normalization YOLO11n YOLOv8n Keras CNN 8 detection classes 5 detection classes 5 classification classes DETECTIONS NMS + class mapping merge overlapping YOLO detections Unified result detections + CNN class LOCAL CNN CLASSIFICATION DEPLOYMENT RULE Vercel YOLO models run in the browser Local Flask YOLO models + CNN API Green: image flow · Blue: model outputs

Project guide

Everything needed for a report or viva.

The page follows the complete journey from training data to final result, with the exact settings and metrics stored in this repository.

System architecture

Detection and classification serve different purposes.

YOLO predicts a behavior class, confidence score, and bounding box for each detected region. The CNN classifies the entire image into one behavior category. Combining them gives a richer academic comparison between object detection and image classification.

End-to-end inference flowThe uploaded image is prepared, sent to parallel model branches, post-processed, and displayed as a result. UploadedimageImagepreparationSoham YOLO11nChaitanya YOLOv8nMishra CNNConfidence filterNMS + mappingVisual resultand report one or many filesresize + normalizeONNX Runtime WebONNX Runtime WebTensorFlow / Flaskscore > 0.25IoU threshold 0.45boxes + confidence

Training layer

Jupyter notebooks inspect datasets, train pretrained nano models, validate performance, and save weights plus charts and prediction artifacts.

Inference layer

Exported ONNX weights run client-side for YOLO. The Keras .h5 model is loaded only by the local Flask server.

Presentation layer

The browser maps model-specific labels into shared behavior names, suppresses duplicate boxes, and builds an understandable result report.

Dataset documentation

Two labeled detection datasets and one directory-based CNN dataset.

Each YOLO dataset uses images paired with bounding-box labels and defines separate training, validation, and test folders. Both YOLO dataset configurations identify their source as Roboflow Universe under CC BY 4.0.

Soham YOLO11 dataset

Roboflow project: Driver Monitoring System, version 1. The dataset has eight classes and was reviewed for class imbalance before training.

Training imagestrain/images
Validation imagesvalid/images
Test imagestest/images
Annotation formatYOLO bounding boxes
LicenseCC BY 4.0
Open Soham dataset source →

Chaitanya YOLOv8 dataset

Roboflow project: Abnormal Driver Behaviour, version 1. Its five classes focus on observable objects and activities around the driver.

Training imagestrain/images
Validation imagesvalid/images
Test imagestest/images
Annotation formatYOLO bounding boxes
LicenseCC BY 4.0
Open Chaitanya dataset source →

Divyanshu CNN dataset

The CNN notebook loads Multi-Class Driver Behavior Image Dataset with Keras image_dataset_from_directory. Folder names become class labels automatically. Corrupt images are removed, images are resized to 128×128, batches contain 32 samples, and a 20% validation split is created with seed 123. The notebook refers to the second split as test_ds, so its reported “test” result comes from this held-out directory split rather than a separately documented external test dataset.

128 × 128 RGBBatch 32Validation split 20%Seed 123Directory labelsCorrupt-file cleanup

Model-by-model analysis

Classes, training strategy, and measured results.

The metrics below are read from the saved final training epoch or notebook evaluation output. Precision describes how often predicted detections were correct; recall describes how many labeled objects were found; mAP summarizes detection quality across classes and confidence thresholds.

Model 01 · Object detector

Soham Jadhav — YOLO11n

A lightweight pretrained YOLO11 Nano detector selected for student hardware. The notebook targets an NVIDIA RTX 3050 Laptop GPU with 4 GB VRAM and includes dataset checks, class-count visualization, imbalance handling, training, and saved evaluation artifacts.

Browser + local

Eight training classes

DistractedDriver attention appears directed away from normal driving.
DrinkingA drinking action or beverage is detected near the driver.
DrowsyVisual cues associated with sleepiness or reduced alertness.
EatingFood consumption activity around the driver.
PhoneUseMobile-phone interaction, mapped to “Phone Usage” in the app.
SafeDrivingNormal driving state; excluded from cross-model warning-box merging.
SeatbeltVisible seatbelt usage across the driver’s body.
SmokingSmoking-related action or object near the driver.

Saved training configuration

Base weightsyolo11n.pt
Epochs / batch50 / 16
Input size640 × 640
OptimizerAdamW
Learning scheduleCosine LR
AugmentationMosaic 1.0, mixup 0.1
OtherAMP, deterministic seed 0
Open YOLO11 training notebook →

Final epoch detection metrics

Precision91.18%
Recall83.43%
mAP@0.5090.68%
mAP@0.50:0.9565.92%

Why this training setup matters

  • Nano model: lowers memory and inference cost for laptops and browser export.
  • Mixup and cosine LR: were intentionally enabled after the notebook identified strong class imbalance.
  • Close mosaic: mosaic augmentation is disabled for the final 10 epochs to refine training on natural images.
  • Saved artifacts: include results CSV, curves, confusion matrices, validation batches, and best/last weights.

Model 02 · Object detector

Chaitanya Kulkarni — YOLOv8n

A compact YOLOv8 Nano detector trained for five visible driver-related activities and objects. It complements YOLO11 by providing a second independently trained source for drinking, eating, phone, seatbelt, and smoking-related detections.

Browser + local

Five training classes

CigaretteSmoking-related object, mapped to “Smoking” in the unified report.
DrinkingBeverage or drinking action associated with the driver.
EatingFood or an eating action inside the driver region.
PhoneMobile phone activity, mapped to “Phone Usage” in the app.
SeatbeltVisible restraint crossing the driver’s upper body.

Saved training configuration

Base weightsyolov8n.pt
Epochs / saved batch50 / 16
Input size640 × 640
OptimizerUltralytics auto
AugmentationMosaic 1.0, mixup 0.0
ValidationEnabled every run
OtherAMP, deterministic seed 0
Open YOLOv8 training notebook →

Final epoch detection metrics

Precision81.15%
Recall84.41%
mAP@0.5083.07%
mAP@0.50:0.9556.17%

How to interpret the comparison

YOLO11 has higher final precision and mAP in its own eight-class dataset, while YOLOv8 has slightly higher recall in its five-class dataset. These are not direct leaderboard scores because the models were trained on different datasets and class sets. In the product, their outputs are used as complementary evidence and merged only after label normalization.

Best weights evaluatedValidation splitTest workflowPrediction samples

Model 03 · Whole-image classifier

Divyanshu Mishra — Sequential CNN

A TensorFlow/Keras convolutional neural network that classifies an entire 128×128 driver image. It achieves strong reported notebook accuracy and adds a classification perspective, but it requires TensorFlow and server-side model loading, so it is enabled only when the Flask app runs locally.

Local Flask only

Five CNN classes

other_activitiesNotebook training folder for activities outside the four named driving classes. The app currently presents this output as “Smoking/Drinking/Yawning”.
safe_drivingNormal driving posture without a targeted distraction behavior.
talking_phoneDriver holding or using a phone for a call.
texting_phoneDriver interacting with a phone for texting or screen input.
turningDriver turning the head or upper body away from the forward position.
Training label versus display label

The notebook’s first class is other_activities. The local API assigns the friendlier but narrower label “Smoking/Drinking/Yawning” to that output index. This should be described as an interface alias, not a separately trained three-action class.

CNN layer architectureThe image passes through three convolution and pooling stages, then flattening, dense, dropout, and softmax layers.SEQUENTIAL CNN ARCHITECTURE128 × 128 × 3 RGB imageConv2D 32, 3×3, ReLUMaxPooling2D 2×2Conv2D 64, 3×3, ReLUMaxPooling2D 2×2Conv2D 128, 3×3, ReLUMaxPooling2D 2×2Flatten → Dense 128 ReLUDropout 0.5Dense 5 + Softmax

Notebook training and result

Input / batch128 × 128 / 32
Training epochs15 configured
Optimizer / lossAdam / sparse categorical crossentropy
Output activationSoftmax
Reported test accuracy91.20%
Reported test loss0.3044
Open CNN training notebook →
Accuracy needs context

The saved classification report contains 1,455 evaluated images and approximately 91% overall accuracy, but only texting_phone (116 samples) and turning (1,339 samples) have non-zero support. The other three classes have zero support in that output. This means the result demonstrates strong performance on the represented split, especially turning, but it does not establish balanced 91% performance across all five classes.

Image preprocessing

Preparing one image for two different input formats.

Models only accept tensors with fixed dimensions and numeric ranges. The app preserves the uploaded image’s aspect ratio for YOLO, records the scale and padding, and later uses those values to translate predicted boxes back to the original image.

1

Decode image

The browser reads JPG/PNG files and creates an image element. Local CNN requests decode a Base64 data URL with Pillow.

2

Letterbox

YOLO scales the image to fit inside 640×640 without distortion and fills unused space with neutral gray.

3

Normalize

RGB byte values from 0–255 are divided by 255 to produce floating-point values from 0–1.

4

Build tensor

YOLO pixels are reordered into NCHW shape [1,3,640,640]. CNN uses NHWC shape [1,128,128,3].

5

Restore boxes

YOLO output coordinates have padding offsets removed and are divided by the saved scale to match the original image.

Letterbox resizingA wide image is centered in a square canvas with gray padding above and below.ASPECT-RATIO-PRESERVING LETTERBOXOriginal image640 × 640 model canvasgray padding stores offsetY

Exact transformations in the app

scale = min(640 / width, 640 / height)
offsetX = (640 - width × scale) / 2
offsetY = (640 - height × scale) / 2
  • Canvas background: #808080.
  • Channel order: red plane, green plane, blue plane.
  • YOLO input data type: 32-bit floating point.
  • CNN local resize: direct 128×128 resize, then divide by 255.
  • Bounding boxes are clamped to the original image boundaries.

Post-processing

How Non-Maximum Suppression removes duplicate boxes.

YOLO can propose several boxes around the same activity. Non-Maximum Suppression (NMS) keeps the strongest candidate and removes nearby duplicates, producing a cleaner report. In this app NMS is applied after both model outputs are mapped into shared class names.

NMS before and after visualizationOverlapping phone detection boxes are reduced to the highest-confidence box.BEFORE NMSAFTER NMS0.620.780.91Phone Usage 0.91The 0.91 box survives; lower-confidence overlapping boxes are suppressed.

The app’s NMS procedure

  1. Discard model candidates whose best class confidence is not greater than 0.25.
  2. Translate model-specific labels into shared labels: for example, Phone and PhoneUse both become Phone Usage.
  3. Group boxes by their unified class and sort each group from highest to lowest confidence.
  4. Keep the highest-confidence box, then compare it with the remaining boxes using Intersection over Union.
  5. Suppress a box when IoU is greater than 0.45. The app also treats two boxes as overlapping when either box’s center lies inside the other.
  6. Repeat until no candidates remain, then send the surviving boxes to the result page.
IoU = intersection area ÷ union area
Cross-model merging detail

NMS operates per unified class across detections from both YOLO models. This means two models predicting the same behavior in the same region compete on confidence. Soham’s SafeDriving output is intentionally skipped during the final warning-detection merge so it does not suppress or dilute actionable behavior detections.

Deployment architecture

Vercel stays lightweight; local mode adds TensorFlow.

The environment variable VERCEL=1 separates static cloud deployment from full local execution. This keeps the hosted project within a browser-compatible stack while preserving the CNN model for demonstrations on a capable local machine.

Vercel deployment

The static site loads both exported ONNX YOLO models in the browser through ONNX Runtime Web. Images remain client-side for those detections. TensorFlow and the .h5 CNN are not installed or loaded.

YOLO11 ONNXYOLOv8 ONNXBrowser inferenceNo CNN

Local Flask execution

The browser still runs both YOLO models. In addition, the UI calls /api/cnn; Flask lazily loads models/mishra/driver_behavior_cnn.h5, preprocesses the image, and returns the predicted class and confidence.

YOLO11 ONNXYOLOv8 ONNXKeras CNNTensorFlow backend
View Flask backend source →

Responsible interpretation

Useful for education, not for real vehicle decisions.

This is an academic demonstration designed for mini-projects, final-year presentations, AI labs, and viva explanations. Its metrics describe held-out project data, not every road, camera, driver, or lighting condition.

Dataset shift

New camera positions, nighttime footage, occlusion, image quality, clothing, and vehicle interiors can differ from training data and reduce accuracy.

Class imbalance

Rare behaviors may have less training support. The YOLO11 notebook addresses imbalance with augmentation, while the CNN report exposes an uneven evaluation split.

Single-image scope

The current product analyzes uploaded still images. It does not model temporal behavior, gaze duration, repeated warnings, vehicle telemetry, or real-time safety intervention.

Academic-use notice

Do not use this project for road-safety decisions, commercial driver scoring, surveillance, insurance decisions, production ADAS, or any situation where a false positive or false negative could affect a person’s safety or rights.

Team and contributions

Built through the Skills4Future Program.

Developed under the Advanced Course on Green Skills and AI (Skills4Future Program), with separate ownership for model training and application infrastructure.

CK

Chaitanya Kulkarni

YOLOv8 model training, five-class driver behavior detector, validation, and model export workflow.

SJ

Soham Jadhav

YOLO11 model training, eight-class detector, class-imbalance analysis, augmentation, and evaluation artifacts.

DM

Divyanshu Mishra

CNN model training, Keras classification architecture, confusion matrix, classification report, and local model capability.

AP

Anurag Pawar

Backend logic and server infrastructure for local inference and advanced deployment scenarios.

Additional models and backend

Divyanshu Mishra trained the CNN-based model that achieves excellent accuracy on the represented evaluation data but is not browser-compatible in this project due to TensorFlow and computational requirements. It can be run locally for enhanced classification capability.

Anurag Pawar developed the backend logic and server infrastructure used for advanced deployment scenarios and local CNN inference.

For complete model access, notebooks, saved training results, backend implementation, and deployment files, visit the project’s GitHub repository.

Explore the complete implementation on GitHub.

Review the training and testing notebooks, datasets, YOLO and CNN weights, ONNX browser inference, Flask backend, Vercel configuration, and project history.

Open Repository →