MSc thesis · interactive demonstration
Multimodal driver behaviour evaluation
A telematics box can see harsh braking and speeding, but it cannot see why — a driver who brakes hard to avoid a child looks identical to a driver who brakes hard because they were distracted. Adding the camera changes the verdict.
That is the whole idea behind my MSc thesis, “A Multimodal Driver Behaviour Evaluation System: Combining Telematics, VLMs and LLMs”. The demonstration below walks through how the method works, then lets you run three representative scenarios stage by stage and see the telematics-only score compared with the score once the camera evidence is added. The public thesis repository is here.
How the method works, stage by stage
Each stage below says what went in, what came out, and what it changed about the verdict. Select a scenario further down to reveal the stages with that scenario’s data.
-
1. Telematics trace
What went in: the raw event stream from the telematics box. What came out: counts of harsh manoeuvres, speeding and fatigue events, aggregated into episodes. What it changed: nothing yet — this is the telematics-only view, the 'base' score.
Select a scenario to reveal this stage.
-
2. VLM observation
What went in: anonymised dashcam frames (faces blurred by an InsightFace step). What came out: a vision-language model's description of what is actually happening — following distance, pedestrians, lighting, road surface. What it changed: it can confirm, refine, contradict or add to the telematics events.
Select a scenario to reveal this stage.
-
3. LLM risk narrative
What went in: the combined telematics + VLM signal. What came out: a plain-language risk narrative. What it changed: it turns the numbers into a reason a fleet manager can act on.
Select a scenario to reveal this stage.
-
4. Risk score (base vs VLM)
What went in: the same scoring weights, applied twice — once to telematics only (base) and once with the camera evidence folded in (VLM). What came out: two scores, their delta, and the driver's rank under each. What it changed: this comparison IS the thesis contribution — you can see the camera move a driver up or down the fleet ranking.
Select a scenario to reveal this stage.
The point: the camera changes the ranking
Every driver is scored twice with the same weights — once from telematics alone (ubi_proxy_score_base) and once with the camera evidence folded in (ubi_proxy_score_vlm). The difference between them is the thesis contribution. Here are three real rows from the public output file final_monthly_rankings_vlm.csv (January 2026 window):
| Vehicle | Score (base) | Score (VLM) | Score delta | Rank (base) | Rank (VLM) | Rank delta | Camera coverage |
|---|---|---|---|---|---|---|---|
| 429570713 | 0.4143 | 0.3947 | -0.0196 | 166 | 205 | -39 | 100.0% |
| 456979367 | 0.4756 | 0.5284 | +0.0529 | 124 | 90 | +34 | 8.3% |
| 507451763 | 0.5691 | 0.5231 | -0.0460 | 64 | 94 | -30 | 3.0% |
Read the top row as: “the camera moved this driver down 39 places in the fleet ranking.” A positive delta means the camera found the driver more at risk than the telematics alone; a negative delta means it found them less at risk. covered_ratio is the share of that driver’s clips the camera actually saw — the first row is fully covered, the others only partly, which is itself part of the story.
The scoring weights (real, from the public repo)
These are the actual UBI-proxy weights used in the thesis, taken from the “SCORING WEIGHTS” cell of notebook 3. They sum to 1.00.
| Behaviour category | Weight | Why |
|---|---|---|
| Harsh manoeuvres (episode-based) | 20% | Cornering, braking, acceleration — the leading cause of fleet accidents |
| Long speeding episodes (>1 event) | 18% | Sustained speeding suggests deliberate risk-taking |
| Driver distraction (phone, seatbelt) | 13% | Phone use and seatbelt non-compliance raise injury severity |
| Short speeding episodes (single event) | 12% | Isolated speed events — lower severity than sustained speeding |
| Fatigue (eye closed, yawn, fatigue) | 12% | Fatigue-related crashes are disproportionately fatal |
| Situational risk (headway, lane) | 10% | Forward-collision and lane-departure indicators |
| Power violations | 10% | External battery disconnect — tampering or avoidance |
| Camera obstruction | 5% | Monitoring integrity, not a direct driving behaviour |
The method also compares a telematics-only score against a telematics+VLM score and carries score_delta, rank_base, rank_vlm, rank_delta and covered_ratio — exactly the columns shown in the ranking table above.
What the camera can change
When the camera evidence is compared with the telematics events, each clip falls into one of these change types (from the thesis’s own event_change_audit.csv):
| Change type | What it means |
|---|---|
confirmed |
The VLM agreed with what the telematics flagged. |
refined |
The VLM agreed AND found additional events — e.g. telematics said v_phone and the VLM also found SEATBELT_D_OFF. |
new_context_added |
The VLM saw context the telematics had no event for. |
contradicted |
The VLM evidence contradicted the flagged event. |
not_confirmed |
The flagged event could not be confirmed from the camera. |
no_vlm_signal |
No VLM signal was available for the clip. |
Walk through a scenario
Each scenario is a representative, precomputed example — not a real client record. Selecting one reveals the four stages above in order, with that scenario’s data, and shows the base-vs-VLM score change at the end.
What this is — and what it is not
What is real. The scoring weights, the change types and the ranking comparison are taken from my public thesis repository. The ranking rows shown are the thesis’s own published numbers.
What is illustrative. The three scenarios are representative examples. The frame observations are illustrative of what a vision-language model reports; they are not real client records.
No live models. This page is a precomputed demonstration: no model runs when you interact with it, no data leaves your browser, and no API key or backend is used. Notebooks 1 and 2 of the thesis ship without stored outputs and need raw dashcam clips that are deliberately not public; notebooks 3 and 4 run from the processed outputs that are public. The real client data behind the underlying work is confidential and is not shown here.
For the research framing and the literature behind the method, see the Evidence page and the Projects & Modules index.