Ravi's Homepage

Correct Looks Better

Tue, 16 Jun 2026 00:00:00 +0000

Paper · Code · ICML 2026

TL;DR: Pairwise comparison-based ranking reflects the accuracy-based ranking of models on verifiable tasks.

Motivation

When we evaluate large language models, we often care about ranking them in one way or another. If the task is well-defined and we have ground truth labels, we can rank them by accuracy, which reflects how well they perform on the task. However, benchmarks measuring accuracy are not our only option for ranking models. In Chatbot Arena¹ models are evaluated on a diverse set of open-ended tasks using pairwise comparisons. The concern is that pairwise judgments are heavily biased to select the output that appears to be correct rather than selecting the output that is correct ²³⁴⁵. So the question arises:

Can pairwise comparison-based ranking reflect task performance?

We set out to answer this question by applying pairwise comparisons on a range of benchmarks, and checking whether the resulting ranking aligns with the accuracy-based ranking of the models.

What we found

High alignment with accuracy-based ranking: high rank and score correlation across 5 benchmarks.
Outperforms baseline in weak model setting: using a weak model (a model that’s not good at the task) directly as a judge (LLM-as-judge) is significantly worse than using it to collect pairwise comparisons.
Rankings robust to judge biases: correcting for style ²³ and self-preference bias ⁴ had minor impact on the ranking.
Echo influences judgment on non-discriminative pairs: on pairs where both candidate answers are correct (or both are incorrect), the judge tends to choose answers that do not have any sequence repetition (echo).

Accuracy (x-axis) vs. Bradley-Terry score (y-axis). Each point represents a model evaluated on a specific benchmark. Bottom: judge model accuracy.

Main results

Preference-based rankings align closely with accuracy-based rankings across all five benchmarks. We observe both high rank correlation ($ρ > 0.9$) and high score correlation ($r > 0.87$) across all judge models. What this means in practice is that

on 4 out of 5 benchmarks, only 6-9% of model pairs need to be swapped to recover the accuracy-based ranking.

Accuracy-based ranking (left) vs. Bradley-Terry based ranking (right). Bottom: Kendall's distance.

Direct Judge Baseline

Is it better to use the model directly as a judge?

A natural alternative to pairwise comparisons is to use the judge to directly classify each answer as correct or incorrect — this is what we call the Direct Judge baseline. This is a strictly harder task: the judge must assess answers in isolation, without the relative signal a comparison provides. A weak judge that struggles with individual classification may still reliably pick the better of two answers. When the selected judge does well on the task, both the Bradley-Terry and the Direct Judge approach perform well. However,

when the judge is weak, the Direct Judge doubles the rank distance on SimpleQA ($K_D=0.2 \rightarrow 0.382$).

Rank correlation of "strong judge" (03, 59.3% acc.) vs "weak judge" (gpt-oss-20b, 4.9% acc.) on SimpleQA.

In other words, Bradley-Terry outperforms the Direct Judge when the judge model has low accuracy on the task. This robustness may prove useful in frontier settings, where it’s difficult to say whether the selected judge is “weak” or “strong”.

Judge bias

Does rank correlation improve after bias correction?

Style and self-preference biases are well-documented in pairwise evaluation — judges tend to favour longer, more formatted answers (style bias), or outputs from their own model family (self-preference bias). While such biases may affect absolute scores, their effect on relative model rankings is less clear. We corrected for both and found that

bias correction has a minor effect on rank correlation.

Rank correlation after bias correction (original in red). m = number of ranked models.

Importantly, this does not imply the absence of bias — only that such biases have limited impact on model ranking in our evaluation.

Echo

What drives the signal in non-discriminative pairs?

On 60% of pairs, correctness provides no discriminating signal. We call these non-discriminative pairs, where both answers are either correct or incorrect. The judge has to rely on superficial cues. And yet, we find that on such pairs rank correlation is still surprisingly high. One strong cue we identified is echo: a failure mode where a model fails to stop after its final answer and repeats content, such as phrase or sequence repetition.

We found echo to be a causal driver of judge preference: $P(A>B)$ drops from 0.81 to 0.29 ($\Delta=−0.52$) depending on which answer has echo. This effect disappears on discriminative pairs ($\Delta=−0.07$).

Judge preference conditional on echo on BBH. Non-discriminative (left) vs discriminative pairs (right).

Closing thoughts

Of course, it’s important that our model rankings are not influenced by subjective biases. On questions where some answers are objectively better, we found that pairwise comparisons (and the resulting rankings) often indeed reflect the objective choice (picking the model with higher accuracy). This means that on discriminative pairs the judge picks the correct answer. On pairs where this objective choice is not possible, however, the judge falls back to appearance-based cues — it avoids picking answers that contain echo — which are also correlated with higher model accuracy. This doesn’t mean the biases can be ignored: they could still be exploited adversarially to inflate a model’s ranking.

Acknowledgements

Cover image source. Drawing by Mariano Salvador Maella.

W. Chiang, L. Zheng, Y. Sheng, et al. “Chatbot Arena: An open platform for evaluating LLMs by human preference.” ICML 2024. ↩
Y. Dubois, P. Liang, T. Hashimoto. “Length-controlled AlpacaEval: A simple debiasing of automatic evaluators.” ICML 2024. ↩ ↩²
T. Li, A. Angelopoulos, W. Chiang. “Does style matter? Disentangling style and substance in Chatbot Arena.” LMSYS Blog, 2024. ↩ ↩²
K. Wataoka, T. Takahashi, R. Ri. “Self-preference bias in LLM-as-a-judge.” NeurIPS Safe Generative AI Workshop, 2024. ↩ ↩²
G.H. Chen, S. Chen, Z. Liu, F. Jiang, B. Wang. “Humans or LLMs as the judge? A study on judgement bias.” EMNLP 2024. ↩

F-score and G-mean

Mon, 25 Nov 2024 00:00:00 +0000

Everyone should be more-or-less familiar with the following two metrics that we commonly use in Machine Learning: F-score and G-mean. The F-score is defined as follows:

\[F=\frac{2*Pr*Re}{Pr+Re} = \bar{H}(Pr,Re)\]

which is the harmonic mean ($\bar{H}$) of precision ($Pr$) and recall ($Re$). Similarly, the somewhat less-known G-mean is another average of two very commonly used metrics; sensitivity ($Se$) and specificity ($Sp$):

\[\bar{G}(Se, Sp)=\sqrt{Se*Sp}\]

Here, a geometric mean ($\bar{G}$) is used. I couldn’t help but wonder: Why do we use the harmonic mean for precision and recall, while we use the geometric mean for sensitivity and specificity? It turns out, that this choice is not entirely arbitrary. Diving into this question was a useful exercise for me to understand these metrics better.

What are we measuring?

Before going into why we use one definition of mean over the other, let’s take a closer look at what we are actually trying to measure. We will be talking about the following scores:

Precision is the number of true positives (TP), divided by all positive predictions (PP).
Recall (or sensitivity) is the number of true positives (TP), divided by all actual positives (P).
Specificity is the number of true negatives (TN), divided by all actual negatives (N).

And the following averages (for two inputs):

Harmonic mean: $\bar{H}(x_1, x_2) = \frac{2*x_1*x_2}{x_1+x_2}$

Geometric mean: $\bar{G}(x_1, x_2) = \sqrt{x_1*x_2}$

The usual argument

If you search on the internet “Why do we use harmonic mean for F-score?” you will typically find the same (somewhat unsatisfying) answer: if either precision or recall are zero, the mean will be zero too. Or conversely: both precision and recall have to be relatively high for the mean to be high. This is easy to verify just by looking at the numerator of the F-score: multiplying precision and recall have this effect. But wait - we could also just use the geometric mean¹ because it contains the same multiplicative expression. So why does it make (more) sense to use harmonic mean instead?

Harmonic mean preserves units

Typically you will want to use the harmonic mean when you are measuring the rate of something. Precision and recall both count the rate of true positives. The nice thing about applying the harmonic mean is that the result preserves the original unit: it is the rate of true positives over the total number of examined cases. Revisiting the F-score formula:

\[F=\frac{2*Pr*Re}{Pr+Re} = \frac{2TP}{2TP+FP+FN} \quad \left[\frac{\texttt{#TP}}{\texttt{#cases}}\right]\]

The right hand side of the equation becomes clearer when you look at a visual representation of what you are counting. It is the total number of true positive ($2TP$) predictions, over the number of positive samples ($P=TP+FN$) plus the number of positive predictions ($PP=TP+FP$). The result is the rate of true positives among samples that are either positive or predicted positive.

For precision and recall, it just makes sense to choose the harmonic mean over the geometric mean to preserve the original unit because it makes the result more interpretable. Sensitivity and specificity on the other hand measure the rate of two different things: true positives and true negatives. Here, we don’t have a common unit, so using either mean seems appropriate.

No deeper meaning

I don’t want to imply that choosing either mean over the other has a special meaning in Machine Learning. Usually, the average has an important connection to some total value depending on how we define it. ² For G-means, I don’t have an intuitive interpretation for the product of scores like sensitivity and specificity. If you do have in intuition for these total values in the context of Machine Learning, please let me know! But for now my conclusion is that the choice of means is not that deep.

When to use F-score or G-mean?

Notice that the F-score does not factor in the number of true negatives ($TN$). I would say that it is a useful metric when predicting the positive class correctly is more important for some reason. This is typically true for predicting diseases - you care about correctly predicting people with the disease correctly (who may represent 1% of the population). The F-score for the negative class is not meaningful in this highly imbalanced case, since a predictor saying no one is sick would have both very high precision and recall³. If you have an imbalanced dataset, and both classes are equally important, then the G-mean is probably a better choice.

Conclusion

Harmonic mean preserves the unit
- makes sense for F-score (rate of true positives)
- otherwise, no reason to choose one over the other
Use F-score when you care about correctly predicting the positive class
Use G-mean when you care about both classes in an imbalanced dataset

Acknowledgements

Cover image source

Footnotes

In fact, it seems that we do have a metric called the Fowlkes–Mallows index, which is the geometric mean of precision and recall. I personally never heard of it before, but it reinforced my belief that the choice of means is not that important. ↩
I recommend watching this youtube video for a good intuition on means. ↩
Recall would be perfect (1), since everyone is predicted healthy. Precision would be 0.99, because 99% of the population is actually healthy. ↩

Driver fingerprinting

Tue, 17 Nov 2020 00:00:00 +0000

Perhaps it doesn’t come as a surprise that we all have unique driving styles. My father is a calm, sometimes even slow driver. My mother is more temperamental in her driving ways - always cutting the line where she can (this was a highly useful skill when we were late to school). When I close my eyes, I can immediately tell whether my mother or father is behind the wheel.

But so can anybody - using in-vehicle logs. Turns out that these driving styles can translate to bits and bytes that our in-vehicle sensors record.

In-vehicle sensors

There are a number of systems built in our cars for various purposes - to regulate emission, to optimize engine performance or to log and detect malfunctions. Think of your dashboard for example - you can see your speed, consumption and RPM all at a glance! The data required for these comes from in-vehicle sensors. This data is converted and processed by ECUs (Electronic Control Unit), which are essentially small computers inside our cars. The ECUs communicate with each other through the so-called CAN-bus (CAN - Controller Area Network).

The CAN-bus whisperer: OBD dongle

The data sent over the CAN-bus can be easily captured. For example, so-called ”insurance dongles” (or On-Board Diagnostics dongles) can collect and send in-vehicle sensor data to insurance companies. They use this information to give you discounts for example, if you drive safely.

Another similar approach to lowering insurance premiums is using telematics data, as described here. The biggest difference however, is that telematics data are currently subject to data protection laws, unlike CAN logs.

The OBD dongles connect to the ODBII interface of the vehicle. These devices are not expensive at all (prices start at 10$), and can come with mobile apps so you can look at your data from your phone!

In some cases, we cannot access CAN logs directly through an OBD port (or at least not all parts of it). There is usually a gateway regulating access to these data.

Even though CAN-bus data can be readily accessible (and not encrypted), the interpretation of the messages is still a challenge. There is no global standard, and the structure of the messages varies from manufacturer to manufacturer. Without the cooperation of manufacturer, it is hard to reverse engineer these messages… but not impossible! If you want to know how to do it using machine learning techniques, check out my talented colleagues’ paper (Lestyan et al., 2019)!

Driver fingerprinting

So what can we do with the obtained CAN-bus data? Turns out, that we can use it to determine who was behind the wheel.

How it works

Driver fingerprinting or driver re-identification uses the data collected from the CAN logs (called traces) to correctly identify drivers. From these traces, we can extract the data from one sensor to obtain a time series (measuring speed for example).

The input of such a model can consist of multiple time series, like in the example image below. However, to obtain these time series, first we need to reverse-engineer them. Then, we can use Machine Learning to make predictions on the driver’s identity.

For my 2018 Scientific Students’ Association paper I used the measurements of 5 sensors (namely brake, clutch, gaspedal, rpm and speed). Brake, clutch and gaspedal sensors measure the position of the respective pedals. As for the rest: rpm records the revolutions per minute and speed measures the vehicle’s speed. I was also able to work on data collected from 33 drivers, whereas state-of-the-art driver re-identification papers only published results on models that had to distinguish 2-15 drivers. My model achieved a 66% mean accuracy, given a mere 10 second trace as input!

One might think that this time-window can be pushed down even further - theorizing that the distinguishing part in everyone’s driving could be the part where they accelerate or shift gears, which is about 2-3 seconds long. However, when driving in a city there are a lot of situations where one must stand still (in morning traffic, in front of a stoplight, or while letting that old lady cross the street). Related work suggests that all the information needed to identify a driver can be obtained from the measurements of a single turn. This confirms the intuition that someone’s driving can be quite accurately distinguished based on how they slow down, switch gears and accelerate.

After writing this paper me and my colleagues were motivated to extend this research by eliminating the first step: reverse engineering the raw data.

Driver re-identification on raw data

In our paper titled Automatic Driver Identification from In-Vehicle Network Logs we demonstrated that it is possible to achieve the same results without having to reverse engineer and pre-select signals from the CAN log.

We collected 72 time series from unknown sensors and trained a Deep Learning model on each of them. Not all time series were equally good inputs for the re-identification task, which is why we combined the top-10 performing classifiers into one mixture model. We achieved a mean accuracy of 75-85% for CAN traces whose length varied between 20 and 120 seconds.

Great! Acquiring an OBD dongle that can read CAN logs is not only cheap, it turns out you don’t have to understand the data to build a working driver-fingerprinting classifier with it.

Privacy implications

So now that we know that our in-vehicle data could potentially identify us, why should we protect it? I mean, aside from well-wishing insurance companies who give us discounts, who could possibly get our data?

Except if… car companies were motivated to sell CAN logs to 3rd parties. And currently they could do so. Without our consent (because it is not considered to be personal data as of yet). Uh-oh.

Driving data: the car industry’s new cash cow?

Many sources suggest the lucrativity of driving data – McKinzey&Co, a management consulting firm, estimates that ”data from connected cars will be worth up to $750 billion by 2030”. Companies are already preparing for the market for data from cars, which is expected to be even bigger than the market for the cars themselves acording to this Forbes article.

Future ideas include selling data to mapping firms looking to provide more accurate traffic information, or selling consumption-related and other statistics to operators of larger vehicle fleets. And if driver identification is possible, it opens all kinds of doors. For instance, you can use driver identification to notify you any time someone else is driving your car. Or your car could “memorize” your driving fingerprint, and adjust your seat, and radio to your liking as soon as you turn the corner. Neat, right? But wait, something is missing from the list - the biggest cash-grabber of them all: ad revenues. In the hands of 3rd party advertisers, our cars could be yet another gateway to personalised advertisements.

Just like with any other personal information that could potentially identify us: using such data should require our explicit consent!

Because of how new these findings are, it is still unclear whether drivers are completely informed about the wealth of personal information that their CAN logs potentially reveal. Hopefully this post was an eye-opener to the inevitable future where even our driving data will be on the market.

Although car companies stress that the collected network logs are “anonymized”, the proper anonymization of such high-dimensional data is notoriously difficult in practice without significantly degrading accuracy (Aggarwal et al., 2005). Even aggregated metrics aren’t always completely safe (Dinur and Nissim, 2004). The feasibility to identify (and/or profile) a driver means that CAN logs are indeed personal data and, as such, are subject to the European General Data Protection Regulation (GDPR) as of 25 May 2018. It is a fundamental duty of companies handling such data to adequately inform drivers and protect their personal data. Failing to do so does not only ruin customer trust, but can also result in a fine up to 20 million Euros.

Closing thoughts

We have seen that it is fairly easy to access this data, and will probably become even more accessible in the not-so-distant future of connected vehicles.

Massachusets has voted (Nov. 4th, 2020) on standardizing access to car data (to extend the state’s right-to-repair law)! This could set a standard for the vehicles to come…

We have also seen that even without the reverse-engineering of the logs, identifying drivers is still possible. Turns out, our phones aren’t the only devices that store and/or process information which could potentially identify us. :)

I don’t know what driving fingerprint I posess, but I hope I can eventually get the best from both of my parents: my father’s calm (but ultimately safe) driving and my mother’s mad parking skills!

A special thank you to the following people who have provided me with useful feedback and suggestions: Gergely Ács, Andris Gazdag, Ferenc Huszár and Viktor Remeli.

Ravi's Homepage

Correct Looks Better

Motivation

What we found

Main results

Direct Judge Baseline

Judge bias

Echo

Closing thoughts

F-score and G-mean

What are we measuring?

The usual argument

Harmonic mean preserves units

No deeper meaning

When to use F-score or G-mean?

Conclusion

Driver fingerprinting

In-vehicle sensors

The CAN-bus whisperer: OBD dongle

Driver fingerprinting

How it works

Driver re-identification on raw data

Privacy implications

Driving data: the car industry’s new cash cow?

GDPR and awareness

Closing thoughts