Coherent LabMethod

Calibration desk / 01

Where do your
reviewers disagree?

Put two sets of labels side by side. Measure agreement and bring the disputed items to your next calibration session.

01 / Input

Paired ratings

Item ID, reviewer A, reviewer B. Tabs separate the columns. Leave a missing rating blank.

Ratings stay in this tab until you export them.

02 / Results

Agreement report

Run a comparison to see how often reviewers match and which labels they mix up.

Field notes

Read the matrix before the score.

Each row is a label from reviewer A. Each column is a label from reviewer B. The diagonal shows matching labels. Use the off-diagonal counts to find where your labeling guide needs a clearer example.

Raw agreement is the fraction of complete pairs that match. Cohen’s kappa adjusts for the agreement expected from each reviewer’s label distribution. If both reviewers give every item the same label, kappa is undefined even though raw agreement is 100%. Use this for unordered labels from two reviewers working independently. It measures consistency between reviewers, with no reference to ground truth.

Method reference: Jacob Cohen, A Coefficient of Agreement for Nominal Scales.