David M. W. Powers - resources relating to the bookmaker algorithm. The original Bookmaker paper (6 pages) and poster (A4) derives Informedness from the idea of an edge in gambling or trading.
The later ECAI 2008 paper (2 pages) and poster (A4) introducing/comparing Markedness and Informedness versus related measures, and summarizing key Correlation, Significance and Confidence Interval formulae from Technical Report SIE-07-001:
56409 ECAI-Evaluation_Evaluation-Short.pdf – ECAI short
paper, July 2008 (2p)
467514 ECAI-Evaluation Evaluation-poster-A4.pdf – ECAI tutorial poster, July 2008 (1p)
296599 ECAIacc-Evaluation_Evaluation.pdf – 5p version of ECAI paper, July 2008
133242 ECAIrej-Significance_Confidence.pdf – 5p sequel to ECAI paper, July 2008
Excel spreadsheets (as discussed in ICCS paper and usable in doc version):
84480 BMExcel.xls – 2x2 case, 3x3 case,
27136 bmsig.xls - shows 2x2 case + significance estimates
27136 bmsmall.xls - shows 2x2 case + mean F&G factors
28160 bmsym.xls - shows 2x2 case + misinformedness case
29184 bmtriple.xls - shows 3x3 case + mean F&G factors
28672 bmwtsym.xls - shows 2x2 case + weighted F&G factors
Brief motivation Powerpoint (abstract as slide 5) motivating Informedness, Markedness and showing the connection to Correlation and Chi-squared Significance (HCSNet 2007, Abstracts p77 and Speedpapers p 29):
Technical Report SIE-07-001 showing full derivation and analysis of Informedness, Markedness and relating them to Recall, Precision, Correlation and Chi-squared Significance (draft to be submitted) as well as to ROC analysis (Receiver Operating Characteristics), AUC (Area under the curve), DeltaP, Regression, etc.
In summary, Precision reflects at chance level performance the Prevalence of the positive case in the dataset, and subtracting off the Prevalence and renormalizing as a probability gives the probability of an informed prediction (versus guessed prediction) – in the binary case this corresponds to DeltaP’ or to 2AUC-1. Conversely, Recall reflects at chance level performance the Bias towards positive labels by the predictor, and subtracting off the Bias and renormalizing as a probability gives the probability of a marked prediction (versus chance association) – in the binary case this corresponds to DeltaP. The Geometric Mean of Informedness and Markedness is the Pearson Correlation. All three can be regarded as different normalizations of the Chi-squared statistic.