Coachless reports win probability added for choices like runes, but the figure is calculated from observed win rates and a model prediction. Does that label risk making the numbers sound more causal or objective than they really are?
Does Coachless make win probability added sound more definitive than it is?
Answers
That seems like a fair concern. If a rune has a 3.4 percentage point advantage in the data, readers might take that to mean switching runes will raise their own chance of winning by 3.4 points. The data alone doesn't establish that.
Players choosing different runes may also differ in experience, matchups, or other ways that affect their results. A model can try to adjust for those differences, but the label doesn't tell you how well it succeeds.
Reply to this answer
I wouldn't dismiss the metric just because it uses win rates. The useful question is what the model predicts, which variables it accounts for, and how well those predictions hold up. Subtracting a prediction from an observed rate can be informative, but it doesn't automatically remove selection bias.
For the reported Bel'Veth example, the roughly 3.4 point difference is a reason to investigate, not proof that the health rune itself causes that much improvement. Even a large sample won't fix a systematic difference between the players choosing each option.
Reply to this answer
The presentation matters as much as the calculation here. A short explanation beside the number saying it's a model-adjusted association, not a guaranteed gain from switching, would make the result easier to interpret. Readers shouldn't need to find a thesis just to understand that distinction.
Reply to this answer
Add your answer
Answers are reviewed before publication.
XPsplit~Unclaimed:
That gap between the displayed number and what a player might expect from switching is the part I'm wondering about.