More on Reporting Bust Rates in Contests

Here I began to address the problem of how to report and compare bust rates in contests in a defensible, objective manner.

At the end of that post, we decided that: rather than simply quoting some kind of "bust rate", a range should be quoted for each station, representing, say, the 99% confidence limit for the rate.

If we do that, and reorder the stations in order of decreasing upper limit (which seems like the most reasonable ordering: it means that we are 99.5% sure that the actual bust rate is less than this number), then we find: and then we follow this text with a table.

Let us apply this methodology to an example contest (CQ WW SSB for 2005, as that is the first CQ WW contest for which public logs are available), and look at the stations that appear to have the best rate of copying. What we tabulate is the 99.5% confidence limit for the probability of a bust, $p_{bust}$, as in this table:

2005 CQ WW SSB -- 99.5% confidence upper bound to $p_{bust}$
Position	Call	$p_{995}$
1	OH5BM	0.0005
2	9A2EU	0.0008
3	N5AU	0.0012
4	UA1OMX	0.0015
5	V31MQ	0.0018
6	UR6IJ	0.0019
7	KF2O	0.0020
8	DL3BRA	0.0021
9	EA1JO	0.0022
10	W3YY	0.0022

Let's expand the table so as to include the verified number of QSOs, $Q_v$, and the number of busts, $B$, for each station:

2005 CQ WW SSB -- 99.5% confidence upper bound to $p_{bust}$
Position	Call	$p_{995}$	$Q_v$	$B$
1	OH5BM	0.0005	1904	0
2	9A2EU	0.0008	1175	0
3	N5AU	0.0012	802	0
4	UA1OMX	0.0015	649	0
5	V31MQ	0.0018	563	0
6	UR6IJ	0.0019	533	0
7	KF2O	0.0020	488	0
8	DL3BRA	0.0021	470	0
9	EA1JO	0.0022	457	0
10	W3YY	0.0022	447	0

It should be comforting to observe that by ordering the stations in increasing order of $p_{995}$, we have also sorted it in decreasing order of $Q_v$.

However, note that none of the listed stations have any busts. What happens when we look at stations that have different numbers of busts? Let us look at an example further down the table:

2005 CQ WW SSB -- 99.5% confidence upper bound to $p_{bust}$
Position	Call	$p_{995}$	$Q_v$	$B$
4025	EC7ALM	0.6096	19	6
4026	GM8KSJ	0.6307	7	1

Let's now look at the actual probability distribution of $p_{bust}$ for these two stations (the vertical lines are at the two values of $p_{995}$):

It is obvious from this plot that the order of these two stations should be reversed in any meaningful table that purports to order stations in increasing order of estimates of $p_{bust}$. What is not so obvious is that the order of the two stations depends on the particular confidence limit chosen. For the 99.5% limit, the order is as stated, but (for example) for the 98% limit, the order is reversed. This arbitrariness is obviously as unacceptable as the basic notion that somehow GM8KSJ has a higher value of $p_{bust}$ than does EC7ALM.

So we need a different approach: although quoting a particular confidence limit is useful for defining $p_{bust}$ for a single station, it is inappropriate for ordering multiple stations, particularly in the case when different numbers of busts are involved (because the shapes of the probability curves differ markedly as $B$ varies).

A better way to order two stations is to determine the weighted mean value of $p_{bust}$ as distributed according to probability function of each station. This automatically results in an ordering such that if one selects a large number of values of $p_{bust}$ distributed according to the two relevant probability functions, the mean value for the first station will be less than the mean value for the second station.

[It is probably worth noting that, because of the lack of symmetry in the probability curves, this is not the same as the first station being necessarily more likely to have a lower value of  $p_{bust}$ than is the second. If this isn't obvious, just plot the difference between the probability curves for two stations with different values of $B$, such as EC7ALM and GM8KSJ.]

Note also that this procedure will leave the order of stations with equal numbers of busts unchanged, which is comforting.

Applying this procedure, our top-ten table now looks like this:

2005 CQ WW SSB -- weighted mean values of $p_{bust}$
Position	Call	weighted mean	$Q_v$	$B$
1	OH5BM	0.0005	1904	0
2	9A2EU	0.0008	1175	0
3	N5AU	0.0012	802	0
4	UA1OMX	0.0015	649	0
5	V31MQ	0.0017	563	0
6	UR6IJ	0.0018	533	0
7	ES5RY	0.0019	1068	1
8	KF2O	0.0020	488	0
9	DL3BRA	0.0021	470	0
10	EA1JO	0.0022	457	0

This looks reasonable (to me, anyway).