2020-06-16

Reverse NILs in CQ WW: 2005

Using the augmented public logs of the CQ WW contests, one of the many analyses that can easily be performed is to see which entrants cause the largest number of NIL (Not-In-Log) strikes against stations who have logged them.

I should make it clear from the outset that NILs can and do occur naturally in contests. A typical instance would be when station A is CQing and is called by stations B and C simultaneously. Station A goes back to station B, but station C, for whatever reason, thinks that station A has come back to him. The common end result is that station A logs station B, but both stations B and C log station A. This will cause a NIL strike against station C -- this is a legitimate NIL strike, because station C never heard his call transmitted by station A, there was no QSO between those two stations, and station C should never have logged the non-existent QSO.

NILs, though, have occasionally been used as a weapon when a station deliberately does not log a perfectly valid QSO: this has no cost to the station that omits the QSO, but can cause a penalty (and perhaps loss of a multiplier or QSO points) against the station that he did not log.

Because of the way that scoring works in CQ WW, this behaviour is particularly pernicious in the case of intra-W QSOs (i.e., QSOs between two US stations). Intra-W QSOs are worth no points, but they do provide multipliers. As QSOs aren't worth any points, it is frowned on for an American station to make more than a handful of QSOs with other Ws, to obtain the necessary multipliers, as the calling station is generally wasting the time of the called station while he obtains the needed mult. The result of this is that if a W station does not log all W QSOs, he is likely depriving the other stations of necessary mults, as they will often not take out insurance QSOs to be sure of the mult, not wanting to waste the time of another station. Failure to log the QSO gives the station that does so a competitive advantage, as it therefore deprives other stations of a legitimately-worked mult, and causes penalties to be applied to the innocent parties -- at no cost to himself.

So I thought that it might be instructive to look at the public CQ WW logs to see which stations cause the largest numbers of NILs to be applied against stations who have claimed a QSO with them. (I call a NIL that is applied to the other station's log a reverse NIL, or rNIL.)

I made a couple of small adjustments to the basic idea outlined above:
  1. I required a minimum of 50 appearances in other stations' logs for a station to be included in the analysis (25 appearances in the case of purely intra-W QSOs);
  2. I removed all stations for which the analysis showed a 100% rNIL rate: this rate almost certainly occurred because of some basic error in the station's log, such as submitting a log that used a different callsign from the one actually used in the contest.
The code to perform the analysis is available here (feel free to let me know of bugs :-)). Below are the results of running this code against the logs for 2005.

2005 SSB:

Callsign Total rQSOs Total rNILs
HK3JJH 1820 1790
M7Z 1242 1166
EA1BVP 1957 972
IR8P 934 927
UU7J 4518 837
SP8IMG 1103 834
OR5N 678 668
JA6GCE 770 660
IZ8DPL 678 643
RU6MM 613 609


Callsign Total rQSOs Total rNILs % rNILs
IT9RBW 383 382 99.7
RU6MM 613 609 99.3
IR8P 934 927 99.3
DL5MK 132 131 99.2
W7QDM 114 113 99.1
TA0U 393 389 99.0
YB0AI 563 555 98.6
OR5N 678 668 98.5
FR1HZ 581 572 98.5
9A4RV 190 187 98.4


Callsign Total rQSOs with Ws rNILs against Ws
KC1XX 139 22
N2IC 91 17
N3RS 67 15
NQ4I 247 14
W4MYA 84 12
W0AIH 136 11
W3LPL 183 11
WB9Z 38 11
K3LR 226 10
W7WA 30 10


Callsign Total rQSOs with Ws Total rNILs against Ws % rNILs against Ws
W7WA 30 10 33.3
K5NZ 25 8 32.0
WB9Z 38 11 28.9
NE3F 25 6 24.0
N3RS 67 15 22.4
N2IC 91 17 18.7
KC1XX 139 22 15.8
W9RE 32 5 15.6
W6YI 26 4 15.4
W4MYA 84 12 14.3

2005 CW:

Callsign Total rQSOs Total rNILs
J88DR 1734 1706
SM7YEA 1496 1488
OJ0B 2126 824
ZY7C 2316 715
UN6T 674 672
RK9CR 666 631
W3BGN 1648 611
Z37M 3983 605
KT1V 2557 550
S54A 973 545


Callsign Total rQSOs Total rNILs % rNILs
UN6T 674 672 99.7
W4ZW 329 328 99.7
SM7YEA 1496 1488 99.5
PV8AZ 82 81 98.8
S52P 389 384 98.7
PY8MGB 457 451 98.7
SP9UOP 204 201 98.5
J88DR 1734 1706 98.4
W1WFZ 176 172 97.7
K1OZ 66 64 97.0


Callsign Total rQSOs with Ws rNILs against Ws
W5UN 34 27
K9NS 181 25
NQ4I 229 12
K3LR 170 11
K1TTT 131 7
K1RX 125 7
KT1V 37 7
K5GO 171 6
K1AR 55 6
W5KFT 66 6


Callsign Total rQSOs with Ws Total rNILs against Ws % rNILs against Ws
W5UN 34 27 79.4
K2QMF 25 5 20.0
KT1V 37 7 18.9
W3BGN 25 4 16.0
K9NS 181 25 13.8
K0EU 26 3 11.5
W2RE 27 3 11.1
K1AR 55 6 10.9
K2LE 47 5 10.6
K4XS 40 4 10.0


A word about these old logs: they contain many formatting and other errors compared to modern logs (for example, the "zones" field might contain serial numbers, even though the correct zone was actually transmitted), so many of the results should not be taken at face value. For example, the second and sixth tables above indicate an unrealistic percentage of rNILs from all the stations in the table. On the other hand, the large difference between W5UN and all the other lines in the final table suggests that one might want to investigate the cause, if one were interested. Generally speaking, and especially in these old, low-quality logs, it's a good idea to focus on what appear to be anomalous stations, rather than attaching too much credence to the raw numbers in the tables.

(If one suspected a particular station of purposefully failing to log QSOs, several statistical approaches suggest themselves. One presumes that nowadays the contest committee would quickly resolve the issue by requesting the required audio recording of the contest from the suspected station.)

2020-05-28

Creating a Local Database From FCC Public Files

The FCC don't make it particularly easy to query their database(s) to find useful information related to a licensee. Yes, there is the web-based "License Search" page, but that's not useful for broader questions (to create a random example: how many Advanced class licensees are there in Texas?).

Fortunately, it's fairly easy to generate a local version of the FCC database, thereby allowing one to use classical tools (such as, for example,  awk) to obtain rapid answers to queries that one deems interesting.

The FCC generates a series of data files weekly (on Sunday), and makes those available over the Internet. (They also generate daily files of changes, as described here.) This allows interested parties to download the data, merge and simplify them, and generate a local file that contains the interesting/useful information.

In total there are eight data files, typically identified by a two-character tag: AM, CO, EN, HD, HS, LA, SC or SF. Each file contains a number of records, and the contents of each record is documented (lightly) in this document. Unfortunately, although that document gives the name of each field, it's not always obvious from the name what the field actually contains. Many (but far from all) fields are documented here; other fields are a matter for guesswork, or even a shrug of the shoulders.

Anyway, for the most part it's easy to combine all the useful fields from the eight data files into a single file in which each record is identified by a (unique) callsign. In practice, it turns out that, at least to my eyes, only the first four of the named data files contain information of general interest.

Consequently, I have created a file, updated weekly (on a Monday) that combines the more interesting data from the eight published data files. The records in this file are delimited by linefeeds, and fields within each record, as in the original data files, are separated by the standard UNIX pipe character, "|". Each output record contains 48 fields.

The contents of the eight original files are described in this document as follows:

Amateur
Position Data Element Definition
[AM]
1   Record Type [AM]            char(2)
2   Unique System Identifier    numeric(9,0)
3   ULS File Number             char(14)
4   EBF Number                  varchar(30)
5   Call Sign                   char(10)
6   Operator Class              char(1)
7   Group Code                  char(1)
8   Region Code                 tinyint
9   Trustee Call Sign           char(10)
10  Trustee Indicator           char(1)
11  Physician Certification     char(1)
12  VE Signature                char(1)
13  Systematic Call Sign Change char(1)
14  Vanity Call Sign Change     char(1)
15  Vanity Relationship         char(12)
16  Previous Call Sign          char(10)
17  Previous Operator Class     char(1)
18  Trustee Name                varchar(50)

Comments
Position Data Element Definition
[CO]
1   Record Type [CO]            char(2)
2   Unique System Identifier    numeric(9,0)
3   ULS File Number             char(14)
4   Call Sign                   char(10)
5   Comment Date                mm/dd/yyyy
6   Description                 varchar(255)
7   Status Code                 char(1)
8   Status Date                 mm/dd/yyyy

Entity
Position Data Element Definition
[EN]
1   Record Type [EN]                char(2)
2   Unique System Identifier        numeric(9,0)
3   ULS File Number                 char(14)
4   EBF Number                      varchar(30)
5   Call Sign                       char(10)
6   Entity Type                     char(2)
7   Licensee ID                     char(9)
8   Entity Name                     varchar(200)
9   First Name                      varchar(20)
10  MI                              char(1)
11  Last Name                       varchar(20)
12  Suffix                          char(3)
13  Phone                           char(10)
14  Fax                             char(10)
15  Email                           varchar(50)
16  Street Address                  varchar(60)
17  City                            varchar(20)
18  State                           char(2)
19  Zip Code                        char(9)
20  PO Box                          varchar(20)
21  Attention Line                  varchar(35)
22  SGIN                            char(3)
23  FCC Registration Number (FRN)   char(10)
24  Applicant Type Code             char(1)
25  Applicant Type Code Other       char(40)
26  Status Code                     char(1)
27  Status Date                     mm/dd/yyyy

1   Record Type [HD]                            char(2)
2   Unique System Identifier                    numeric(9,0)
3   ULS File Number                             char(14)
4   EBF Number                                  varchar(30)
5   Call Sign                                   char(10)
6   License Status                              char(1)
7   Radio Service Code                          char(2)
8   Grant Date                                  mm/dd/yyyy
9   Expired Date                                mm/dd/yyyy
10  Cancellation Date                           mm/dd/yyyy
11  Eligibility Rule Num                        char(10)
12  Reserved                                    char(1)
13  Alien                                       char(1)
14  Alien Government                            char(1)
15  Alien Corporation                           char(1)
16  Alien Officer                               char(1)
17  Alien Control                               char(1)
18  Revoked                                     char(1)
19  Convicted                                   char(1)
20  Adjudged                                    char(1)
21  Reserved                                    char(1)
22  Common Carrier                              char(1)
23  Non Common Carrier                          char(1)
24  Private Comm                                char(1)
25  Fixed                                       char(1)
26  Mobile                                      char(1)
27  Radiolocation                               char(1)
28  Satellite                                   char(1)
29  Developmental or STA or Demonstration       char(1)
30  InterconnectedService                       char(1)
31  Certifier First Name                        varchar(20)
32  Certifier MI                                char(1)
33  Certifier Last Name                         varchar(20)
34  Certifier Suffix                            char(3)
35  Certifier Title                             char(40)
36  Female                                      char(1)
37  Black or African-American                   char(1)
38  Native American                             char(1)
39  Hawaiian                                    char(1)
40  Asian                                       char(1)
41  White                                       char(1)
42  Hispanic                                    char(1)
43  Effective Date                              mm/dd/yyyy
44  Last Action Date                            mm/dd/yyyy
45  Auction ID                                  integer
46  Broadcast Services - Regulatory Status      char(1)
47  Band Manager - Regulatory Status            char(1)
48  Broadcast Services - Type of Radio Service  char(1)
49  Alien Ruling                                char(1)
50  Licensee Name Change                        char(1)
51  Whitespace Indicator                        char(1)

History
Position Data Element Definition
[HS]
1   Record Type [HS]            char(2)
2   Unique System Identifier    numeric(9,0)
3   ULS File Number             char(14)
4   Call Sign                   char(10)
5   Log Date                    mm/dd/yyyy
6   Code                        char(6)

License Attachment
Position Data Element Definition
[LA]
1   Record Type [LA]            char(2)
2   Unique System Identifier    numeric(9,0)
3   Call Sign                   char(10)
4   Attachment Code             char(1)
5   Attachment Description      varchar(60)
6   Attachment Date             mm/dd/yyyy
7   Attachment File Name        varchar(60)
8   Action Performed            char(1)

Special Condition
Position Data Element Definition
[SC]
1   Record Type [SC]            char(2)
2   Unique System Identifier    numeric(9,0)
3   ULS File Number             char(14)
4   EBF Number                  varchar(30)
5   Call Sign                   char(10)
6   Special Condition Type      char(1)
7   Special Condition Code      int
8   Status Code                 char(1)
9   Status Date                 mm/dd/yyyy

License Free Form Special Condition
Position Data Element Definition
[SF]
1   Record Type [SF]                    char(2)
2   Unique System Identifier            numeric(9,0)
3   ULS File Number                     char(14)
4   EBF Number                          varchar(30)
5   Call Sign                           char(10)
6   License Free Form Type              char(1)
7   Unique License Free Form Identifier numeric(9,0)
8   Sequence Number                     integer
9   License Free Form Condition         varchar(255)
10  Status Code                         char(1)
11  Status Date                         mm/dd/yyyy
The following extract from the code that creates the output database maps these on a one-to-one basis to internal identifiers:

[AM]
RECORD_TYPE,
ID,
ULS_NUMBER,
EBF_NUMBER,
CALLSIGN,
OPERATOR_CLASS,
GROUP_CODE,
REGION_CODE,
TRUSTEE_CALLSIGN,
TRUSTEE_INDICATOR,
PHYSICIAN_CERTIFICATION,
VE_SIGNATURE,
SYSTEMATIC_CALLSIGN_CHANGE,
VANITY_CALLSIGN_CHANGE,
VANITY_RELATIONSHIP,
PREVIOUS_CALLSIGN,
PREVIOUS_OPERATOR_CLASS,
TRUSTEE_NAME
 [CO]
RECORD_TYPE,
ID,
ULS_NUMBER,
CALLSIGN,
COMMENT_DATE,
DESCRIPTION,
STATUS_CODE,
STATUS_DATE
 [EN]
RECORD_TYPE,
ID,
ULS_NUMBER,
EBF_NUMBER,
CALLSIGN,
ENTITY_TYPE,
LICENSE_ID,
ENTITY_NAME,
FIRST_NAME,
MIDDLE_INITIAL,
LAST_NAME,
SUFFIX,
PHONE,
FAX,
EMAIL,
STREET_ADDRESS,
CITY,
STATE,
ZIP_CODE,
PO_BOX,
ATTENTION_LINE,
SGIN,
FRN,
APPLICANT_TYPE_CODE,
APPLICANT_TYPE_CODE_OTHER,
STATUS_CODE,
STATUS_DATE
 [HD]
RECORD_TYPE,
ID,
ULS_NUMBER,
EBF_NUMBER,
CALLSIGN,
LICENSE_STATUS,
RADIO_SERVICE_CODE,
GRANT_DATE,
EXPIRED_DATE,
CANCELLATION_DATE,
ELIGIBILITY_RULE_NUM,
RESERVED_1,
ALIEN,
ALIEN_GOVERNMENT,
ALIEN_CORPORATION,
ALIEN_OFFICER,
ALIEN_CONTROL,
REVOKED,
CONVICTED,
ADJUDGED,
RESERVED_2,
COMMON_CARRIER,
NON_COMMON_CARRIER,
PRIVATE_COMM,
FIXED,
MOBILE,
RADIOLOCATION,
SATELLITE,
DEVELOPMENTAL_STA_DEMONSTRATION,
INTERCONNECTED_SERVICE,
CERTIFIER_FIRST_NAME,
CERTIFIER_MIDDLE_INITIAL,
CERTIFIER_LAST_NAME,
CERTIFIER_SUFFIX,
CERTIFIER_TITLE,
FEMALE,
BLACK_AFRICAN_AMERICAN,
NATIVE_AMERICAN,
HAWAIIAN,
ASIAN,
WHITE,
HISPANIC,
EFFECTIVE_DATE,
LAST_ACTION_DATE,
AUCTION_ID,
BROADCAST_SERVICES_REGULATORY_STATUS,
BAND_MANAGER_REGULATORY_STATUS,
BROADCAST_SERVICES_SERVICE_TYPE,
ALIEN_RULING,
LICENSEE_NAME_CHANGE,
WHITESPACE_INDICATOR
 [HS]
RECORD_TYPE,
ID,
ULS_NUMBER,
CALLSIGN,
LOG_DATE,
CODE
 [LA]
RECORD_TYPE,
ID,
CALLSIGN,
ATTACHMENT_CODE,
ATTACHMENT_DESCRIPTION,
ATTACHMENT_DATE,
ATTACHMENT_FILENAME,
ACTION_PERFORMED
 [SC]
RECORD_TYPE,
ID,
ULS_NUMBER,
EBF_NUMBER,
CALLSIGN,
SPECIAL_CONDITION_TYPE,
SPECIAL_CONDITION_CODE,
STATUS_CODE,
STATUS_DATE
 [SF]
RECORD_TYPE,
ID,
ULS_NUMBER,
EBF_NUMBER,
CALLSIGN,
LICENSE_FREEFORM_TYPE,
UNIQUE_LICENSE_FREEFORM_ID,
SEQUENCE_NUMBER,
LICENSE_FREEFORM_CONDITION,
STATUS_CODE,
STATUS_DATE

The 48 output fields selected from the above lists are (arranged in groups of ten for easy counting):

ID,
CALLSIGN,
OPERATOR_CLASS,
GROUP_CODE,
REGION_CODE,
TRUSTEE_CALLSIGN,
TRUSTEE_INDICATOR,
SYSTEMATIC_CALLSIGN_CHANGE,
VANITY_CALLSIGN_CHANGE,
VANITY_RELATIONSHIP,

PREVIOUS_CALLSIGN,
PREVIOUS_OPERATOR_CLASS,
TRUSTEE_NAME,
COMMENT_DATE,
DESCRIPTION,
CO_STATUS_CODE, (i.e., STATUS_CODE from [CO])
CO_STATUS_DATE, (i.e., STATUS_DATE from [CO])
ENTITY_NAME,
FIRST_NAME,
MIDDLE_INITIAL,

LAST_NAME,
SUFFIX,
PHONE,
FAX,
EMAIL,
STREET_ADDRESS,
CITY,
STATE,
ZIP_CODE,
PO_BOX,

ATTENTION_LINE,
FRN,
APPLICANT_TYPE_CODE,
APPLICANT_TYPE_CODE_OTHER,
EN_STATUS_CODE, (i.e., STATUS_CODE from [EN])
EN_STATUS_DATE, (i.e., STATUS_DATE from [EN])
LICENSE_STATUS,
RADIO_SERVICE_CODE,
GRANT_DATE,
EXPIRED_DATE,

CANCELLATION_DATE,
ELIGIBILITY_RULE_NUM,
REVOKED,
CONVICTED,
ADJUDGED,
EFFECTIVE_DATE,
LAST_ACTION_DATE,
LICENSEE_NAME_CHANGE
The contents of these fields are based on the original equivalent entries in the original data files. The entries for the fields are subject to the following transformations before being written to the output file:
  • The entry is converted to upper case;
  • Any line feeds (yes, the FCC allows line feeds within a field) are converted to the four-character sequence: <LF>;
  • Leading and trailing spaces are removed;
  • If the field is a date, it is converted from FCC format (mm/dd/yyyy) to ISO 8601 extended format: YYYY-MM-DD.
The latest output file created in this manner (and its MD5 checksum) may be downloaded from this directory.

The full source code to generate the output file may be downloaded here.

To create the binary from the source code, go to the directory that contains the makefile and type:
make fcc-db
This should generate the executable program as: bin/fcc-db. The program may be executed from within the bin directory as:
fcc-db [directory]
where [directory] is the name of the directory that contains the input FCC AM.dat, CO.dat, EN.dat and HD.dat files. Those files should be processed and the output written to stdout.

For what it's worth, it takes somewhat less than 15 seconds for the program to execute to completion on my desktop computer if stdout is redirected to an output file.

2020-05-26

RBN Posts From Current Year Added

The RBN data directory on adrive now contains a file of the RBN posts from the current year. This file is updated once per week, on Monday. As before, files containing all the posts from prior years are available in the same directory.

2020-03-16

RBN-based Proxy Metric for Relative CW Activity

The Reverse Beacon Network (RBN) makes available an Internet-based historical record of all CW stations on the HF amateur bands detected, decoded and posted by the stations that participate in the RBN. This suggests the possibility of using those data in some form to provide a more or less reliable indicator of the number of CW-active stations since early 2009, when the RBN came into being.

Performing such an analysis, though, is not a trivial matter because of several characteristics of the RBN. In particular, one cannot simply count the number of callsigns detected over a period and assume that that number may reasonably be simply compared to the number of callsigns detected over any other period of similar duration. The following characteristics (at least) argue against such a naïve comparison:
  1. The RBN comprises, at any one moment, a number of receiving stations that post detected CW callsigns to the Internet. However, no record is kept of the number of stations actively recording and posting at any given moment; that number varies widely over time.
  2. The geographical distribution of posting stations varies from moment to moment.
  3. Even at the best of times, large swathes of the Earth's surface contain no posting stations.
  4. Different posting stations listen to different sets of HF bands, and a single posting station may monitor different bands at different times.
  5. The software that drives the RBN is tuned with the intent of eliminating callsigns of stations that are calling other stations. That is, the RBN is intended to post only the callsigns of stations seeking contacts (generally by calling CQ).
  6. The software may decode callsigns incorrectly, and thus post a callsign that is not in fact present on the air.
  7. Hardly the RBN's fault, but some stations send so poorly that several variants of their call may be decoded and posted.
The effect of these characteristics is to pollute any list of posted callsigns, and also to cause some callsigns that were actually used on the air not to appear in any such list. The question is, then: is it possible still to produce a useful metric from the RBN data?

First, let's look at issue number 5 above: namely that callsigns only of CQing stations are posted. In the past this might have been regarded as a serious shortcoming; it is less clear that the same is true today, especially if we are attempting to measure (in some sense) those stations whose operators actively prefer and enjoy CW. With the modern proliferation of huge CW pile-ups on major DXpeditions there are now many stations whose goal is simply to work the DXpedition (perhaps for a "slot"; perhaps for a new one) and operators of such stations may well have next to no knowledge of CW at all, using pre-recorded macros for transmission, and either code readers or simple aural recognition of the pattern of their callsign to decode the high-speed CW from the DX station. It would not seem right to include such stations in any listing of CW activity, since their use of CW is mere happenstance. So issue number 5 merely serves to cause us to define "CW activity" in a particular manner that might not have been appropriate in the past but nowadays, in the current milieu, seems quite defensible, and perhaps even preferable to merely listing all callsigns heard on the air. Thus, "CW activity" herein means "CW activity by stations calling CQ".

Having finessed issue #5, the others remain; to some extent they must all affect the calls posted by the RBN. Any one of them might be amenable to reasonable analysis or at least intelligent guesswork as to its effect; taken together, it seems prudent to simply say that in any group of calls posted over a particular time, some calls are likely to be in error, and some genuine calls that were active may well have been missed. That much is obvious, but it does not necessarily mean that a reasonable metric cannot be derived with pragmatic simplicity from the actual postings made by the RBN.

We can begin by restricting the domain of the problem, for now, to a single band. The obvious choice is 20m, as it has a reasonably consistent high level of activity throughout a solar cycle. So let's look at some numbers and graphs created from the raw RBN postings over the period from its inception in early 2009 to the end of 2019, using the calendar year as the basic element of time (thus, the values for 2009 will be affected by the shorter period for which samples are available, in addition to any effects from the  considerations listed above).

To get some feel for the data, we begin by defining a value $V(n)$ that is the number of callsigns that appear exactly $n$ times within a particular year. We can then plot the values of $V(1)$ to $V(100)$ for each of the years for which we have data:


Despite the lack of detail at this scale, a couple of interesting things are immediately apparent:
  1. The plots of $V(n)$ are remarkably consistent from year to year;
  2. The transition between large negative gradient and shallow negative gradient occurs over a relatively short range of $n$.
It is perhaps useful to put into words the most obvious feature of the plot: a large percentage of the calls posted by the RBN occur just a very few times. If one thinks about this, if a call appears only once (or just a small number of times) in the course of a year, it would seem overwhelmingly likely that it is a bust of some kind: the circumstances that would lead to a valid call being correctly decoded and posted only once (or a small number of times) in an entire year would seem to be overwhelmingly unlikely. Similarly, if a call appears a relatively large number of times, it is highly unlikely to be a bust. We shall return to this simple fact below.

By switching to a logarithmic scale, we can see more detail in the data:


Apart from 2009, when, as mentioned above, there are fewer data, the plots for each year essentially lie on top of each other; further, there are no obvious characteristics other than a more or less monotonic decrease in $V(n)$ as $n$ increases. This is good, as it suggests that there is some robustness to this kind of analysis, and therefore it may be possible to use it as a basis for calculation of an activity metric.

What happens if we look at a band with great variation over the course of a solar cycle; 10m, for instance?

No new features appear, further suggesting that this is a robust approach.

The graphs suggest that an activity metric MCW may be defined over an interval of time by:
$$
MCW = \sum_{n=1}\alpha(n) V(n)
$$
where the form of $\alpha(n)$ is to be determined.

For small values of $n$, all or almost all calls will be busts -- that is, $\alpha(\rm small) \approx 0$; for large values of $n$ no or almost no calls will be busts  -- that is, $\alpha(\rm large) \approx 1$.

We can reasonably define "large" to be the lowest value $n$ for which $V(n)$ is statistically indistinguishable from $V(n+1)$ and (to be safe) for which $V(n+1)$ is statistically indistinguishable from $V(n+2)$. And we can reasonably define "statistically indistinguishable" as meaning that $V(m+1)$ lies in the range:
$$
V(m) \pm 2\sqrt{(V(m))}
$$

We can then define "small" simply as unity, and $\alpha(n)$ as a linear function with the value zero for $n=1$ and unity for $n \ge L$, where $L$ is the smallest "large" number defined by the procedure described above. [NB there is nothing magic about these precise definitions; reasonable variations on this theme provide essentially identical relative results from year to year -- another indication of robustness. One simply needs to be careful to maintain the same definition of $\alpha(n)$ for all the periods under examination. [Actually, it works even with the same meta-definition, but let's not bother with that]] By this definition, "large" will vary from year to year and from band to band, meaning that $\alpha(n)$ will similarly vary. And that is one way to proceed, but it means calculating the meaning of "large" for each year and each band. A simpler way to analyze the data is to choose a universal value for "large" that is large enough to encompass all years and bands being analysed.

The actual values of "large" as defined above, for all the years and bands for which we have annual data, are given in the following table:

Band20092010201120122013201420152016201720182019
10m711121313141414999
12m48161012101288710
15m1011161717171213161013
17m7139112110129131811
20m2116161820211927242120
30m1217141518131318121514
40m1418172224212523202015
80m1622121815131414192222
160m912111011111218121414
HF1819272324233223202222

The last line, marked "HF" is derived from all the postings for the individual bands for a given year. [Note that this bears no simple relationship to the other lines; if this isn't obvious, just think of a call that is posted once on 20m and once on 160m in the course of a year: this will contribute to $V(1)$ on 20m and 160m, but to $V(2)$ on HF.]

The largest value in the above table is 32 (HF, 2015). Therefore one plausible $\alpha(n)$ function would go linearly from a value of zero at $n=1$ to a value of 1 at $n=35$, and have the value 1 at all values >35.

Applying this procedure, we finally arrive at the following figure:


The CW Activity Metric as defined above is arguably a poor choice for showing small changes in activity over time (although it is not obvious how to create a better metric for such purposes given the limitations of the RBN listed above); but if the metric shows little variation over a period, it is hard to see how the underlying CW activity can have varied much either, absent a rather precisely anti-correlated change in the factors that lead to busts in the RBN postings.

The effect of the solar cycle is readily apparent in the figure (see the lines for the higher frequency bands, where activity does indeed change as a function of the cycle, and the metric reflects this change); however, taking HF activity as a whole, it is remarkable how consistent activity has been for many years. There is certainly no evidence from this analysis of the RBN data that CW activity taken as a whole has shown any consistent wane over the past decade. 

2020-03-09

Signals from the VP8PJ and VP8ORK DXpeditions to South Orkney

A few interesting (I think, anyway) graphs of CW signals from the most recent two DXpeditions to South Orkney as reported by the Reverse Beacon Network:





Quite a difference.

Worth noting is that:
  1. VP8ORK was operational in a period with the solar flux index in the high 70s, whereas VP8PJ was operational close to the cycle minimum, with a solar flux index in the high 60s.

  2. During the VP8ORK DXpedition a total of 100 RBN stations were active; during the VP8PJ DXpedition, a total of 234 RBN stations were active.