2017-01-12

Summary File for RBN data 2009 to 2016

The complete set of RBN data for 2009 to the end of 2016, after uncompression, is some 50GB in size. As not all analyses need the complete dataset, I have constructed a summary file that contains an overview of the data and which is sufficient for many kinds of analysis that do not depend on the details of individual posts to the RBN. (The script used to generate this summary file may be found here.)

The summary file, after being uncompressed, comprises a single large table of values separated by white space. The name of each column (there are nine columns in all) is on the first row. The columns are:
  1. band: a string that identifies the band pertaining to this row. Typical values are "15m" or "160m"; if a row contains data that are not distinguished by band, then the characters "NA" are used.
  2. mode: a string that identifies the mode pertaining to this row. Typical values are "CW" or "RTTY"; if a row contains data that are not distinguished by mode, then the characters "NA" are used.
  3. type: a single character that identifies whether the data on this row are for a period of a year ("A"), a month ("M") or a day ("D").
  4. year: the numeric four-digit value of the year to which the current row pertains.
  5. month: the numeric value of the month (January = 1, etc.) of the data in this row. If the data are of type A or D, then this element has the value "NA".
  6. doy: the numeric value of the day number of the year (January 1st = 1, etc.). The maximum value in each year is 366 (even if the year is not a leap year). In the event that the year is not a leap year, the data in columns 7, 8 and 9 will be set to 0 when doy is 366. If the data are of type A or M, then this element has the value "NA".
  7. posts: the total number of posts recorded by the RBN for the band, mode and period identified by the first six columns. 
  8. calls: the total number of distinguishable calls recorded by the RBN for the band, mode and period identified by the first six columns. 
  9. posters: the total number of distinguishable posters recorded by the RBN for the band, mode and period identified by the first six columns.
 For example, the first two lines of the summary file are:
      band     mode     type     year    month      doy    posts    calls  posters
     NA       NA        A     2009       NA       NA  5007040   143724      151

This tells us that the first line of actual data comprises annual data for the year 2009, with no separation by band or mode. In 2009, we see that there were 50,007,040 posts of 143,724 callsigns by 151 posters.

The summary file allows rather rapid analysis of the RBN. For example, this plot in this post was originally generated rather tediously from the entire 50GB dataset. Using the summary file, the same plot --

 -- can be generated in less than ten seconds.

The summary file can be used to generate a simple plot of the growth of the RBN since its inception just as quickly:




2017-01-07

Most-Logged Stations in CQ WW SSB 2016

The public CQ WW SSB logs allow us easily to tabulate the stations that appear in the largest number of entrants' logs. For 2016, the ten stations with the largest number of appearances were:

Callsign Appearances % logs
CN3A 8696 57
CN2R 8333 55
9A1A 8111 53
LZ9W 8072 52
PJ2T 7729 45
EF8R 7677 53
YT8A 7629 50
CN2AA 7268 51
PJ4X 7087 44
DF0HQ 6752 47


The first column in the table is the callsign. The second column is the total number of times that the call appears in logs. That is, if a station worked CN3A on six bands, that will increment the value in the second column of the CN3A row by six. The third column is the percentage of logs that contain the callsign at least once.

For comparison, here is the equivalent table for 2015:

Callsign Appearances % logs
CN2AA 15100 78
CN2R 10017 63
9A1A 9872 61
CN3A 9693 61
DF0HQ 9237 59
LZ9W 9045 56
HK1NA 8936 54
PJ2T 8456 51
K3LR 8310 54
EF8R 8254 57

We can also perform the same analysis for, say, a ten-year span, to show which stations have most consistently appeared in other stations' logs. So, for CQ WW SSB for the period 2007 to 2016, we find:


Callsign Appearances % logs
LZ9W 80063 57
DF0HQ 76871 57
CN3A 74097 58
OT5A 71180 55
K3LR 66814 51
PJ2T 66702 47
P33W 64957 52
DR1A 53237 46
V26B 55209 43
LY7A 55161 44


 For comparison, the table for the period 2006 to 2015 is:

Callsign Appearances % logs
DF0HQ 76024 59
LZ9W 71994 54
CN3A 69456 58
OT5A 66397 53
K3LR 65974 52
DR1A 65622 49
PJ2T 63768 47
LY7A 58901 49
P33W 58520 49
V26B 56197 45

Tables relating to earlier years are here.





2017-01-06

Summary of RBN data, 2009 - 2016

A simple plot of the RBN data from inception to the end of 2016 shows a few noteworthy features:









  1. The hebdomadal periodicity caused by increased activity at weekends is obvious.
  2. The weekends of major SSB contests (CQ WW, CQ WPX, and, to a lesser extent, ARRL DX) show little or no increase over the activity during the surrounding week.
  3. The weekend with the largest activity is consistently the weekend of CQ WW CW. The next-most-active weekend is the weekend of CQ WPX. During 2016 for the first time the RDXC contest engendered more activity (just) than did ARRL DX CW.

2017-01-05

Some CQ WW SSB Analyses

A huge number of analyses can be performed with the various CQ WW SSB files. There follow a few that interested me. There is plenty of scope for further analyses (some of which are suggested below).

Geographical Participation

How has the geographical distribution of entries changed over time? 




Zones 11 and 28 seem to show a slow but sustained increase in the number of logs submitted. Nevertheless, compared to the behemoths like zones 14 and 15, the number of logs from these areas is miniscule. This can be seen more clearly if we plot the percentage of logs received from each zone as a function of time:



It is also worth noting that after a lustrum or so of somewhat increased participation, zone 3 has declined perceptibly in relative participation in recent years.


Popularity

By definition, popularity requires some measure of people (or, in our case, the simple proxy of callsigns). So we can look at the number of calls in the logs as a function of time:


Regardless of how many logs a call has to appear in before we regard it as a legitimate callsign, the popularity of CQ WW SSB dropped considerably in 2016, to a level rarely (if ever) seen in the public logs.

Conditions were quite bad for the 2016 running of the contest; nevertheless, while poor conditions might reasonably be expected to have a major bearing on metrics related to the number of QSOs, it is not obvious, a priori, that it should have a substantial effect on the raw popularity as estimated by this metric. It seems that there were simply fewer stations QRV for the contest in 2016 than in earlier years.

I note that a reasonable argument can be made that the number of uniques will be more or less proportional to the number of QSOs made (I have not tested that hypothesis; I leave it as an exercise for the interested reader to determine whether it is true), but there is no obvious reason why the same would be true for, for example, callsigns that appear in, say, ten or more logs.


Activity

Total activity in a contest depends both on the number of people who participate and on how many QSOs each of those people makes. We can use the public logs to count the total number of distinct QSOs in the logs (that is, each QSO is counted only once, even if both participants have submitted a log).



As was the case for popularity, there was a precipitous decline in activity in the 2016 CQ WW SSB contest, with both 10m and 15m showing major decreases. Activity on the other bands was down slightly from the each band's peak year, but was more or less in line with activity for the past few years. The overall activity, though, was down nearly a third from 2015, a greater decline even than the decline in popularity: thus, fewer stations were active, and, on the average, the stations that were active made fewer QSOs.

Activity would be reasonably expected to be down in 2016, partly because the pool of stations to work was smaller in 2016 (i.e., the contest was less popular in that year), and partly because propagation between the stations that were active was worse.

(Everything else being equal, one might naïvely think that the number of QSOs would scale roughly as the square of the number of active stations active; however, given the highly non-random nature of the geographical distribution of stations in CQ WW and the propagation paths between stations, there is no reason to expect, a priori and without further analysis, that such a relationship would necessarily pertain. And, in any case, everything is not equal from year to year: a reasonable initial hypothesis is that the amount of activity would scale roughly as the product of the square of the number of active stations and some linear measure of the ease with which a "typical" QSO can be completed. There are, I think, sufficient data in the logs to perform a more quantitative analysis along these lines, should anyone be so inclined.)

2017-01-04

New CQ WW SSB video maps

I have updated the set of CQ WW SSB video maps on my youtube channel (channel N7DR). These video maps cover all the years for which public CQ WW SSB logs are currently available (2005 to 2016).

To access individual videos directly:

2017-01-03

CQ WW SSB 2016 logs available

The logs for the 2016 running of the CQ WW SSB contest are now available from CQ. The logs for the CW contest are not yet available.

The logs are also available in compressed form that may be more easily and quickly downloaded here.

I have created a compressed file that contains all the cleaned QSO lines from the Cabrillo files from all the SSB logs for all the years for which data are available. Currently, the file covers all the SSB QSOs in the years from 2005 to 2016. The MD5 checksum of this file is: ac555db15417fa76466ed636936e0efe.

I have also created an augmented file, in compressed format, that adds useful data to each QSO. Each QSO line in the augmented file includes an additional four columns, with the following meanings:

  1. The letter "A" or "U" indicating "assisted" or "unassisted"
  2. A four-digit number representing the time if the contact in minutes measured from the start of the contest. (I realise that this can be calculated from the other information on the line, but it saves a lot of time to have the number readily available in the file without having to calculate it each time.)
  3. Band
  4. A set of ten flags, each encoded as T/F: 
    • a. QSO is confirmed by a log from the second party 
    • b. QSO is a reverse bust (i.e., the second party appears to have bust the call of the first party) 
    • c. QSO is an ordinary bust (i.e., the first party appears to have bust the call of the second party) 
    • d. the call of the second party is unique 
    • e. QSO appears to be a NIL 
    • f. QSO is with a station that did not send in a log, but who did make 20 or more QSOs in the contest 
    • g. QSO appears to be a country mult 
    • h. QSO appears to be a zone mult 
    • i. the QSO is a zone bust (i.e., the received zone appears to be a bust)
    • j. the QSO is a reverse zone bust (i.e. the second party appears to have bust the zone of the first party)
The MD5 checksum of this file is: 4acae023af4ea41903249f54446dd79a.
  
Note that the flags in the augmented data are calculated from the raw data independently of the CQ contest committee. This is because:
  1. CQ still does not make the actual scoring code available ;
  2. the checklogs are not public, and hence represent additional data that CQ can use in determining the values of the flags.


2017-01-02

2016 RBN data

All the postings to the Reverse Beacon Network in 2016, along with the postings from prior years, are now available in the directory https://www.adrive.com/public/cQwkEB/rbn.

Some simple annual statistics for the period 2009 to 2016 follow (the 2009 numbers cover only part of that year, as the RBN was instantiated partway through that year).

Total posts:
2009:   5,007,040
2010:  25,116,810
2011:  49,705,539
2012:  71,584,195
2013:  92,875,152
2014:  108,862,505
2015:  116,385,762
2016:  111,027,068
 Total posting stations:
2009: 151
2010: 265
2011: 320
2012: 420
2013: 473
2014: 515
2015: 511
2016: 590
 Total posted callsigns:
2009: 143,724
2010: 266,189
2011: 271,133
2012: 308,010
2013: 353,952
2014: 398,293
2015: 433,197
2016: 375,613
Obviously, statistics that are considerably more comprehensive may be derived rather easily from the files in the directory.

Note that if you intend to use the databaseߴs reported signal strengths in an analysis, you should be sure that you understand the ramifications of what the RBN means by SNR.