2022-03-13

CW Activity from RBN data: 2009 to 2021

This post extends a prior post from last year to cover the period until the end of 2021. Note that the plots below do not show the number of posts, nor the number of stations posted, on the RBN. Rather, the algorithm for generating the plots is described here. Twelve equally-separated points are plotted for each year.

As before, this allows one to see the peaks corresponding to the annual CQ WW and ARRL DX contests, as well as the annual cycles of activity on the different bands as propagation changes through the year, in addition to the overall sunspot cycle. Despite the various comments I read about CW activity declining, the data seem to show no evidence of any actual decrease.

We can filter the data by continent:







2022-03-10

Evaluating Station Contributions to the Reverse Beacon Network: 2021

Applying the algorithm described here to the Reverse Beacon Network data for 2021 (1.1GB; MD5: b8451f143d8bc338e5afa98266a0dbb4) we obtain the following tables for the stations that, on the basis of that algorithm, made the highest-valued contributions to the RBN in 2021:

Band Position Call Value
ALL 1 OE9GHV 385,797
ALL 2 DO4DXA 313,738
ALL 3 OH6BG 313,728
ALL 4 G4ZFE 294,276
ALL 5 KM3T 288,770
ALL 6 WZ7I 282,629
ALL 7 HA1VHF 278,130
ALL 8 HA6PX 261,748
ALL 9 DL3DTH 260,242
ALL 10 OK2EW 255,624


Band Position Call Value
10m 1 DO4DXA 55,424
10m 2 WZ7I 36,250
10m 3 OH6BG 31,929
10m 4 DL3DTH 26,409
10m 5 KM3T 18,005
10m 6 KU7T 16,958
10m 7 K5TR 16,626
10m 8 KD7YZ 16,507
10m 9 G4ZFE 15,681
10m 10 N6TV 13,352


Band Position Call Value
12m 1 OH6BG 6,845
12m 2 CX6VM 5,400
12m 3 DL3DTH 4,954
12m 4 LZ4UX 3,969
12m 5 W6YX 3,858
12m 6 3V8SS 3,665
12m 7 EA5WU 3,440
12m 8 K1TTT 3,045
12m 9 WZ7I 2,861
12m 10 N6TV 2,816


Band Position Call Value
15m 1 OH6BG 25,876
15m 2 KM3T 22,789
15m 3 CX6VM 22,468
15m 4 W6YX 21,059
15m 5 K1TTT 19,149
15m 6 WZ7I 18,563
15m 7 PJ2A 18,139
15m 8 JQ1BVI 16,914
15m 9 KM3T-2 16,786
15m 10 KD7YZ 16,638


Band Position Call Value
17m 1 OH6BG 27,706
17m 2 KM3T 20,876
17m 3 DL3DTH 17,534
17m 4 OE9GHV 15,606
17m 5 WZ7I 15,589
17m 6 KO7SS 15,171
17m 7 W3LPL 14,225
17m 8 WA7LNW 13,378
17m 9 CX6VM 12,607
17m 10 TF4X 11,839


Band Position Call Value
20m 1 KM3T 108,490
20m 2 OE9GHV 106,480
20m 3 K1TTT 91,437
20m 4 G4ZFE 88,793
20m 5 W1NT-6 87,517
20m 6 VE2WU 87,270
20m 7 OH6BG 87,067
20m 8 WZ7I 86,057
20m 9 KM3T-2 85,875
20m 10 TF4X 73,110


Band Position Call Value
30m 1 SE5E 46,565
30m 2 DL3DTH 31,050
30m 3 OE9GHV 29,435
30m 4 G4ZFE 28,220
30m 5 OH6BG 25,873
30m 6 TF4X 24,838
30m 7 DF7GB 22,226
30m 8 SJ2W 20,509
30m 9 OK2EW 20,263
30m 10 EA5WU 19,325


Band Position Call Value
40m 1 OE9GHV 121,488
40m 2 HA6PX 89,668
40m 3 JH7CSU1 80,898
40m 4 G4ZFE 75,595
40m 5 WZ7I 72,409
40m 6 W1NT-6 71,201
40m 7 SE5E 69,428
40m 8 DO4DXA 68,342
40m 9 HA1VHF 68,129
40m 10 KD7YZ 67,749


Band Position Call Value
80m 1 OE9GHV 71,998
80m 2 DO4DXA 61,865
80m 3 HA6PX 57,536
80m 4 G4ZFE 50,389
80m 5 DF7GB 46,100
80m 6 SM6FMB 45,871
80m 7 OL7M 41,211
80m 8 OK2EW 40,783
80m 9 SJ2W 38,084
80m 10 SM7IUN 37,888


Band Position Call Value
160m 1 OK2EW 90,347
160m 2 HA1VHF 88,226
160m 3 HA6PX 29,671
160m 4 DK8NE 27,666
160m 5 JQ1BVI 26,145
160m 6 DL3DTH 26,136
160m 7 DO4DXA 24,001
160m 8 N2QT 22,878
160m 9 OE9GHV 22,772
160m 10 K9IMM 22,017


2022-03-09

Summary File for RBN data, 2009 to 2021

This is an update to a similar post last year.

The complete set of RBN data for 2009 to the end of 2021, after uncompression, easily exceeds 100GB in size. As not all analyses need the complete dataset, I have constructed a summary file (rbn-summary-data.xz in the above-linked directory to the complete set of RBN data) that contains an overview of the data and which is sufficient for many kinds of analysis that do not depend on the details of individual posts to the RBN. (The basic script used to generate this summary file may be found here; the actual summary file is created by running this basic script for each individual year from 2009 to 2021 and concatenating the results after removing the header line from all except the first year.)

The summary file, after being uncompressed, comprises a single large table of values separated by white space. The name of each column (there are twelve columns in all) is on the first row. The columns are:

  1. band: a string that identifies the band pertaining to this row. Typical values are "15m" or "160m"; if a row contains data that are not distinguished by band, then the characters "NA" are used.
  2. mode: a string that identifies the mode pertaining to this row. Typical values are "CW" or "RTTY"; if a row contains data that are not distinguished by mode, then the characters "NA" are used.
  3. type: a single character that identifies whether the data on this row are for a period of a year ("A"), a month ("M") or a day ("D").
  4. year: the numeric four-digit value of the year to which the current row pertains.
  5. month: the numeric value of the month (January = 1, etc.) of the data in this row. If the data are of type A or D, then this element has the value "NA".
  6. doy: the numeric value of the day number of the year (January 1st = 1, etc.). The maximum value in each year is 366 (even if the year is not a leap year). In the event that the year is not a leap year, the data in columns 7, 8 and 9 will be set to 0 when doy is 366. If the data are of type A or M, then this element has the value "NA".
  7. posts: the total number of posts recorded by the RBN for the band, mode and period identified by the first six columns. 
  8. calls: the total number of distinguishable calls recorded by the RBN for the band, mode and period identified by the first six columns. 
  9. posters: the total number of distinguishable posters recorded by the RBN for the band, mode and period identified by the first six columns. 
  10. scatter: the value of a scatter metric that characterises the geography of the RBN for the band, mode and period identified by the first six columns. The scatter metric is the sum of all possible distance pairs of good posters (measure in km), divided by the number of distance pairs.
  11. good posters: the total number of distinguishable posters recorded by the RBN for the band, mode and period identified by the first six columns, and for which location data are available from the RBN.
  12.  grid metric: the total number of G(15, 100) grid cells that contain good posters.
For example, the first two lines of the summary file are (presented here as a table, in order to make it easier to view on mobile devices):

band NA
mode NA
type A
year 2009
month NA
doy NA
posts 5007040
calls 143724
posters 151
scatter 5541
good_posters 150
grid_metric 22


This tells us that the first line of actual data in the file comprises annual data for the year 2009, with no separation by band or mode. In 2009, we see that there were 50,007,040 posts of 143,724 callsigns by 151 posters; the scatter metric, which is a measure of the geographic dispersion of the posters on the RBN., was 5,541; 150 different posters contributed the data, spread across 22 distinct G(15, 100) grid cells.

The summary file allows rather rapid analysis of many RBN overview statistics. For example, a plot of the daily number of posts covering the period from the inception of the RBN to the end of 2021 --

-- can be generated on a desktop PC in just a few seconds. From this plot, for example, we can immediately see that the largest number of daily posts occurred during the 2021 running of the CQ WW CW contest in late November (the second-highest cluster of peaks is for the CQ WPX contest, and the third is for the ARRL DX CW contest); also, the burst of activity that coincides with weekends is unmistakable.

For what it's worth, this is the code I used to generate the above plot (I apologise for the awful layout caused by the wrapping of long lines as they are forced into the narrow format used by blogger.com):

#!/usr/bin/Rscript

# generate a plot of the diurnal number of posts by the RBN, stacked by year

MIN_YEAR <- 2009
MAX_YEAR <- 2021

filename <- "/zd1/rbn/rbn-summary-data"  # the local location of the RBN summary data file

# first two lines of the file:
#band          mode          type          year         month           doy         posts         calls       posters       scatter  good_posters   grid_metric
#NA            NA             A          2009            NA            NA       5007040        143724           151          5541           150            22

# rounding function
round_n <- function(x, n) { return ( ( as.integer( (x - 1) / n) +1 ) * n ) }    # function to return next higher integral multiple of n, unless value is already such a multiple

data <- read.table(filename, header=TRUE)

# select diurnal data
diurnal_data <- subset(data, type=='D')

# drop the per-band and per-mode data
diurnal_all_bands_and_modes_data <- subset(diurnal_data, is.na(band) & is.na(mode))

# drop a bunch of columns that we don't want from the summary file
diurnal_all_bands_and_modes_data\$band <- NULL
diurnal_all_bands_and_modes_data\$mode <- NULL
diurnal_all_bands_and_modes_data\$type <- NULL
diurnal_all_bands_and_modes_data\$month <- NULL
diurnal_all_bands_and_modes_data\$calls <- NULL
diurnal_all_bands_and_modes_data\$posters <- NULL
diurnal_all_bands_and_modes_data\$scatter <- NULL
diurnal_all_bands_and_modes_data\$good_posters <- NULL
diurnal_all_bands_and_modes_data\$grid_metric <- NULL

# get ready to start to plot
graphics.off()

png(filename=paste(sep="", "/tmp/rbn-posts-from-summary.png"),  width=800, height=600)

x_lab <- 'DOY'

#            2009   2010                2011      2012     2013    2014      2015               2016      2017             2018     2019   2020
clrs <- c("black", "red", rgb(0.1, 0.1, 0.5), "yellow", "green", "blue", "violet", rgb(0.6, 0.2, 0.2), "white", "cornflowerblue", "gold1", "darkorange", "darkgreen")

# create a frame to map between year and days in the year
days_in_year <- data.frame(seq(MIN_YEAR, MAX_YEAR), 365)
names(days_in_year) <- c("year", "days")

days_in_year\$days[days_in_year\$year %% 4 == 0] <- 366

# set boundaries
plot(0, 0, xlim = c(0.5, 366.5), ylim = c(0, round_n(max(diurnal_all_bands_and_modes_data\$posts), 1000000)), xaxt = "n", yaxt = "n", xlab = x_lab, ylab = "", type = 'n', yaxs="i")         # define the plotting region, but don't actually plot anything
rect(par("usr")[1], par("usr")[3], par("usr")[2], par("usr")[4], col = 'grey')

# now generate the plot, superimposing each year
for (this_year in seq(MIN_YEAR, MAX_YEAR))
{ this_years_data <- subset(diurnal_all_bands_and_modes_data, year==this_year)
  max_element <- days_in_year\$days[this_year - MIN_YEAR + 1]
 
# set up so that this_years_data\$id_365[365] = this_years_data\$id_366[366] = 366, so either column can be used as a vector of abscissæ,
# depending on whether there are 365 or 366 ordinate values
  this_years_data\$id_366<-seq.int(nrow(this_years_data))
  this_years_data\$id_365<-((this_years_data\$id_366 - 1) * 365.0 / 364.0) + 1

# remove a couple of columns that we no longer need
  this_years_data\$doy <- NULL
  this_years_data\$year <- NULL

# move the two new columns of days to the left of the frame: 365, then 366
  this_years_data <- (this_years_data[ c(ncol(this_years_data), ncol(this_years_data) - 1, 1:(ncol(this_years_data)-2))])


  lines(this_years_data[,(max_element-364)][1:max_element], this_years_data\$posts[1:max_element], type = 'l', col = clrs[this_year - MIN_YEAR + 1], lwd = 1) 
}

title_str <- paste(sep="", 'RBN POSTS PER DAY')
title(title_str)

title(ylab = '# OF POSTS (m)', line = 2.1, cex.lab = 1.0)

x_ticks_at <- c(1, 31, 61, 91, 121, 151, 181, 211, 241, 271, 301, 331, 361)
x_labels_at <- x_ticks_at
x_tick_labels <- x_ticks_at

axis(side = 1, at = x_ticks_at, labels = FALSE )    # ticks on x axis
axis(side = 1, at = x_labels_at, labels = x_tick_labels, tick = FALSE )

y_ticks_at <- seq(0, round_n(max(diurnal_all_bands_and_modes_data\$posts), 1000000), 100000)
y_labels_at <- seq(0, round_n(max(diurnal_all_bands_and_modes_data\$posts), 1000000), 1000000)
y_tick_labels <- seq(0, round_n(max(diurnal_all_bands_and_modes_data\$posts), 1000000) / 1000000, 1)

axis(side = 2, at = y_ticks_at, labels = FALSE )
axis(side = 2, at = y_labels_at, labels = y_tick_labels, tick = FALSE )

minx <- par("usr")[1]
maxx <- par("usr")[2]
miny <- par("usr")[3]
maxy <- par("usr")[4]

xrange <- maxx - minx
yrange <- maxy - miny

xpos <- minx + 0.025 * xrange
ypos <- miny + 0.975 * yrange

par(xpd=T, mar=c(0,0,4,0))

legend(x = xpos, y = ypos, legend = seq(MIN_YEAR, MAX_YEAR),
       lty=c(1, 1), lwd=c(2,2), col = clrs,
       bty = 'n', text.col = 'black')

graphics.off()

 

Of course, many other insights may be gleaned rather rapidly from the summary file, without recourse to the complete dataset.

2022-03-08

Historical cty.dat files

Usually, the many historical versions of the cty.dat files that may be used to determine the country in which a station is located on a particular date may be downloaded here. Sometimes, however, that link breaks and the files are unavailable. Those files may now also be downloaded from here.   

2022-03-07

HF Beacons and the Reverse Beacon Network, 2021

Here is a table of the twenty fixed-frequency stations most often posted by the RBN in 2021:

Position Station Frequency (kHz) Number of Posts
1 OK1CF
1826 142,854
2 CS3B
14100 133,798
3 YV5B
14100
114,184
4 OH2B
14100
95,011
5 W6WX 14100
90,524
6 I1MMR
7026
74,540
7 4U1UN
14100 68,877
8 HB4FV/B
10134
62,834
9 DL8LAS 1831
60,296
10 4X6TU 14100
58,076
11 LY2PX 14027
51,797
12 4X6TU 18110
50,886
13 IZ3DVW/B 10132
50,431
14 YV5B 18110
49,640
15 W6WX 18110
46,905
16 AA1K 1821
46,591
17 SP3CW 3565
45,686
18 LZ1QI 7006
45,424
19 CS3B 18110
45,152
20 LZ1QI
7004
44,457

Notes:
  1. Frequencies are rounded to the nearest kHz;
  2. I am unsure how the U.S. stations in the list can be legal, since the FCC's regulations appear to limit [unattended] HF beacons to a portion of 10m;
  3. It is my memory that the original HF beacons were all located on 28 MHz, so that listeners could be made aware of an opening. It is noticeable that not a single one of the stations on the list above is on 10m: the vast majority are on bands that can reasonably be expected to support some kind of non-local propagation at almost all times (which is probably the very reason that they are posted by the RBN so often -- but one does wonder what the putative purpose of such a beacon is).

Below are figures showing, for each of the stations in the table above, the signal strength as reported by the ten RBN stations that most frequently posted each individual beacon station.

In the following figures:

  1. The ordinate for each of the strip charts ranges between 0 dB and the value shown as FSD (i.e., full scale deflection) near the bottom right-hand corner; in this case, the maximum value of each strip is therefore 80 dB.
  2. The value plotted in this manner is the value denoted SNR by the RBN.  Remember that the RBN has an odd definition of SNR.
  3. The abscissa is divided into a number of bins of equal duration. On each plot there are 100 such bins; because the duration covered by each plot is one year, each bin therefore covers about 3½ days.
  4. At the bottom of each strip chart is a coloured bar. Each bin in these bars is coloured so as to represent the total number of times that the RBN station spotted the beacon in the period covered by the bin.The colour legend for each figure is to the right of the figure.
  5. For the period covered by each 3½-day bin, the lower quartile of SNR readings is coloured grey, the upper quartile is coloured white, and the middle two quartiles are coloured blue.
  6. The vertical order of the various RBN stations is determined solely by the chronological order in which each station first spotted the beacon.

 


 



 



 

 



 


 

 


 

 

 

 


 


 
 

 

 

 

 

 

 

2022-03-01

Per-Band Reverse Beacon Network Actvity: 2009-2021

The plots shown in this post can be extended on a per-band basis.


So we can plot the total number of posters as a function of time (that is, a more conventional series of plots that represent the data in this figure).










These show a more optimistic picture than the plot of the total number of posters as a function of time. It seems that, while the total number of posters is more or less moribund, the number of posters providing posts on each individual band continues (mostly) to increase.

G(15, 100) as a Function of Time

Turning to the geographical distribution of the posting stations, we can display the mensal values of G(15, 100) in a similar manner:









These show that, on most bands, the geographical distribution of the RBN continues to increase.

2022-02-27

2021 RBN Data

All the postings to the Reverse Beacon Network in 2021, along with the postings from prior years, are now available in this directory.

Some simple annual statistics for the period 2009 to 2021 follow (the 2009 numbers cover only part of that year, as the RBN was instantiated partway through that year).

Total posts:
2009:   5,007,040
2010:  25,116,810
2011:  49,705,539
2012:  71,584,195
2013:  92,875,152
2014:  108,862,505
2015:  116,385,762
2016:  111,027,068
2017:  117,973,111
2018:  131,930,432
2019:  135,558,461
2020:  173,655,453
2021:  180,965,313
  Total posting stations:
2009: 151
2010: 265
2011: 320
2012: 420
2013: 473
2014: 515
2015: 511
2016: 590
2017: 625
2018: 550
2019: 583
2020: 616
2021: 659
 Total posted distinct callsigns:
2009: 143,724
2010: 266,189
2011: 271,133
2012: 308,010
2013: 353,952
2014: 398,293
2015: 433,197
2016: 375,613
2017: 356,461
2018: 361,058
2019: 337,246
2020: 369,580
2021: 376,243
Obviously, statistics that are considerably more comprehensive may be derived rather easily from the files in the directory.

Note that if you intend to use the databaseߴs reported signal strengths in an analysis, you should be sure that you understand the ramifications of what the RBN means by SNR.

2022-02-26

Reverse Beacon Network Actvity: 2009-2021

I here show various plots of the G(15, 100) grid-based scatter metric, G(15, 100), for the Reverse Beacon Network (RBN), using data from the inception of the RBN up to the end of 2020.

As in the past I note that a reasonable a priori case can be made on the basis of propagation characteristics that somewhat different metrics in the G(Δ, n) series might be better representations of RBN coverage on some of the bands. However, rather than make this into a full-scale research project, I shall here simply continue to use the G(15, 100) metric on the basis that it seems "good enough" on all bands.

RBN Posting Stations as a Function of Time


We begin by looking simply at how the number of per-band posters to the RBN has varied since the RBN's inception. (NB Throughout this post, we ignore posters for which the location is not recorded by the RBN; plots for which the abscissa is time show one datum per month.)

First, a plot of the total number of posters as a function of time:


This can be more compactly represented, along with similar per-band data for 160m through 10m (excluding 60m):


 

G(15, 100) as a Function of Time


Turning now to the geographical distribution of the posting stations, we can display the mensal values of G(15, 100) in a similar manner:


 

These figures seem to make rather clearly the point that, while the total number of stations posting to the RBN has increased slightly over the past few years (although in an uncertain manner), the underlying geographical distribution has barely changed, and is essentially static at 30% of the world's surface. One assumes (although I haven't checked) that this is because all the "easy" land-based locations now have at least one spotting station, and any expansion of the network, if such is desired, will probably have to concentrate on island states.

G(15, 100) as a Function of the Number of Posters

Finally, we can combine the mensal values of G(15, 100) and the number of posters. Firstly, including all bands:

The summary plot for these data is slightly different, as the ordinate is multi-valued for some values of the abscissa. So, in this summary plot, we take the mean value of G(15, 100) in bins of width equivalent to ten posters, and plot rectangles in the equivalent colours:

All in all, a picture emerges in which the RBN, after expanding and increasing coverage for the better part of a decade, became essentially static in early 2017 and is now more or less in statis.