Summary File for RBN data, 2009 to 2021

This is an update to a similar post last year.

The complete set of RBN data for 2009 to the end of 2021, after uncompression, easily exceeds 100GB in size. As not all analyses need the complete dataset, I have constructed a summary file (rbn-summary-data.xz in the above-linked directory to the complete set of RBN data) that contains an overview of the data and which is sufficient for many kinds of analysis that do not depend on the details of individual posts to the RBN. (The basic script used to generate this summary file may be found here; the actual summary file is created by running this basic script for each individual year from 2009 to 2021 and concatenating the results after removing the header line from all except the first year.)

The summary file, after being uncompressed, comprises a single large table of values separated by white space. The name of each column (there are twelve columns in all) is on the first row. The columns are:

1. band: a string that identifies the band pertaining to this row. Typical values are "15m" or "160m"; if a row contains data that are not distinguished by band, then the characters "NA" are used.
2. mode: a string that identifies the mode pertaining to this row. Typical values are "CW" or "RTTY"; if a row contains data that are not distinguished by mode, then the characters "NA" are used.
3. type: a single character that identifies whether the data on this row are for a period of a year ("A"), a month ("M") or a day ("D").
4. year: the numeric four-digit value of the year to which the current row pertains.
5. month: the numeric value of the month (January = 1, etc.) of the data in this row. If the data are of type A or D, then this element has the value "NA".
6. doy: the numeric value of the day number of the year (January 1st = 1, etc.). The maximum value in each year is 366 (even if the year is not a leap year). In the event that the year is not a leap year, the data in columns 7, 8 and 9 will be set to 0 when doy is 366. If the data are of type A or M, then this element has the value "NA".
7. posts: the total number of posts recorded by the RBN for the band, mode and period identified by the first six columns. 
8. calls: the total number of distinguishable calls recorded by the RBN for the band, mode and period identified by the first six columns. 
9. posters: the total number of distinguishable posters recorded by the RBN for the band, mode and period identified by the first six columns. 
10. scatter: the value of a scatter metric that characterises the geography of the RBN for the band, mode and period identified by the first six columns. The scatter metric is the sum of all possible distance pairs of good posters (measure in km), divided by the number of distance pairs.
11. good posters: the total number of distinguishable posters recorded by the RBN for the band, mode and period identified by the first six columns, and for which location data are available from the RBN.
12.  grid metric: the total number of G(15, 100) grid cells that contain good posters.

For example, the first two lines of the summary file are (presented here as a table, in order to make it easier to view on mobile devices):

band	NA
mode	NA
type	A
year	2009
month	NA
doy	NA
posts	5007040
calls	143724
posters	151
scatter	5541
good_posters	150
grid_metric	22

This tells us that the first line of actual data in the file comprises annual data for the year 2009, with no separation by band or mode. In 2009, we see that there were 50,007,040 posts of 143,724 callsigns by 151 posters; the scatter metric, which is a measure of the geographic dispersion of the posters on the RBN., was 5,541; 150 different posters contributed the data, spread across 22 distinct G(15, 100) grid cells.

The summary file allows rather rapid analysis of many RBN overview statistics. For example, a plot of the daily number of posts covering the period from the inception of the RBN to the end of 2021 --

-- can be generated on a desktop PC in just a few seconds. From this plot, for example, we can immediately see that the largest number of daily posts occurred during the 2021 running of the CQ WW CW contest in late November (the second-highest cluster of peaks is for the CQ WPX contest, and the third is for the ARRL DX CW contest); also, the burst of activity that coincides with weekends is unmistakable.

For what it's worth, this is the code I used to generate the above plot (I apologise for the awful layout caused by the wrapping of long lines as they are forced into the narrow format used by blogger.com):

#!/usr/bin/Rscript

# generate a plot of the diurnal number of posts by the RBN, stacked by year

MIN_YEAR <- 2009
MAX_YEAR <- 2021

filename <- "/zd1/rbn/rbn-summary-data"  # the local location of the RBN summary data file

# first two lines of the file:
#band          mode          type          year         month           doy         posts         calls       posters       scatter  good_posters   grid_metric
#NA            NA             A          2009            NA            NA       5007040        143724           151          5541           150            22

# rounding function
round_n <- function(x, n) { return ( ( as.integer( (x - 1) / n) +1 ) * n ) }    # function to return next higher integral multiple of n, unless value is already such a multiple

data <- read.table(filename, header=TRUE)

# select diurnal data
diurnal_data <- subset(data, type=='D')

# drop the per-band and per-mode data
diurnal_all_bands_and_modes_data <- subset(diurnal_data, is.na(band) & is.na(mode))

# drop a bunch of columns that we don't want from the summary file
diurnal_all_bands_and_modes_data\$band <- NULL
diurnal_all_bands_and_modes_data\$mode <- NULL
diurnal_all_bands_and_modes_data\$type <- NULL
diurnal_all_bands_and_modes_data\$month <- NULL
diurnal_all_bands_and_modes_data\$calls <- NULL
diurnal_all_bands_and_modes_data\$posters <- NULL
diurnal_all_bands_and_modes_data\$scatter <- NULL
diurnal_all_bands_and_modes_data\$good_posters <- NULL
diurnal_all_bands_and_modes_data\$grid_metric <- NULL

# get ready to start to plot
graphics.off()

png(filename=paste(sep="", "/tmp/rbn-posts-from-summary.png"),  width=800, height=600)

x_lab <- 'DOY'

#            2009   2010                2011      2012     2013    2014      2015               2016      2017             2018     2019   2020
clrs <- c("black", "red", rgb(0.1, 0.1, 0.5), "yellow", "green", "blue", "violet", rgb(0.6, 0.2, 0.2), "white", "cornflowerblue", "gold1", "darkorange", "darkgreen")

# create a frame to map between year and days in the year
days_in_year <- data.frame(seq(MIN_YEAR, MAX_YEAR), 365)
names(days_in_year) <- c("year", "days")

days_in_year\$days[days_in_year\$year %% 4 == 0] <- 366

# set boundaries
plot(0, 0, xlim = c(0.5, 366.5), ylim = c(0, round_n(max(diurnal_all_bands_and_modes_data\$posts), 1000000)), xaxt = "n", yaxt = "n", xlab = x_lab, ylab = "", type = 'n', yaxs="i")         # define the plotting region, but don't actually plot anything
rect(par("usr")[1], par("usr")[3], par("usr")[2], par("usr")[4], col = 'grey')

# now generate the plot, superimposing each year
for (this_year in seq(MIN_YEAR, MAX_YEAR))
{ this_years_data <- subset(diurnal_all_bands_and_modes_data, year==this_year)
  max_element <- days_in_year\$days[this_year - MIN_YEAR + 1]
 
# set up so that this_years_data\$id_365[365] = this_years_data\$id_366[366] = 366, so either column can be used as a vector of abscissæ,
# depending on whether there are 365 or 366 ordinate values
  this_years_data\$id_366<-seq.int(nrow(this_years_data))
  this_years_data\$id_365<-((this_years_data\$id_366 - 1) * 365.0 / 364.0) + 1

# remove a couple of columns that we no longer need
  this_years_data\$doy <- NULL
  this_years_data\$year <- NULL

# move the two new columns of days to the left of the frame: 365, then 366
  this_years_data <- (this_years_data[ c(ncol(this_years_data), ncol(this_years_data) - 1, 1:(ncol(this_years_data)-2))])


  lines(this_years_data[,(max_element-364)][1:max_element], this_years_data\$posts[1:max_element], type = 'l', col = clrs[this_year - MIN_YEAR + 1], lwd = 1) 
}

title_str <- paste(sep="", 'RBN POSTS PER DAY')
title(title_str)

title(ylab = '# OF POSTS (m)', line = 2.1, cex.lab = 1.0)

x_ticks_at <- c(1, 31, 61, 91, 121, 151, 181, 211, 241, 271, 301, 331, 361)
x_labels_at <- x_ticks_at
x_tick_labels <- x_ticks_at

axis(side = 1, at = x_ticks_at, labels = FALSE )    # ticks on x axis
axis(side = 1, at = x_labels_at, labels = x_tick_labels, tick = FALSE )

y_ticks_at <- seq(0, round_n(max(diurnal_all_bands_and_modes_data\$posts), 1000000), 100000)
y_labels_at <- seq(0, round_n(max(diurnal_all_bands_and_modes_data\$posts), 1000000), 1000000)
y_tick_labels <- seq(0, round_n(max(diurnal_all_bands_and_modes_data\$posts), 1000000) / 1000000, 1)

axis(side = 2, at = y_ticks_at, labels = FALSE )
axis(side = 2, at = y_labels_at, labels = y_tick_labels, tick = FALSE )

minx <- par("usr")[1]
maxx <- par("usr")[2]
miny <- par("usr")[3]
maxy <- par("usr")[4]

xrange <- maxx - minx
yrange <- maxy - miny

xpos <- minx + 0.025 * xrange
ypos <- miny + 0.975 * yrange

par(xpd=T, mar=c(0,0,4,0))

legend(x = xpos, y = ypos, legend = seq(MIN_YEAR, MAX_YEAR),
       lty=c(1, 1), lwd=c(2,2), col = clrs,
       bty = 'n', text.col = 'black')

graphics.off()
 

Of course, many other insights may be gleaned rather rapidly from the summary file, without recourse to the complete dataset.