2020-01-12

2020-01-11

Video Maps of CQ WW SSB QSOs, 2005 to 2019

I have updated the set of CQ WW video maps on my youtube channel (channel N7DR) to include the logs from the 2019 running of CQ WW SSB. These video maps cover all the years for which public logs are currently available (2005 to 2019).

To access individual videos directly:

2020-01-06

Reverse Beacon Network Actvity: 2009-2019

I here show various plots of the G(15, 100) grid-based scatter metric, G(15, 100), for the Reverse Beacon Network (RBN), using data from the inception of the RBN up to the end of 2019.

As in the past I note that a reasonable a priori case can be made on the basis of propagation characteristics that somewhat different metrics in the G(Δ, n) series might be better representations of RBN coverage on some of the bands. However, rather than make this into a full-scale research project, I shall here simply continue to use the G(15, 100) metric on the basis that it seems "good enough" on all bands.

RBN Posting Stations as a Function of Time


We begin by looking simply at how the number of per-band posters to the RBN has varied since the RBN's inception. (NB Throughout this post, we ignore posters for which the location is not recorded by the RBN; plots for which the abscissa is time show one datum per month.)

First, a plot of the total number of posters as a function of time:


This can be more compactly represented, along with similar per-band data for 160m through 10m (excluding 60m):


G(15, 100) as a Function of Time


Turning now to the geographical distribution of the posting stations, we can display the mensal values of G(15, 100) in a similar manner:



These figures seem to make rather clearly the rather depressing point that since early 2017 there has been no substantive or sustained increase in either the number or geographical distribution of the stations posting to the RBN.

G(15, 100) as a Function of the Number of Posters


Finally, we can combine the mensal values of G(15, 100) and the number of posters. Firstly, including all bands:

 
The summary plot for these data is slightly different, as the ordinate is multi-valued for some values of the abscissa. So, in this summary plot, we take the mean value of G(15, 100) in bins of width equivalent to ten posters, and plot rectangles in the equivalent colours:


All in all, a rather unhappy picture emerges, in which the RBN, after expanding and increasing coverage rather nicely for the better part of a decade, became essentially static in early 2017 and has effectively failed to expand numerically or in geographical coverage since then.

2020-01-03

2019 RBN data

All the postings to the Reverse Beacon Network in 2019, along with the postings from prior years, are now available in this directory.

Some simple annual statistics for the period 2009 to 2019 follow (the 2009 numbers cover only part of that year, as the RBN was instantiated partway through that year).

Total posts:
2009:   5,007,040
2010:  25,116,810
2011:  49,705,539
2012:  71,584,195
2013:  92,875,152
2014:  108,862,505
2015:  116,385,762
2016:  111,027,068
2017:  117,973,111
2018:  131,930,432
2019:  135,558,461
  Total posting stations:
2009: 151
2010: 265
2011: 320
2012: 420
2013: 473
2014: 515
2015: 511
2016: 590
2017: 625
2018: 550
2019: 583
 Total posted distinct callsigns:
2009: 143,724
2010: 266,189
2011: 271,133
2012: 308,010
2013: 353,952
2014: 398,293
2015: 433,197
2016: 375,613
2017: 356,461
2018: 361,058
2019: 337,246
Obviously, statistics that are considerably more comprehensive may be derived rather easily from the files in the directory.

Note that if you intend to use the databaseߴs reported signal strengths in an analysis, you should be sure that you understand the ramifications of what the RBN means by SNR.

2019-12-20

Cleaned and Augmented Logs for ARRL DX CW and SSB contests, 2018 to 2019

Cleaned Logs


Cleaned versions of the logs for the ARRL DX CW and SSB contests are now available for 2018 and 2019.

Links to the cleaned logs may be followed here.

The cleaned logs are the result of processing the QSO: lines from the entrants' submitted Cabrillo files (as [gratuitously] modified by the ARRL) to ensure that all fields contain valid values and all the data match the column-specific standard format for this contest.

Any line containing illegal data in a field has simply been removed. Also, only the QSO: lines are retained, so that each line in the file can be processed easily. All QTH multipliers are rendered as two letters, and the power is rendered as four digits, regardless of how the submitted log recorded these two fields; this should simplify processing the logs by scripts or programs, as should the use of fixed-length records in these cleaned files.

Augmented Logs


Links to the augmented logs may be followed here.

The augmented logs for the ARRL DX contests contain the same information as the cleaned logs, but with the addition of some useful (derived) information on each line. The information added to each line comprises:
  1. The sequence of four characters that are the same for each entry in a particular log:
    •  a. letter "A" or "U" indicating "assisted" or "unassisted"
    •  b. letter "Q", "L", "H" or "U", indicating respectively QRP, low power, high power or unknown power level
    •  c. letter "S", "M", "C" or "U", indicating respectively a single-operator, multi-operator, checklog or unknown operator category 
    •  d. character "1", "2", "+" or "U", indicating respectively that the number of transmitters is one, two, unlimited or unknown
  2. A four-digit number representing the time if the contact in minutes measured from the start of the contest. (I realise that this can be calculated from the other information on the line, but it saves subsequent processors of the file considerable time to have the number readily available in the file without having to calculate it each time.)
  3. Band
  4. A set of fourteen flags, each -- apart from column k and column n -- encoded as T/F: 
    • a. QSO is confirmed by a log from the second party 
    • b. QSO is a reverse bust (i.e., the second party appears to have bust the call of the first party) 
    • c. QSO is an ordinary bust (i.e., the first party appears to have bust the call of the second party) 
    • d. the call of the second party is unique 
    • e. QSO appears to be a NIL 
    • f. QSO is with a station that did not send in a log, but who did make 20 or more QSOs in the contest 
    • g. QSO appears to be a country mult (may be T for W/VE stations only)
    • h. QSO appears to be a state/province mult (may be T for DX stations only)
    • i. QSO is an exchange bust (i.e., the received exchange appears to be a bust)
    • j. QSO is a reverse exchange bust (i.e. the second party appears to have bust the exchange of the first party)
    • k. This entry has three possible values rather than just T/F:
      • T: QSO appears to be made during a run by the first party
      • F: QSO appears not to be made during a run by the first party
      • U: the run status is unknown because insufficient frequency information is available in the first party's log
    • l. QSO is a dupe
    • m. QSO is a dupe in the second party's log
    • n. RBN information (see below)
  5. If the QSO is a reverse bust, the call logged by the second party; otherwise, the placeholder "-"
  6. If the QSO is an ordinary bust, the correct call that should have been logged by the first party; otherwise, the placeholder "-"
  7. If the QSO is a reverse exchange bust, the exchange logged by the second party; otherwise, the placeholder "-"
  8.  If the QSO is an ordinary exchange bust, the correct exchange that should have been logged by the first party; otherwise, the placeholder "-"

RBN Information


In CW contests from 2009 onwards, the RBN has been active, automatically spotting the frequency at which any station calling CQ was transmitting. To reflect possible use of RBN information, the augmented files include a fourteenth column. For the sake of uniformity, this column is present in all the augmented files, regardless of whether the RBN actually contributed useful information to a particular contest.

Each QSO has one of several characters in the fourteenth column of flags. These characters should be interpreted as follows:

'-'
  No useful RBN-derived information is available for this QSO.

'0'
  The worked station (i.e., the second call on the log line) appears to have begun to CQ on this frequency within (roughly) 60 seconds prior to the QSO.

'A' to 'Z'
  For the nth letter of the alphabet: the worked station appears to have been CQing on this frequency for (roughly) n minutes prior to the QSO.

'+'
  The worked station appears to have been CQing for more than 26 minutes on this frequency.

'<'
  Because the the RBN is distributed, and because each contest entrant station has its own clock, there is generally a skew between the reading of the clock of the station making the QSO and the timestamp from the RBN at which it believes a posting was made (indeed, it's unclear from the RBN's [lack of] documentation exactly how the timestamp on an individual RBN posting is to be interpreted). If the character '<' appears in the the RBN column, it indicates that the raw values of the clocks suggest that the QSO took place up to two minutes before the RBN reported the worked station commencing to CQ at this frequency. When this occurs, the most likely interpretation is that there is non-negligible skew between the two clocks, and the station was actually worked almost as soon as a CQ was posted by the RBN. But it might also mean that the entrant was simply lucky and found the CQing station just as it fired up on a new frequency.

Notes:
  • The encoding of some of the flags requires subjective decisions to be made as to whether the flag should be true or false; consequently, and because the ARRL has yet to understand the importance of making the scoring code public, the value of a flag for a specific QSO line in some circumstances might not match the value that the ARRL has assigned. (Also, the ARRL has additional, non-public, data available.)
  • I made no attempt to deduce or infer the run status of a QSO in the second party's log (if such exists), regardless of the status in the first party's log. This allows one cleanly to perform correct statistical analyses anent the number of QSOs made by running stations merely by excluding QSOs marked with a U in column k.
  • No attempt is made to detect the case in which both participants of a QSO bust the other station's call. This is a problematic situation because of the relatively high probability of a false positive unless both stations log the frequency as opposed to the band. (Also, on bands on which split-frequency QSOs are common, the absence of both transmit and receive frequency is a problem.) Because of the likelihood of false positives, it seems better, given the presumed rarity of double-bust QSOs, that no attempt be made to mark them.
  • The entries for the exchanges in the case of exchange or reverse exchange busts are normalised to two-letter or four-digit values in the same manner as described above for the exchanges in the cleaned logs.

2019-12-13

Revised Augmented Logs for CQ WW CW and SSB Contests, 2005 to 2018

AD1C has recently once more made accessible historical cty.dat and associated files. A copy of the cty,dat files is here.

This allows one to regenerate the augmented contest files, but using data that were current at the time of the contest. A pointer to these revised augmented files is here.

The augmented logs contain the same information as cleaned logs, but with the addition of some useful (derived) information on each line. The information added to each line comprises:
  1. The sequence of four characters that are the same for each entry in a particular log:
    •  a. letter "A" or "U" indicating "assisted" or "unassisted"
    •  b. letter "Q", "L", "H" or "U", indicating respectively QRP, low power, high power or unknown power level
    •  c. letter "S", "M", "C" or "U", indicating respectively a single-operator, multi-operator, checklog or unknown operator category [ the contest organisers have stated that checklogs are not made public, but in fact at least some of them from the early years have been, hence the need for the "C" category ]
    •  d. character "1", "2", "+" or "U", indicating respectively that the number of transmitters is one, two, unlimited or unknown
  2. A four-digit number representing the time if the contact in minutes measured from the start of the contest. (I realise that this can be calculated from the other information on the line, but it saves subsequent processors of the file considerable time to have the number readily available in the file without having to calculate it each time.)
  3. Band
  4. A set of fourteen flags, each -- apart from column k and column n -- encoded as T/F: 
    • a. QSO is confirmed by a log from the second party 
    • b. QSO is a reverse bust (i.e., the second party appears to have bust the call of the first party) 
    • c. QSO is an ordinary bust (i.e., the first party appears to have bust the call of the second party) 
    • d. the call of the second party is unique 
    • e. QSO appears to be a NIL 
    • f. QSO is with a station that did not send in a log, but who did make 20 or more QSOs in the contest 
    • g. QSO appears to be a country mult 
    • h. QSO appears to be a zone mult 
    • i. QSO is a zone bust (i.e., the received zone appears to be a bust)
    • j. QSO is a reverse zone bust (i.e. the second party appears to have bust the zone of the first party)
    • k. This entry has three possible values rather than just T/F:
      • T: QSO appears to be made during a run by the first party
      • F: QSO appears not to be made during a run by the first party
      • U: the run status is unknown because insufficient frequency information is available in the first party's log
    • l. QSO is a dupe
    • m. QSO is a dupe in the second party's log
    • n. RBN information (see below)
  5. If the QSO is a reverse bust, the call logged by the second party; otherwise, the placeholder "-"
  6. If the QSO is an ordinary bust, the correct call that should have been logged by the first party; otherwise, the placeholder "-"
  7. If the QSO is a reverse zone bust, the zone logged by the second party; otherwise, the placeholder "-"
  8.  If the QSO is an ordinary zone bust, the correct zone that should have been logged by the first party; otherwise, the placeholder "-"

RBN Information


In the CW contests from 2009 onwards, the RBN was active, automatically spotting the frequency at which any station calling CQ was transmitting. To reflect possible use of RBN information, the augmented files now include a fourteenth column. For the sake of uniformity, this column is present in all the augmented files, regardless of whether the RBN actually contributed useful information to a particular contest.

Each QSO has one of several characters in the fourteenth column of flags. These characters should be interpreted as follows:

'-'
  No useful RBN-derived information is available for this QSO.

'0'
  The worked station (i.e., the second call on the log line) appears to have begun to CQ on this frequency within (roughly) 60 seconds prior to the QSO.

'A' to 'Z'
  For the nth letter of the alphabet: the worked station appears to have been CQing on this frequency for (roughly) n minutes prior to the QSO.

'+'
  The worked station appears to have been CQing for more than 26 minutes on this frequency.

'<'
  Because the the RBN is distributed, and because each contest entrant station has its own clock, there is generally a skew between the reading of the clock of the station making the QSO and the timestamp from the RBN at which it believes a posting was made (indeed, it's unclear from the RBN's [lack of] documentation exactly how the timestamp on an individual RBN posting is to be interpreted). If the character '<' appears in the the RBN column, it indicates that the raw values of the clocks suggest that the QSO took place up to two minutes before the RBN reported the worked station commencing to CQ at this frequency. When this occurs, the most likely interpretation is that there is non-negligible skew between the two clocks, and the station was actually worked almost as soon as a CQ was posted by the RBN. But it might also mean that the entrant was simply lucky and found the CQing station just as it fired up on a new frequency.

Notes:
  • The encoding of some of the flags requires subjective decisions to be made as to whether the flag should be true or false; consequently, and because CQ has yet to understand the importance of making their scoring code public, the value of a flag for a specific QSO line in some circumstances might not match the value that CQ would assign. (Also, CQ has more data available in the form of check logs, which are generally not made public.)
  • I made no attempt to deduce or infer the run status of a QSO in the second party's log (if such exists), regardless of the status in the first party's log. This allows one cleanly to perform correct statistical analyses anent the number of QSOs made by running stations merely by excluding QSOs marked with a U in column k.
  • No attempt is made to detect the case in which both participants of a QSO bust the other station's call. This is a problematic situation because of the relatively high probability of a false positive unless both stations log the frequency as opposed to the band. (Also, on bands on which split-frequency QSOs are common, the absence of both transmit and receive frequency is a problem.) Because of the likelihood of false positives, it seems better, given the presumed rarity of double-bust QSOs, that no attempt be made to mark them.
  • The entries for the zones in the case of zone or reverse zone busts are normalised to two-digit values.

2019-11-03

drmap

drmap is a program for generating amateur-radio-related maps from USGS National Map data. From the description in the source code, with graphics added inline:

    drmap
      -ant  <antenna height>
     
        The height of the antenna. If -imperial is present, the height is in feet, otherwise it is in metres.
     
      -call <callsign>
     
        The callsign associated with the plot. Must be present.
       
      -cells <number of cells>
     
        The number of cells from the centre of the plot to the edges. The default is 3/8 of the width of the plot, in pixels. For the default width of 800, the value is therefore 300.
       
      -datadir <directory>
     
        The directory that contains USGS GridFloat tiles
       
      -elev
     
        Create an elevation plot: the plotted values are the elevation of each cell as seen from the antenna. Most are therefore negative.
       
      -grad
     
        Create a gradient plot: the plotted values are the gradient of the terrain in the direction from the QTH.
       
      -hzn [distance limit]
     
        Plot the elevation of the horizon around the periphery of the figure. Eye-level is set in the same way as eye-level for the -los option. Only distances out to the distance limit are used in this calculation. If the distance limit value is not present, it is assumed to be the same as the radius of the figure.
       
      -imperial
     
        Use imperial units instead of metric. That is, miles instead of kilometres and feet instead of metres. Applies both to values on the command line and to values on the output plot(s).
       
      -lat <latitude>
     
        Latitude in degrees north. If present, -long should also be present.
       
      -long <longitude>
     
        Longitude in degrees east. If present, -lat should also be present. Note that because the USGS data covers only the US, longitude should be negative; but if it is positive, the program will negate the value before use.
       
      -los
     
        Create a line-of-sight plot in addition to the standard height-field plot. Eye-level is assumed to be 1.5m or 5 feet, unless the -ant option is presewnt, in which case eye-level is the same as the height of the antenna.

      -outdir <directory>
     
        The directory into which the output maps should be written
       
      -radius <distance1[,distance2[,distance3...]]>
     
        One or more radii for the plot(s), in units of km unless -imperial is present, in which case the units are miles.
       
      -qthdb <QTH database filename>
     
        A file linking QTH information to callsigns. Each line of the file should contain three entries pertaining to a station, separated by white space: the callsign, the latitude and the longitude. This database will be used only if one or both of the -lat and -long parameters is missing from the command line.
       
      -sm
     
        USGS tiles are each about 450MB in size. This parameter ("small memory") tells drmap to use the disk files that contain the tiles as-is, rather than moving them into RAM where their contents can be accessed much more quickly. Using this parameter therefore slows access, but means that there is essentially no limit to the number of tiles that may be used to build a plot. drmap automatically stops loading tiles into RAM when there is less than about 500MB of free RAM and switches to using the tiles on disk, so ordinarily there is no need to worry about whether to use the "-sm" parameter. This parameter will be removed in  future versions of drmap if it seems to be unneeded in practice.
       
      -width <pixels>
     
        width, in pixels, of the plot(s). The default is 800. The height is automatically set to be three quarters of this value.
       
    Examples:
      drmap -call n7dr -datadir /zfs1/data/usgs/drmap -outdir /tmp/drmap -qthdb ~/radio/qthdb -imperial -ant 50 -radius 2 -los -hzn 5
     
        Look up the call "n7dr" (case is ignored) in the file "~/radio/qthdb". Each line in that file is of the form:


        <callsign>     <latitude>     <longitude>
       
        In particular the following line appears in that file on my system:


        N7DR        40.108016   -105.051700
       
        The latitude and longitude information for N7DR are extracted from that file.
       
        The program will look for relevant USGS files in the directory "/zfs1/data/usgs/drmap". If it fails to find any needed files, it will download them from the USGS and place them in that directory prior to using them.
       
        The program will write output plots in the directory tmp/drmap.
       
        The program will use imperial units (miles and feet), and assume an antenna 50 feet above ground.
       
        It will generate a height plot displaying a radius of 2 miles around the N7DR QTH:


       
        It will also create a line-of-sight plot (showing the terrain visible from the putative antenna, 50 feet above ground level):


        
        In both plots, the elevation of the horizon as seen from 50 feet above ground level is drawn around the periphery. The program assumes that no contribution to the horizon is more than five miles from the QTH. It also (always) calculates and displays the M[ean] H[eight] A[bove] T[errain] of the antenna for the area inside the provided radius.


      drmap -call RMNP -datadir /zfs1/data/usgs/drmap -outdir /tmp/drmap -lat 40.441358 -long -105.753685 -radius 1
     
        Create a plot for the point 40°.441358N, 105°.753685W (which is in Rocky Mountain National Park).
       
        The program will look for relevant USGS files in the directory "/zfs1/data/usgs/drmap". If it fails to find any needed files, it will download them from the USGS and place them in that directory prior to using them.

        The program will write output plots in the directory tmp/drmap.
       
        The program will use metric units (kilometres and metres).
       
        It will generate a height plot displaying a radius of 1 kilometre around the designated location:




      drmap -call RMNP -datadir /zfs1/data/usgs/drmap -outdir /tmp/drmap -lat
40.441358 -long -105.753685 -radius 10 -los -hzn 5 -grad -elev
   
        Create a plot for the point 40°.441358N, 105°.753685W (which is in Rocky Mountain National Park).
       
        The program will look for relevant USGS files in the directory "/zfs1/data/usgs/drmap". If it fails to find any needed files, it will download them from the USGS and place them in that directory prior to using them.

        The program will write output plots in the directory tmp/drmap.
       
        The program will use metric units (kilometres and metres).
       
        It will generate a height plot displaying a radius of 10 kilometres around the designated location:



       It will also generate a line-of-sight plot, assuming the default eye level of 1.5m:

       
        It will also generate a gradient plot, showing at each point the gradient measured in a direction from the centre of the plot to that point:



        It will also generate an elevation plot (reminiscent of 1960s tie-dye), showing at each point the elevation angle measured in a direction from the centre of the plot to that point:


        In all these plots, the elevation of the horizon as seen from eye level is drawn around the periphery. The program assumes that no contribution to the horizon is more than five kilometres from the QTH.