Auroral buzz

VA2BBW sent me this file, recorded by him during an SKCC QSO with me on 10m. The QSO took place on 2023-09-25 at 1353z, while Ante was operating as VC3K/VE2 in the SKCC Canadian Straight Key Month event. Normally, 10m is yet to open here at that time, but VA2BBW was a solid S5 here in Colorado, while he gave me an honest 599.

I believe that the QSO was enabled by a burst of auroral activity; the aurora was probably being modulated by the North American electric grid at 60Hz; as we can see from this spectrum plot (taken from the elongated dah in the "d" of "de") the raspy buzz is clearly due to 60Hz modulation.

This is the first time I have heard such a manifestation; normally, auroral QSOs that I have heard display distinct widening of the signal, along with blurring of the individual CW elements as different parts of the auroral region move around at different speeds.

Or to sum it all up in three words: radio is magic.

CQ WPX logs for 2023

CQ WPX SSB and CW logs for 2023 have been added to this repository of public CQ WPX logs.

 

Unofficial Station Reports, ARRL DX SSB and CW, 2018 to 2023

 

Using the public logs, it is rather easy to generate unofficial station-by-station reports for the entrants in the ARRL DX contests.

The ARRL generates official reports and generally makes these reports available individually to each entrant. But these are not made public. The unofficial reports, while not necessarily identical to the official ones, may therefore hold some interest.

The unofficial reports may differ from the official ones because the contest committee has access to checklogs, which are not made public. Also, there are various pathological occurrences in logs that require a decision to be made as to how to classify one or more QSOs; the rules by which such decisions are made are not public, so the decisions that I made when constructing the unofficial reports may well be different from those made by the ARRL. Nevertheless, pathological logs (or pathological QSOs within a log) are relatively rare, so these decisions should affect a relatively small percentage of logs and QSOs. (Typical examples [there are many more] of circumstances in which decisions must made be are: by how much may clocks be skewed and a QSO still be considered valid? what to do if the transmitted callsign changes for some number of QSOs in the contest? what do to if more than one entrant claims to have used the same transmitted callsign?)

The complete set of unofficial reports for the CW and SSB versions of the ARRL DX contest for the years 2018 to 2023 may be found in appropriately named files in this directory.

I note that despite explicitly informing me in 2017 that they would do so, the ARRL have never made public the logs that they hold for the ARRL DX contests for years prior to 2018.

One note regarding interpretation of the information in these unofficial reports: all the fields should be self-explanatory, except that in the listing for EXCHANGE BUSTS, some values are enclosed in parentheses: this indicates that the worked station did not submit a log, and the value of the exchange sent by that station was deduced from QSO: lines in the logs of other entrants.

For example, the report for HL2ZN in the 2021 ARRL DX CW contest contains the line (the line below may be wrapped on your display):

QSO: 14000 CW 2021-02-21 2245 HL2ZN         599 0500   N7DR          599 KY      [ (CO) ]

This indicates that we can deduce that HL2ZN probably bust N7DR's exchange, even though N7DR did not send in a log: HL2ZN recorded N7DR's state as KY, even though N7DR probably sent CO (indeed, I did send CO).

Cleaned and Augmented Logs for ARRL DX CW and SSB contests, 2018 to 2023

 

Cleaned Logs

Cleaned logs for the years 2018 to 2023 may be downloaded from this directory. The cleaned logs are combined into a single file; but data for individual stations and years may be extracted trivially from the combined file.

The cleaned logs are the result of processing the QSO: lines from the entrants' submitted Cabrillo files (as [gratuitously] modified by the ARRL) to ensure that all fields contain valid values and all the data match the column-specific standard format for this contest.

Any line containing illegal data in a field has simply been removed. Also, only the QSO: lines are retained, so that each line in the file can be processed easily. All QTH multipliers are rendered as two letters, and the power is rendered as four digits, regardless of how the submitted log recorded these two fields; this should simplify processing the logs by scripts or programs, as should the use of fixed-length records in these cleaned files.

Augmented Logs

Links to augmented logs for the same years may likewise be downloaded from the same directory. The augmented logs are combined into a single file; but data for individual stations and years may be extracted trivially from the combined file.

The augmented logs for the ARRL DX contests contain the same information as the cleaned logs, but with the addition of some useful (derived) information on each line. The information added to each line comprises:

1. The sequence of four characters that are the same for each entry in a particular log:
•  a. letter "A" or "U" indicating "assisted" or "unassisted"
•  b. letter "Q", "L", "H" or "U", indicating respectively QRP, low power, high power or unknown power level
•  c. letter "S", "M", "C" or "U", indicating respectively a single-operator, multi-operator, checklog or unknown operator category 
•  d. character "1", "2", "+" or "U", indicating respectively that the number of transmitters is one, two, unlimited or unknown
2. A four-digit number representing the time if the contact in minutes measured from the start of the contest. (I realise that this can be calculated from the other information on the line, but it saves subsequent processors of the file considerable time to have the number readily available in the file without having to calculate it each time.)
3. Band
4. A set of fourteen flags, each -- apart from column k and column n -- encoded as T/F: 
   • a. QSO is confirmed by a log from the second party 
   • b. QSO is a reverse bust (i.e., the second party appears to have bust the call of the first party) 
   • c. QSO is an ordinary bust (i.e., the first party appears to have bust the call of the second party) 
   • d. the call of the second party is unique 
   • e. QSO appears to be a NIL 
   • f. QSO is with a station that did not send in a log, but who did make 20 or more QSOs in the contest 
   • g. QSO appears to be a country mult (may be T for W/VE stations only)
   • h. QSO appears to be a state/province mult (may be T for DX stations only)
   • i. QSO is an exchange bust (i.e., the received exchange appears to be a bust)
   • j. QSO is a reverse exchange bust (i.e. the second party appears to have bust the exchange of the first party)
   • k. This entry has three possible values rather than just T/F:
     • T: QSO appears to be made during a run by the first party
     • F: QSO appears not to be made during a run by the first party
     • U: the run status is unknown because insufficient frequency information is available in the first party's log
   • l. QSO is a dupe
   • m. QSO is a dupe in the second party's log
   • n. RBN information (see below)
5. If the QSO is a reverse bust, the call logged by the second party; otherwise, the placeholder "-"
6. If the QSO is an ordinary bust, the correct call that should have been logged by the first party; otherwise, the placeholder "-"
7. If the QSO is a reverse exchange bust, the exchange logged by the second party; otherwise, the placeholder "-"
8.  If the QSO is an ordinary exchange bust, the correct exchange that should have been logged by the first party; otherwise, the placeholder "-"

RBN Information

In CW contests from 2009 onwards, the RBN has been active, automatically spotting the frequency at which any station calling CQ was transmitting. To reflect possible use of RBN information, the augmented files include a fourteenth column. For the sake of uniformity, this column is present in all the augmented files, regardless of whether the RBN actually contributed useful information to a particular contest.

Each QSO has one of several characters in the fourteenth column of flags. These characters should be interpreted as follows:

'-'
  No useful RBN-derived information is available for this QSO.

'0'
  The worked station (i.e., the second call on the log line) appears to have begun to CQ on this frequency within (roughly) 60 seconds prior to the QSO.

'A' to 'Z'
  For the nth letter of the alphabet: the worked station appears to have been CQing on this frequency for (roughly) n minutes prior to the QSO.

'+'
  The worked station appears to have been CQing for more than 26 minutes on this frequency.

'<'
  Because the the RBN is distributed, and because each contest entrant station has its own clock, there is generally a skew between the reading of the clock of the station making the QSO and the timestamp from the RBN at which it believes a posting was made (indeed, it's unclear from the RBN's [lack of] documentation exactly how the timestamp on an individual RBN posting is to be interpreted). If the character '<' appears in the the RBN column, it indicates that the raw values of the clocks suggest that the QSO took place up to two minutes before the RBN reported the worked station commencing to CQ at this frequency. When this occurs, the most likely interpretation is that there is non-negligible skew between the two clocks, and the station was actually worked almost as soon as a CQ was posted by the RBN. But it might also mean that the entrant was simply lucky and found the CQing station just as it fired up on a new frequency.

Notes:

• The encoding of some of the flags requires subjective decisions to be made as to whether the flag should be true or false; consequently, and because the ARRL has yet to understand the importance of making the scoring code public, the value of a flag for a specific QSO line in some circumstances might not match the value that the ARRL has assigned. (Also, the ARRL has additional, non-public, data available.)
• I made no attempt to deduce or infer the run status of a QSO in the second party's log (if such exists), regardless of the status in the first party's log. This allows one cleanly to perform correct statistical analyses anent the number of QSOs made by running stations merely by excluding QSOs marked with a U in column k.
• No attempt is made to detect the case in which both participants of a QSO bust the other station's call. This is a problematic situation because of the relatively high probability of a false positive unless both stations log the frequency as opposed to the band. (Also, on bands on which split-frequency QSOs are common, the absence of both transmit and receive frequency is a problem.) Because of the likelihood of false positives, it seems better, given the presumed rarity of double-bust QSOs, that no attempt be made to mark them.
• The entries for the exchanges in the case of exchange or reverse exchange busts are normalised to two-letter or four-digit values in the same manner as described above for the exchanges in the cleaned logs.

ARRL DX CW and SSB logs for 2023

The raw logs for the ARRL DX CW and SSB contests for 2023 have been added to this directory.

 

WAG Contest Logs 2017 to 2022

Logs for the Worked All Germany contest from 2017 to 2022 are now available in this repository of public WAG logs.

Fixing Problems After Debian Bullseye -> Bookworm

First three items are from upgrade of my shack computer.

Item #4 is from upgrade of my main desktop computer.

Item #5 is from my home server. 

Item #6 occurred on both the main desktop computer and the home server.

 

Following an error-free upgrade from debian bullseye to bookworm, I encountered the following problems:

1. Problem: The DejaVu Sans Mono font was no longer available.

Solution: I installed the package fonts-dejavu.

2. Problem: Even though the termName was set to xterm-256color in the ~/.Xresources file, the actual value of TERM in an xterm was "xterm".

Solution: Explicitly export TERM=256-color in the ~/.bashrc file. 

Comment: This looks like a bug; setting termName in the resources file has always worked in the past.

3. Problem: core dumps were no longer written to the current working file. (In fact, it wasn't clear where they were being written.)

Solution: Add the line:

kernel.core_pattern = core

to /etc/sysctl.conf.

Comment: I have been using versions of UNIX since the late 1980s. Every single version until debian bookworm would automatically dump core to the current working directory. This seems to be Yet One More example of the madness of systemd taking everything over. Why debian would change the default behaviour of something on which so many people rely seems ridiculous. But then, so did their decision to support systemd.

4. Problem: autotrash has been removed by the upgrade.

The autotrash website says that the script can be installed with:

 pip install --user autotrash

However, that produces:

 [ZB:bin] pip install --user autotrash
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
   python3-xyz, where xyz is the package you are trying to
   install.
    
   If you wish to install a non-Debian-packaged Python package,
   create a virtual environment using python3 -m venv path/to/venv.
   Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
   sure you have python3-full installed.
    
   If you wish to install a non-Debian packaged Python application,
   it may be easiest to use pipx install xyz, which will manage a
   virtual environment for you. Make sure you have pipx installed.
    
   See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
[ZB:bin]

Solution:

Install the python3-full package. Then execute:

[ZB:~] python3 -m venv ~/.venvs/autotrash
[ZB:~] ~/.venvs/autotrash/bin/python -m pip install autotrash
Collecting autotrash
 Downloading autotrash-0.4.5-py3-none-any.whl (22 kB)
Installing collected packages: autotrash
Successfully installed autotrash-0.4.5
[ZB:~]

Now autotrash can be run as:

 ~/.venvs/autotrash/bin/autotrash

5. Problem: no networking.

This computer uses two ethernet ports: one to my ISP and hence to the Internet; one to my home LAN. On completing the upgrade (without error) and rebooting the machine, neither network was functioning.

The networking was originally configured as part of the process of installing stretch on this machine, probably about six years ago; networking has worked flawlessly since then, until this upgrade.

Solution:

The directory /etc/NetworkManager/system-connections directory contains two files, rather oddly called:

• 'Wired connection enp11s0(eth0).nmconnection'

• 'Wired connection enp12s0(eth1)'
  

The contents of these files are:

Wired connection enp11s0(eth0).nmconnection:

[connection]
id=Wired connection enp11s0(eth0)
uuid=8e308949-5232-47aa-b9b6-8aab6369ec4d
type=ethernet
permissions=

[ethernet]
mac-address-blacklist=

[ipv4]
address1=209.97.232.18/24,209.97.232.1
dns=127.0.0.1;209.97.224.2;209.97.224.3;
dns-search=
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

[proxy]

Wired connection enp12s0(eth1):

[connection]
id=Wired connection enp12s0(eth1)
uuid=251dd901-f8b7-42ee-b568-b15ba565a81b
type=ethernet
permissions=

[ethernet]
mac-address=D8:50:E6:C2:76:03
mac-address-blacklist=

[ipv4]
address1=192.168.0.1/24
dns-search=
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

The fix is to add a MAC address to the first file.

I note that it is odd that this is necessary: as enp12s0 is mapped directly to a particular port by the MAC address, Network Manager should know that enp11s0 should therefore be mapped to the other port. It has worked that way on this machine since stretch, but apparently this behaviour has changed in bookworm.

It took quite a while to figure this out; more details of the flailing around can be found in the threads that start at:

https://lists.debian.org/debian-user/2023/09/msg00024.html and 

https://lists.debian.org/debian-user/2023/09/msg00061.html.

There is also an official response to the issue at:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051086,

which I note is remarkably unhelpful (if only because it suggests use of a utility that is not part of debian stable; but it also suggests that the problem is due to kernel timing, which is hard to believe given the discussion in the threads on lists.debian.org -- can a problem that depends on timing of the order of many seconds really be due to a change in kernel timing? -- also, the suggested fix seems less clean than the one I provide above; and, in any case, if a human can figure out which connection goes with which port, Network Manager should have been able to do the same. And Network Manager should in no circumstance have tried to assign two connections to each other, as is described in the first lists.debian.org post; the worst that should have happened is that the connection without a MAC address should have failed, and a useful error message provided).

6. Problem: Errors occur when attempting to connect to remote computers over ssh.

When trying to connect to remote systems over ssh, various error messages would be received. This is a typical such message:

Unable to negotiate with 77.109.148.18 port 22: no matching host key type found. Their offer: ssh-dss,ssh-ed25519

The messages all suggested that ssh was unable to find a common acceptable key type.

It seems that the default types of keys that are deemed acceptable must have changed in bookworm. The easiest fix is simply to restore the old keys types as being acceptable, by adding the following lines to ~/.ssh/config:

PubkeyAcceptedKeyTypes +ssh-rsa
PubkeyAcceptedKeyTypes +ssh-dss

HostKeyAlgorithms +ssh-rsa
HostkeyAlgorithms +ssh-dss