2017-02-13

Compressed Archive of LANL GPS Charged-Particle Data

Recently, the National Oceanic and Atmospheric Administration (NOAA) made available to the public a 16-year dataset of measurements of the charged-particle environment at 23 GPS satellites prepared  by the Combined X-Ray Dosimeter (CXD) team at Los Alamos National Laboratory. The dataset comprises a hierarchy of directories that total about 65 GB in size. As the dataset is uncompressed, I have made available a tarred, compressed mirror of the dataset, which is about 5.7 GB in size. This mirror has the MD5 checksum: 443c5787665a9ff7456ca037101b72e0. All the files in the original LANL release are contained in the mirror, without change.
The basic documentation for the dataset is contained in a README file; the following description of the data is taken from that file:

For all satellites except Space Vehicle Navstar (SVN) numbers 41 and 48, the data are:

Variable name type Dim. description
decimal_day
double


1 GPS time, a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years.
Geographic_Latitude double 1 Latitude of satellite (deg)
Geographic_Longitude double 1 Longitude of satellite (deg)
Rad_Re double 1 (radius of satellite)/Rearth
rate_electron_measured double 11 Measured rate (Hz) in each of the 11 CXD electron channels
rate_proton_measured double 5 Measured rate (Hz) in each of the 5 CXD proton channels (P1-P5)
LEP_thresh double 1 LEP threshold in E1 channels (0 means low, 1 means high)
collection_interval double 1 dosimeter collection period (seconds)
year int 1 year (e.g. 2015)
decimal_year double 1 decimal year = year + (decimal_day-1.0)/(days in year)
SVN_number int 1 SVN number of satellite
dropped_data int 1 if =1 it means something is wrong with the data record, do not use it
b_coord_radius double 1 radius from earth's dipole axis (earth radii)
b_coord_height double 1 height above the earth's dipole equatorial plane (earth radii)
magnetic_longitude double 1 Magnetic longitude (degrees)
L_shell


double 1 L_shell (earth radii) – currently this is the same as L_LGM_T89IGRF but this is intended to be our suggested choice for the L shell calculation in the long run.
L_LGM_TS04IGRF double 1 LanlGeoMag L-shell McIlwain calculation, TS04 External Field, IGRF Internal Field.
L_LGM_OP77IGRF double 1 LanlGeoMag L-shell McIlwain calculation, OP77 External Field, IGRF Internal Field (not currently filled)
L_LGM_T89CDIP double 1 LanlGeoMag L-shell McIlwain calculation, T89 External Field, Centered Dipole Internal Field
L_LGM_T89IGRF double 1 LanlGeoMag L-shell McIlwain calculation, T89 External Field, IGRF Internal Field
bfield_ratio double 1 Bsatellite/Bequator
local_time double 1 magnetic local time (0-24 hours)
utc_lgm double 1 UTC (0-24 hours)
b_sattelite double 1 B field at satellite (gauss)
b_equator double 1 B field at equator (on this field line I think) (gauss)
electron_background double 11 estimated background in electron channels E1-E11 (Hz)
proton_background double 5 estimated background in proton channels P1-P5 (Hz)
proton_activity int 1 =1 if there is significant proton activity
proton_temperature_fit


double 1 characteristic momentum -- R0 in the expression given above (MeV/c)
proton_density_fit double 1 N0 parameter in fit to proton flux ((protons/(cm2 sec sr MeV))
electron_temperature_fit


double 1 electron temperature from a one Maxwellian fit (MeV)
electron_density_fit double 1 electron number density from a one Maxwellian fit (cm-3)
model_counts_electron_fit_pf double 11 E1-E11 rates due to proton background based on proton flux fit -- currently not filled (all -1's)
model_counts_proton_fit_pf double 5 P1-P5 rate from proton fit (using proton_temperature_fit, proton_density_fit)
model_counts_electron_fit double 11 E1-E11 rates from the 9-parameter electron flux model
model_counts_proton_fit double 5 P1-P5 rates from electron background -- currently not filled (all -1's)
proton_integrated_flux_fit double 6 integral of proton flux (based on fit) above 10, 15.85, 25.11, 30, 40, 79.43 MeV (proton kinetic energy)
proton_flux_fit


double 31 intended to be proton flux at 31 energies, not filled currently
proton_fluence_fit double 6
not filled currently


integral_flux_instrument


double 30 (based on 9 parameter fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec)
integral_flux_energy double 30
energies for the integral of integral_flux_instrument (MeV)


electron_diff_flux_energy double 15 energies for the fluxes in electron_diff_flux_energy (MeV)
electron_diff_flux double 15 (based on 9 parameter fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec))
Efitpars double 9 fit parameters for 9 parameter electron fit
Pfitpars double 4 Fit parameters for 4 parameter proton fit. These are still a work in progress. The parameters are here as placeholders until we finalize the fit function and parameters.
 
 
For SVN numbers 41 and 48, the data are:


Variable name type Dim. Description
decimal_day
double


1 GPS time -- a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years
Geographic_Latitude double 1 Latitude of satellite (deg)
Geographic_Longitude double 1 Longitude of satellite (deg)
Rad_Re double 1 (radius of satellite)/Rearth
rate_electron_measured double 8 Measured rate (Hz) in each of the 8 BDD electron channels (E1-E8)
rate_proton_measured double 8 Measured rate (Hz) in each of the 8 BDD proton channels (P1-P8)
collection_interval double 1 dosimeter collection period (seconds)
year int 1 year (e.g. 2015)
decimal_year double 1 decimal year = year + (decimal_day-1.0)/(days in year)
svn_number int 1 SVN number of satellite
dropped_data int 1 if =1 it means something is wrong with the data record, do not use it
b_coord_radius double 1 radius from earth's dipole axis (earth radii)
b_coord_height double 1 height above the earth's dipole equatorial plane (earth radii)
magnetic_longitude double 1 Magnetic longitude (degrees)
L_shell


double 1 L_shell (earth radii) -- I do not clearly understand the origin of the calculation, but it seems to be a dipole field/T-89
bfield_ratio double 1 Bsatellite/Bequator
local_time double 1 magnetic local time (0-24 hours)
b_sattelite double 1 B field at satellite (gauss)
b_equator double 1 B field at equator (on this field line I think) (gauss)
Diffp double 1 No longer used
sigmap double 1 No longer used
electron_background double 8 estimated background in electron channels E1-E8 (Hz)
proton_background double 8 estimated background in proton channels P1-P8 (Hz)
proton_activity int 1 =1 if there is significant proton activity
electron_temperature double 1 electron temperature from a one Maxwellian fit (MeV)
electron_density_fit double 1 electron number density from a one Maxwellian fit (cm-3)
model_counts_electron_fit double 8 E1-E8 rates from the 2-parameter Maxwellian fit to the electron data
dtc_counts_electron double 8 Dead time corrected electron rates (from data, not fit)
integral_flux_instrument


double 30 (based on 2 parameter Maxwellian fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec)
integral_flux_energy double 30 energies for the integral of integral_flux_instrument (MeV)
electron_diff_flux_energy double 15 energies for the fluxes in electron_diff_flux_energy (MeV)
electron_diff_flux double 15 (based on 2 parameter Maxwellian fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec))


Notes (confirmed with the CXD team):
  1. Some of the data are given in terms of Earth radii. For the purposes of those measurements, the Earth is assumed to be a sphere of radius 6371 km.
  2. A record's timestamp is the time of the centre of the collection interval.
  3. Magnetic local time is defined as the difference (in units of time, where 24 hours is equivalent to 360°) between the magnetic longitude of the record and the magnetic longitude of the anti-solar point at the time of the record.


2017-02-10

Stations That Have Not Responded to Direct QSL Requests

I generally QSL DX stations worked on new bands/modes via the bureau. Occasionally, if I need the QSL for an award and I have not received a reply card via the bureau after a reasonable period, I will send a second QSL direct, always accompanied by (for US destinations) an SASE, or (for DX destinations) a self-addressed airmail envelope and several dollars (nowadays typically $3, or more if I am requesting multiple cards), hidden as carefully as possible within the sending envelope.

NB (1) If the station operator posted different instructions somewhere such as on qrz.com, I tried to follow those instructions.

NB (2) Some stations have more recently stated that they no longer respond to requests for paper cards. Nevertheless, there was no such indication at the time at which I submitted the request to which no response was received. 

I realise that there are many reasons why I might not receive a response to such a direct request, many of which are not the fault of the other station. Nevertheless, reading around the Internet, some stations do seem consistently not to respond to QSL requests -- even some that advertise that they respond to all requests no matter how they are received.

Emphasising that some of these stations may well be blameless -- international mail in particular seems always to be a bit hit-or-miss -- nevertheless here is a list of those stations to whom direct QSLs have been sent, and from whom no reply was received. All the pertinent QSOs were made in 2010, so there has been plenty of time for a card to arrive via any method.

Station Manager
CO6LP
C6AMS NA6M
HP8/NC6Q NC6Q
OA4BHY
PJ2A N5UCF
PJ5/AA4NC AA4NC
PJ7/K5IL K5IL
PZ1EH
UN7PL
V21C
YT8A YU1EA
7V2PI EA5FL
9K2MU

If you are on this list and believe that I have made a mistake (it happens often enough -- keeping track of several thousand QSLs per year is not an error-free process) please let me know and I will investigate further.

2017-02-06

More on the Growth of the RBN

Recently, I looked briefly at the year-by-year growth of the RBN. In that post, I showed plots of the annual state of the RBN only in toto; i.e., taking the data as a whole, plotted on a latitude/longitude grid. But the RBN does not exist only in the two geographic dimensions: it has a third dimension: frequency (or, of course, its equivalent, wavelength). Here, then, are similar plots showing the posts by the RBN on an annual basis, band by band, from 160m to 10m. To view the plots with sufficient detail to be useful, you will probably have to view the images at full size, rather than as they are rendered on this page.

We begin with an integrated plot showing all the data (so this is like the plots in the last post on this subject, but consolidated into a single image), then follow this with the equivalent plots for the separate bands.

All bands, 160m to 10m

160m

80m

40m

30m

20m

17m

15m

12m

10m



2017-02-02

Revised augmented logs for CQ WW SSB from 2005 to 2016

I recently added another column to the flags on each line of the CQ WW SSB augmented logs for 2005 to 2016 (the MD5 hash of the file is 3ba5bc3d157cb7b7d1974832b82f81e5). However, K0KR reminded me that some logs contain frequencies that merely indicate the band, not the actual frequency of the QSO. In such logs, it is not possible to determine whether a logged QSO was part of a run. Therefore I have changed the values and meaning of the new column (column #11) as described below.

The additions to the standard CQ WW Cabrillo QSO line are now (changes in bold):
  1. The letter "A" or "U" indicating "assisted" or "unassisted"
  2. A four-digit number representing the time if the contact in minutes measured from the start of the contest. (I realise that this can be calculated from the other information on the line, but it saves a lot of time to have the number readily available in the file without having to calculate it each time.)
  3. Band
  4. A set of eleven flags, each -- apart from column k -- encoded as T/F: 
    • a. QSO is confirmed by a log from the second party 
    • b. QSO is a reverse bust (i.e., the second party appears to have bust the call of the first party) 
    • c. QSO is an ordinary bust (i.e., the first party appears to have bust the call of the second party) 
    • d. the call of the second party is unique 
    • e. QSO appears to be a NIL 
    • f. QSO is with a station that did not send in a log, but who did make 20 or more QSOs in the contest 
    • g. QSO appears to be a country mult 
    • h. QSO appears to be a zone mult 
    • i. QSO is a zone bust (i.e., the received zone appears to be a bust)
    • j. QSO is a reverse zone bust (i.e. the second party appears to have bust the zone of the first party)
    • k. This entry has three possible values rather than just T/F:
      • T: QSO appears to be made during a run by the first party
      • F: QSO appears not to be made during a run by the first party
      • U: the run status is unknown because insufficient frequency information is available in the first party's log 
Note that the encoding of some of the flags requires subjective decisions to be made as to whether the flag should be true or false; consequently, and because CQ has yet to understand the importance of making their scoring code public, the value of a flag for a specific QSO line in some circumstances might not match the value that CQ would assign. (Also, CQ has more data available in the form of check logs, which are not made public.)

Note also that I made no attempt to deduce the run status of a QSO in the second party's log (if such exists), regardless of the status in the first party's log. This allows one cleanly to perform correct statistical analyses anent the number of QSOs made by running stations merely by excluding QSOs marked with a U in column k.

Why I Don't Use a Word Processor (2)

Before I begin to explain in some detail why I don't use a word processor to write my books, I should probably interject a post about what I do use instead.

The basic information can be found in the colophon of any of my books, but it's probably a good idea to describe the software and the process here in a bit more detail.

I use two programs to generate the PDFs that are used to print my books: VEDIT PLUS and pdfTeX (the covers use a third program, Scribus).

The Basic Idea

 

A word processor tries to be the universal answer to the problem of creating documents. In particular, it conflates two processes that are fundamentally distinct: the creation of content and the layout of that content on the page. I will go into some detail in a later post why this is not the good idea that it might seem to be, but for now all I need to say is that instead of using a single program to perform these tasks, I use two very different programs, each dedicated to performing just one. I use VEDIT PLUS for the job of actually writing, and pdfTeX for the job of creating the PDFs that I print for the purpose of editing drafts, and for producing the final PDFs that are used to print the published book.

VEDIT PLUS

 

The first task, that of actually writing the text, can be done in any reasonable editor program. There is a huge number of these programs, and which to use is mostly a matter of personal preference, although some useful features may be missing from some low-end editors.

The VEDIT PLUS program that I use (the name has undergone several changes over the years: I'm not even sure that it's not simply called "VEDIT" nowadays, although originally that referred to a version of the program with fewer features) has been around since at least the early 1980s. I have used it as my main editor for about 35 years (starting with a CP/M-86 version, I think), and still use it daily, not only for writing purposes but also for general editing of text files. Although it is a Windows program, I run it in Linux using the WINE compatibility layer that allows Windows programs to run under Linux.

There is really no good reason to use VEDIT PLUS instead of a more well-known and readily available editor such as emacs, but VEDIT PLUS does everything I need, and using the program has long been second nature to me (also, although it's not relevant to the production of books, it is the best and fastest editor I've ever used for working on large files). VEDIT PLUS is the one piece of commercial software that I have on my systems, which I suppose says something about how useful I find it.

pdfTeX

 

pdfTeX is an extended version of the TeX typesetting program, and is capable of producing PDF files natively. (When I started writing, I used the original TeX program as documented in the TeXbook, and converted the output to PDF via a series of conversion programs.)

The process then is:
  1. Write text using VEDIT PLUS;
  2. Use pdfTeX to convert the text file produced by VEDIT PLUS to PDF;
  3. Print the PDF.
So that's what I use. Next time, we'll start to look in more depth at why I use these two programs rather than a single word processor program.

2017-01-29

UA9BA's 160m Signal

Willy, UA9BA, has posted a brief article about his 160m antenna. Therein, he compares his 160m signal to those of RG9A, R8TT and UN7LZ in this year's CQ 160m contest.

Unfortunately, the graphs he presents are impossible to interpret quantitatively and with statistically meaningful confidence limits, a situation exacerbated by the RBN's unique use of the term SNR.

A much clearer picture emerges if we issue the command:
  compare-multiple-sigs 20170128 20170128 160m EU UA9BA RG9A R8TT UN7LZ
which generates this plot:


This tells us immediately that, for the periods in which UA9BA and one of the other stations could be heard by the European portion of the RBN, there is no evidence that there was a substantive difference between the signals from UA9BA and RG9A or between the signals from UA9BA and UN7LZ. We can, however, be 99% certain that UA9BA was stronger than R8TT in Europe, with a signal difference of a few dB.

I note that although Willy refers to the other station as his "local rivals", they are actually rather distant from him, and (if the maps on qrz.com are correct) the stations are actually considerably closer to Europe, which, of course, makes his signal even more impressive.

NB: The graphs in Willy's post show that there are periods when not all the stations generated posts at a given RBN receiver. This is hardly surprising, given the relatively large separation of the stations and the nature of propagation on 160m. A statistically meaningful signal comparison requires enough pairwise measurements be made at about the same time so that a statistical analysis with usefully narrow 99% confidence limits can be performed.