Rationalisation of the LANL GPS Charged-Particle Dataset

Earlier posts in this series:

• Original data
• Stages 1 to 4
• Revised stages 1 to 5 

We now begin to rationalise the data by removing fields that are unused or provide no useful information.

Stage 6: remove dropped_data

The data table for ns41 and ns48, as described in the README file, is reproduced below, with the addition of the column number for each variable. We can now remove the dropped_data field, since that always has the value zero (because of the filtering at stage 3, which removed all records in which the dropped_data field was non-zero).
 

Column	Variable name	type	Dim.	Description
1	decimal_day	double	1	GPS time -- a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years
2	Geographic_Latitude	double	1	Latitude of satellite (deg)
3	Geographic_Longitude	double	1	Longitude of satellite (deg)
4	Rad_Re	double	1	(radius of satellite)/Rearth
5-12	rate_electron_measured	double	8	Measured rate (Hz) in each of the 8 BDD electron channels (E1-E8)
13-20	rate_proton_measured	double	8	Measured rate (Hz) in each of the 8 BDD proton channels (P1-P8)
21	collection_interval	double	1	dosimeter collection period (seconds)
22	year	int	1	year (e.g. 2015)
23	decimal_year	double	1	decimal year = year + (decimal_day-1.0)/(days in year)
24	svn_number	int	1	SVN number of satellite
25	dropped_data	int	1	if =1 it means something is wrong with the data record, do not use it
26	b_coord_radius	double	1	radius from earth's dipole axis (earth radii)
27	b_coord_height	double	1	height above the earth's dipole equatorial plane (earth radii)
28	magnetic_longitude	double	1	Magnetic longitude (degrees)
29	L_shell	double	1	L_shell (earth radii) -- I do not clearly understand the origin of the calculation, but it seems to be a dipole field/T-89
30	bfield_ratio	double	1	Bsatellite/Bequator
31	local_time	double	1	magnetic local time (0-24 hours)
32	b_sattelite	double	1	B field at satellite (gauss)
33	b_equator	double	1	B field at equator (on this field line I think) (gauss)
34	Diffp	double	1	No longer used
35	sigmap	double	1	No longer used
36-43	electron_background	double	8	estimated background in electron channels E1-E8 (Hz)
44-51	proton_background	double	8	estimated background in proton channels P1-P8 (Hz)
52	proton_activity	int	1	=1 if there is significant proton activity
53	electron_temperature	double	1	electron temperature from a one Maxwellian fit (MeV)
54	electron_density_fit	double	1	electron number density from a one Maxwellian fit (cm-3)
55-62	model_counts_electron_fit	double	8	E1-E8 rates from the 2-parameter Maxwellian fit to the electron data
63-70	dtc_counts_electron	double	8	Dead time corrected electron rates (from data, not fit)
71-100	integral_flux_instrument	double	30	(based on 2 parameter Maxwellian fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec)
101-130	integral_flux_energy	double	30	energies for the integral of integral_flux_instrument (MeV)
131-145	electron_diff_flux_energy	double	15	energies for the fluxes in electron_diff_flux_energy (MeV)
146-160	electron_diff_flux	double	15	(based on 2 parameter Maxwellian fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec))

It is easy to remove the dropped_data field:
 awk '{$25=""; print $0}' ns<nn> | tr -s " " > ../gps-stage-6/ns<nn>

Similarly, the data table for the other satellites is reproduced below with the addition of the column number for each variable, and we can remove the dropped_data field from this table also, since that also always has the value zero (because of the filtering at stage 3).

Column	Variable name	type	Dim.	description
1	decimal_day	double	1	GPS time, a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years.
2	Geographic_Latitude	double	1	Latitude of satellite (deg)
3	Geographic_Longitude	double	1	Longitude of satellite (deg)
4	Rad_Re	double	1	(radius of satellite)/Rearth
5-15	rate_electron_measured	double	11	Measured rate (Hz) in each of the 11 CXD electron channels
16-20	rate_proton_measured	double	5	Measured rate (Hz) in each of the 5 CXD proton channels (P1-P5)
21	LEP_thresh	double	1	LEP threshold in E1 channels (0 means low, 1 means high)
22	collection_interval	double	1	dosimeter collection period (seconds)
23	year	int	1	year (e.g. 2015)
24	decimal_year	double	1	decimal year = year + (decimal_day-1.0)/(days in year)
25	SVN_number	int	1	SVN number of satellite
26	dropped_data	int	1	if =1 it means something is wrong with the data record, do not use it
27	b_coord_radius	double	1	radius from earth's dipole axis (earth radii)
28	b_coord_height	double	1	height above the earth's dipole equatorial plane (earth radii)
29	magnetic_longitude	double	1	Magnetic longitude (degrees)
30	L_shell	double	1	L_shell (earth radii) – currently this is the same as L_LGM_T89IGRF but this is intended to be our suggested choice for the L shell calculation in the long run.
31	L_LGM_TS04IGRF	double	1	LanlGeoMag L-shell McIlwain calculation, TS04 External Field, IGRF Internal Field.
32	L_LGM_OP77IGRF	double	1	LanlGeoMag L-shell McIlwain calculation, OP77 External Field, IGRF Internal Field (not currently filled)
33	L_LGM_T89CDIP	double	1	LanlGeoMag L-shell McIlwain calculation, T89 External Field, Centered Dipole Internal Field
34	L_LGM_T89IGRF	double	1	LanlGeoMag L-shell McIlwain calculation, T89 External Field, IGRF Internal Field
35	bfield_ratio	double	1	Bsatellite/Bequator
36	local_time	double	1	magnetic local time (0-24 hours)
37	utc_lgm	double	1	UTC (0-24 hours)
38	b_sattelite	double	1	B field at satellite (gauss)
39	b_equator	double	1	B field at equator (on this field line I think) (gauss)
40-50	electron_background	double	11	estimated background in electron channels E1-E11 (Hz)
51-55	proton_background	double	5	estimated background in proton channels P1-P5 (Hz)
56	proton_activity	int	1	=1 if there is significant proton activity
57	proton_temperature_fit	double	1	characteristic momentum -- R0 in the expression given above (MeV/c)
58	proton_density_fit	double	1	N0 parameter in fit to proton flux ((protons/(cm2 sec sr MeV))
59	electron_temperature_fit	double	1	electron temperature from a one Maxwellian fit (MeV)
60	electron_density_fit	double	1	electron number density from a one Maxwellian fit (cm-3)
61-71	model_counts_electron_fit_pf	double	11	E1-E11 rates due to proton background based on proton flux fit -- currently not filled (all -1's)
72-76	model_counts_proton_fit_pf	double	5	P1-P5 rate from proton fit (using proton_temperature_fit, proton_density_fit)
77-87	model_counts_electron_fit	double	11	E1-E11 rates from the 9-parameter electron flux model
88-92	model_counts_proton_fit	double	5	P1-P5 rates from electron background -- currently not filled (all -1's)
93-98	proton_integrated_flux_fit	double	6	integral of proton flux (based on fit) above 10, 15.85, 25.11, 30, 40, 79.43 MeV (proton kinetic energy)
99-129	proton_flux_fit	double	31	intended to be proton flux at 31 energies, not filled currently
130-135	proton_fluence_fit	double	6	not filled currently
136-165	integral_flux_instrument	double	30	(based on 9 parameter fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec)
166-195	integral_flux_energy	double	30	energies for the integral of integral_flux_instrument (MeV)
196-210	electron_diff_flux_energy	double	15	energies for the fluxes in electron_diff_flux_energy (MeV)
211-225	electron_diff_flux	double	15	(based on 9 parameter fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec))
226-234	Efitpars	double	9	fit parameters for 9 parameter electron fit
235-238	Pfitpars	double	4	Fit parameters for 4 parameter proton fit. These are still a work in progress. The parameters are here as placeholders until we finalize the fit function and parameters.

To remove the dropped_data field:

  for file in ns[567]*
  do
  awk '{$26=""; print $0}' $file | tr -s " " > ../gps-stage-6/$file
  done

Stage 7: remove Diffp from ns41 and ns48

The documentation for ns41 and ns48 says that the Diffp field is no longer used, so we can remove that field from those files.

Following stage 6, the data table looks like this:

Column	Variable name	type	Dim.	Description
1	decimal_day	double	1	GPS time -- a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years
2	Geographic_Latitude	double	1	Latitude of satellite (deg)
3	Geographic_Longitude	double	1	Longitude of satellite (deg)
4	Rad_Re	double	1	(radius of satellite)/Rearth
5-12	rate_electron_measured	double	8	Measured rate (Hz) in each of the 8 BDD electron channels (E1-E8)
13-20	rate_proton_measured	double	8	Measured rate (Hz) in each of the 8 BDD proton channels (P1-P8)
21	collection_interval	double	1	dosimeter collection period (seconds)
22	year	int	1	year (e.g. 2015)
23	decimal_year	double	1	decimal year = year + (decimal_day-1.0)/(days in year)
24	svn_number	int	1	SVN number of satellite
25	b_coord_radius	double	1	radius from earth's dipole axis (earth radii)
26	b_coord_height	double	1	height above the earth's dipole equatorial plane (earth radii)
27	magnetic_longitude	double	1	Magnetic longitude (degrees)
28	L_shell	double	1	L_shell (earth radii) -- I do not clearly understand the origin of the calculation, but it seems to be a dipole field/T-89
29	bfield_ratio	double	1	Bsatellite/Bequator
30	local_time	double	1	magnetic local time (0-24 hours)
31	b_sattelite	double	1	B field at satellite (gauss)
32	b_equator	double	1	B field at equator (on this field line I think) (gauss)
33	Diffp	double	1	No longer used
34	sigmap	double	1	No longer used
35-42	electron_background	double	8	estimated background in electron channels E1-E8 (Hz)
43-50	proton_background	double	8	estimated background in proton channels P1-P8 (Hz)
51	proton_activity	int	1	=1 if there is significant proton activity
52	electron_temperature	double	1	electron temperature from a one Maxwellian fit (MeV)
53	electron_density_fit	double	1	electron number density from a one Maxwellian fit (cm-3)
54-61	model_counts_electron_fit	double	8	E1-E8 rates from the 2-parameter Maxwellian fit to the electron data
62-69	dtc_counts_electron	double	8	Dead time corrected electron rates (from data, not fit)
70-99	integral_flux_instrument	double	30	(based on 2 parameter Maxwellian fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec)
100-129	integral_flux_energy	double	30	energies for the integral of integral_flux_instrument (MeV)
130-144	electron_diff_flux_energy	double	15	energies for the fluxes in electron_diff_flux_energy (MeV)
145-159	electron_diff_flux	double	15	(based on 2 parameter Maxwellian fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec))

So we remove the Diffp field with the command:

  awk '{$33=""; print $0}' ns<nn> | tr -s " " > ../gps-stage-7/ns<nn>

Stage 8: remove sigmap from ns41 and ns48

The documentation for ns41 and ns48 says that the sigmap field is no longer used, so we can remove that field from those files.

Following stage 7, the data table looks like this:

Column	Variable name	type	Dim.	Description
1	decimal_day	double	1	GPS time -- a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years
2	Geographic_Latitude	double	1	Latitude of satellite (deg)
3	Geographic_Longitude	double	1	Longitude of satellite (deg)
4	Rad_Re	double	1	(radius of satellite)/Rearth
5-12	rate_electron_measured	double	8	Measured rate (Hz) in each of the 8 BDD electron channels (E1-E8)
13-20	rate_proton_measured	double	8	Measured rate (Hz) in each of the 8 BDD proton channels (P1-P8)
21	collection_interval	double	1	dosimeter collection period (seconds)
22	year	int	1	year (e.g. 2015)
23	decimal_year	double	1	decimal year = year + (decimal_day-1.0)/(days in year)
24	svn_number	int	1	SVN number of satellite
25	b_coord_radius	double	1	radius from earth's dipole axis (earth radii)
26	b_coord_height	double	1	height above the earth's dipole equatorial plane (earth radii)
27	magnetic_longitude	double	1	Magnetic longitude (degrees)
28	L_shell	double	1	L_shell (earth radii) -- I do not clearly understand the origin of the calculation, but it seems to be a dipole field/T-89
29	bfield_ratio	double	1	Bsatellite/Bequator
30	local_time	double	1	magnetic local time (0-24 hours)
31	b_sattelite	double	1	B field at satellite (gauss)
32	b_equator	double	1	B field at equator (on this field line I think) (gauss)
33	sigmap	double	1	No longer used
34-41	electron_background	double	8	estimated background in electron channels E1-E8 (Hz)
42-49	proton_background	double	8	estimated background in proton channels P1-P8 (Hz)
50	proton_activity	int	1	=1 if there is significant proton activity
51	electron_temperature	double	1	electron temperature from a one Maxwellian fit (MeV)
52	electron_density_fit	double	1	electron number density from a one Maxwellian fit (cm-3)
53-60	model_counts_electron_fit	double	8	E1-E8 rates from the 2-parameter Maxwellian fit to the electron data
61-68	dtc_counts_electron	double	8	Dead time corrected electron rates (from data, not fit)
69-98	integral_flux_instrument	double	30	(based on 2 parameter Maxwellian fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec)
99-128	integral_flux_energy	double	30	energies for the integral of integral_flux_instrument (MeV)
129-143	electron_diff_flux_energy	double	15	energies for the fluxes in electron_diff_flux_energy (MeV)
144-158	electron_diff_flux	double	15	(based on 2 parameter Maxwellian fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec))

So we remove the sigmap field with the command:

  awk '{$33=""; print $0}' ns<nn> | tr -s " " > ../gps-stage-8/ns<nn>

Stage 9: remove Pfitpars from ns53 to ns73

The documentation for the Pfitpars fields states that the values are placeholders. Looking at the values in the files, one sees that they are out of range for typical double precision C or C++ implementations (for example, in ns53, the value 6.919449e-310 appears multiple times; this number is less that the smallest positive integer (~2e–308) representable in many 64-bit compilers; the number ~2e-308 is the maximum subnormal double precision number representable in accordance with the current (2008) IEEE 754 standard, and also the minimum positive double precision number defined in the prior (1985) version of the standard). Therefore it is reasonable to remove the four fields corresponding to the Pfitpars values, since reading positive values less than about 2e-308 is likely to be problematic on some machines with some compilers.

After stage 6 (and, hence, after stage 8), the data table for these satellites looks like this:

Column	Variable name	type	Dim.	description
1	decimal_day	double	1	GPS time, a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years.
2	Geographic_Latitude	double	1	Latitude of satellite (deg)
3	Geographic_Longitude	double	1	Longitude of satellite (deg)
4	Rad_Re	double	1	(radius of satellite)/Rearth
5-15	rate_electron_measured	double	11	Measured rate (Hz) in each of the 11 CXD electron channels
16-20	rate_proton_measured	double	5	Measured rate (Hz) in each of the 5 CXD proton channels (P1-P5)
21	LEP_thresh	double	1	LEP threshold in E1 channels (0 means low, 1 means high)
22	collection_interval	double	1	dosimeter collection period (seconds)
23	year	int	1	year (e.g. 2015)
24	decimal_year	double	1	decimal year = year + (decimal_day-1.0)/(days in year)
25	SVN_number	int	1	SVN number of satellite
26	b_coord_radius	double	1	radius from earth's dipole axis (earth radii)
27	b_coord_height	double	1	height above the earth's dipole equatorial plane (earth radii)
28	magnetic_longitude	double	1	Magnetic longitude (degrees)
29	L_shell	double	1	L_shell (earth radii) – currently this is the same as L_LGM_T89IGRF but this is intended to be our suggested choice for the L shell calculation in the long run.
30	L_LGM_TS04IGRF	double	1	LanlGeoMag L-shell McIlwain calculation, TS04 External Field, IGRF Internal Field.
31	L_LGM_OP77IGRF	double	1	LanlGeoMag L-shell McIlwain calculation, OP77 External Field, IGRF Internal Field (not currently filled)
32	L_LGM_T89CDIP	double	1	LanlGeoMag L-shell McIlwain calculation, T89 External Field, Centered Dipole Internal Field
33	L_LGM_T89IGRF	double	1	LanlGeoMag L-shell McIlwain calculation, T89 External Field, IGRF Internal Field
34	bfield_ratio	double	1	Bsatellite/Bequator
35	local_time	double	1	magnetic local time (0-24 hours)
36	utc_lgm	double	1	UTC (0-24 hours)
37	b_sattelite	double	1	B field at satellite (gauss)
38	b_equator	double	1	B field at equator (on this field line I think) (gauss)
39-49	electron_background	double	11	estimated background in electron channels E1-E11 (Hz)
50-54	proton_background	double	5	estimated background in proton channels P1-P5 (Hz)
55	proton_activity	int	1	=1 if there is significant proton activity
56	proton_temperature_fit	double	1	characteristic momentum -- R0 in the expression given above (MeV/c)
57	proton_density_fit	double	1	N0 parameter in fit to proton flux ((protons/(cm2 sec sr MeV))
58	electron_temperature_fit	double	1	electron temperature from a one Maxwellian fit (MeV)
59	electron_density_fit	double	1	electron number density from a one Maxwellian fit (cm-3)
60-70	model_counts_electron_fit_pf	double	11	E1-E11 rates due to proton background based on proton flux fit -- currently not filled (all -1's)
71-75	model_counts_proton_fit_pf	double	5	P1-P5 rate from proton fit (using proton_temperature_fit, proton_density_fit)
76-86	model_counts_electron_fit	double	11	E1-E11 rates from the 9-parameter electron flux model
87-91	model_counts_proton_fit	double	5	P1-P5 rates from electron background -- currently not filled (all -1's)
92-97	proton_integrated_flux_fit	double	6	integral of proton flux (based on fit) above 10, 15.85, 25.11, 30, 40, 79.43 MeV (proton kinetic energy)
99-128	proton_flux_fit	double	31	intended to be proton flux at 31 energies, not filled currently
129-134	proton_fluence_fit	double	6	not filled currently
135-164	integral_flux_instrument	double	30	(based on 9 parameter fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec)
165-194	integral_flux_energy	double	30	energies for the integral of integral_flux_instrument (MeV)
195-209	electron_diff_flux_energy	double	15	energies for the fluxes in electron_diff_flux_energy (MeV)
210-224	electron_diff_flux	double	15	(based on 9 parameter fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec))
225-233	Efitpars	double	9	fit parameters for 9 parameter electron fit
234-237	Pfitpars	double	4	Fit parameters for 4 parameter proton fit. These are still a work in progress. The parameters are here as placeholders until we finalize the fit function and parameters.

A simple awk command suffices to remove these fields:

  awk '{$234=$235=$236=$237=""; print $0}' ns<nn> | head -1 | tr -s " " | sed 's/ $//' > ../gps-stage-9/ns<nn>

Stage 10: convert collection_interval to integer

As mention in this post, values of collection_interval are drawn from the small set { 24, 120, 240, 4608 }. Therefore it is unnecessary that the values are reported as doubles in the data tables. For ns 41 and ns48:

Column	Variable name	type	Dim.	description
21	collection_interval	double	1	dosimeter collection period (seconds)

and for the other satellites:

Column	Variable name	type	Dim.	description
22	collection_interval	double	1	dosimeter collection period (seconds)

It is more efficient to change the values to integers, saving a little space and some processing time when the values are read and used. This is easy to accomplish. For ns41 and ns48:

  awk '$21=="4.608000e+03" {$21="4608"}; $21=="2.400000e+02" {$21="240"}; \ 
    $21=="1.200000e+02" {$21="120"}; $21=="2.400000e+01" {$21="24"}; \ 
      {print $0};' ns<nn> > ../gps-stage-10/ns<nn>

and similarly for the other spacecraft:

  awk '$22=="4.608000e+03" {$22="4608"}; $22=="2.400000e+02" {$22="240"}; \ 
    $22=="1.200000e+02" {$22="120"}; $22=="2.400000e+01" {$22="24"}; \ 
      {print $0};' ns<nn> > ../gps-stage-10/ns<nn>

I have uploaded the stage 10 data as another checkpoint, for which the MD5 checksum is: 3742da01b6b133b1c02d1c6dcfd998cc.

At this point the two tables describing the data are as below.

For ns41 and ns48:

Column	Variable name	type	Dim.	Description
1	decimal_day	double	1	GPS time -- a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years
2	Geographic_Latitude	double	1	Latitude of satellite (deg)
3	Geographic_Longitude	double	1	Longitude of satellite (deg)
4	Rad_Re	double	1	(radius of satellite)/Rearth
5-12	rate_electron_measured	double	8	Measured rate (Hz) in each of the 8 BDD electron channels (E1-E8)
13-20	rate_proton_measured	double	8	Measured rate (Hz) in each of the 8 BDD proton channels (P1-P8)
21	collection_interval	int	1	dosimeter collection period (seconds)
22	year	int	1	year (e.g. 2015)
23	decimal_year	double	1	decimal year = year + (decimal_day-1.0)/(days in year)
24	svn_number	int	1	SVN number of satellite
25	b_coord_radius	double	1	radius from earth's dipole axis (earth radii)
26	b_coord_height	double	1	height above the earth's dipole equatorial plane (earth radii)
27	magnetic_longitude	double	1	Magnetic longitude (degrees)
28	L_shell	double	1	L_shell (earth radii) -- I do not clearly understand the origin of the calculation, but it seems to be a dipole field/T-89
29	bfield_ratio	double	1	Bsatellite/Bequator
30	local_time	double	1	magnetic local time (0-24 hours)
31	b_sattelite	double	1	B field at satellite (gauss)
32	b_equator	double	1	B field at equator (on this field line I think) (gauss)
33-40	electron_background	double	8	estimated background in electron channels E1-E8 (Hz)
41-48	proton_background	double	8	estimated background in proton channels P1-P8 (Hz)
49	proton_activity	int	1	=1 if there is significant proton activity
50	electron_temperature	double	1	electron temperature from a one Maxwellian fit (MeV)
51	electron_density_fit	double	1	electron number density from a one Maxwellian fit (cm-3)
52-59	model_counts_electron_fit	double	8	E1-E8 rates from the 2-parameter Maxwellian fit to the electron data
60-67	dtc_counts_electron	double	8	Dead time corrected electron rates (from data, not fit)
68-97	integral_flux_instrument	double	30	(based on 2 parameter Maxwellian fit) integral of electron flux above integral_flux_energy[i] particles/(cm2sec)
98-127	integral_flux_energy	double	30	energies for the integral of integral_flux_instrument (MeV)
128-142	electron_diff_flux_energy	double	15	energies for the fluxes in electron_diff_flux_energy (MeV)
143-157	electron_diff_flux	double	15	(based on 2 parameter Maxwellian fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec))

And for ns53 to ns73:

Column	Variable name	type	Dim.	description
1	decimal_day	double	1	GPS time, a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years.
2	Geographic_Latitude	double	1	Latitude of satellite (deg)
3	Geographic_Longitude	double	1	Longitude of satellite (deg)
4	Rad_Re	double	1	(radius of satellite)/Rearth
5-15	rate_electron_measured	double	11	Measured rate (Hz) in each of the 11 CXD electron channels
16-20	rate_proton_measured	double	5	Measured rate (Hz) in each of the 5 CXD proton channels (P1-P5)
21	LEP_thresh	double	1	LEP threshold in E1 channels (0 means low, 1 means high)
22	collection_interval	int	1	dosimeter collection period (seconds)
23	year	int	1	year (e.g. 2015)
24	decimal_year	double	1	decimal year = year + (decimal_day-1.0)/(days in year)
25	SVN_number	int	1	SVN number of satellite
26	b_coord_radius	double	1	radius from earth's dipole axis (earth radii)
27	b_coord_height	double	1	height above the earth's dipole equatorial plane (earth radii)
28	magnetic_longitude	double	1	Magnetic longitude (degrees)
29	L_shell	double	1	L_shell (earth radii) – currently this is the same as L_LGM_T89IGRF but this is intended to be our suggested choice for the L shell calculation in the long run.
30	L_LGM_TS04IGRF	double	1	LanlGeoMag L-shell McIlwain calculation, TS04 External Field, IGRF Internal Field.
31	L_LGM_OP77IGRF	double	1	LanlGeoMag L-shell McIlwain calculation, OP77 External Field, IGRF Internal Field (not currently filled)
32	L_LGM_T89CDIP	double	1	LanlGeoMag L-shell McIlwain calculation, T89 External Field, Centered Dipole Internal Field
33	L_LGM_T89IGRF	double	1	LanlGeoMag L-shell McIlwain calculation, T89 External Field, IGRF Internal Field
34	bfield_ratio	double	1	Bsatellite/Bequator
35	local_time	double	1	magnetic local time (0-24 hours)
36	utc_lgm	double	1	UTC (0-24 hours)
37	b_sattelite	double	1	B field at satellite (gauss)
38	b_equator	double	1	B field at equator (on this field line I think) (gauss)
39-49	electron_background	double	11	estimated background in electron channels E1-E11 (Hz)
50-54	proton_background	double	5	estimated background in proton channels P1-P5 (Hz)
55	proton_activity	int	1	=1 if there is significant proton activity
56	proton_temperature_fit	double	1	characteristic momentum -- R0 in the expression given above (MeV/c)
57	proton_density_fit	double	1	N0 parameter in fit to proton flux ((protons/(cm2 sec sr MeV))
58	electron_temperature_fit	double	1	electron temperature from a one Maxwellian fit (MeV)
59	electron_density_fit	double	1	electron number density from a one Maxwellian fit (cm-3)
60-70	model_counts_electron_fit_pf	double	11	E1-E11 rates due to proton background based on proton flux fit -- currently not filled (all -1's)
71-75	model_counts_proton_fit_pf	double	5	P1-P5 rate from proton fit (using proton_temperature_fit, proton_density_fit)
76-86	model_counts_electron_fit	double	11	E1-E11 rates from the 9-parameter electron flux model
87-91	model_counts_proton_fit	double	5	P1-P5 rates from electron background -- currently not filled (all -1's)
92-97	proton_integrated_flux_fit	double	6	integral of proton flux (based on fit) above 10, 15.85, 25.11, 30, 40, 79.43 MeV (proton kinetic energy)
99-128	proton_flux_fit	double	31	intended to be proton flux at 31 energies, not filled currently
129-134	proton_fluence_fit	double	6	not filled currently
135-164	integral_flux_instrument	double	30	(based on 9 parameter fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec)
165-194	integral_flux_energy	double	30	energies for the integral of integral_flux_instrument (MeV)
195-209	electron_diff_flux_energy	double	15	energies for the fluxes in electron_diff_flux_energy (MeV)
210-224	electron_diff_flux	double	15	(based on 9 parameter fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec))
225-233	Efitpars	double	9	fit parameters for 9 parameter electron fit

Call Busts and Reverse Busts in CQ WW, 2013

This is the ninth in a series of posts on busts and reverse busts in the CQ WW contests. These posts are based on the augmented versions of the CQ WW public logs.

Prior posts in the series:

• 2005
• 2006 
• 2007
• 2008 
• 2009 
• 2010
• 2011 
• 2012

2013 SSB -- Most Busts
Position	Call	QSOs	Busts	% Busts
1	II9P	14,935	209	1.4
2	IG9Y	13,771	205	1.5
3	D4C	20,645	200	1.0
4	SV9FBG	5,816	185	3.2
5	PJ2T	16,939	185	1.1
6	CN3A	14,098	179	1.3
7	OT5A	12,613	173	1.4
8	HA30S	11,692	172	1.5
9	ED1R	12,152	164	1.3
10	A73A	7,899	161	2.0

2013 SSB -- Most Reverse Busts
Position	Call	QSOs	Reverse Busts	% Reverse Busts
1	HA30S	11,692	498	4.3
2	DF0HQ	11,429	234	2.0
3	HK1NA	17,942	221	1.2
4	JA3YBK	8,050	215	2.7
5	WE3C	10,315	205	2.0
6	JA5FDJ	9,643	193	2.0
7	TC90A	3,182	169	5.3
8	B9/BY9GA	6,295	165	2.6
9	TF3CY	1,965	164	8.3
10	CN2AA	11,390	162	1.4

2013 SSB -- Highest Percentage of Busts (≥100 QSOs)
Position	Call	QSOs	% Busts
1	UA6YT	160	18.1
2	K4DRL	123	15.4
3	HJ4DLO	193	13.5
4	W0PPF	205	12.2
5	AJ2E	222	12.2
6	CO8AW	125	12.0
7	K8TS	152	11.8
8	KP4JED	110	11.8
9	N8NYI	245	11.0
10	K2JMY	410	11.0

2013 SSB -- Highest Percentage of Reverse Busts (≥100 QSOs)
Position	Call	QSOs	% Reverse Busts
1	YO2LFP	203	15.3
2	BD3QZ	106	15.1
3	YB6DE	187	15.0
4	WM5DX	102	10.8
5	JG3SVP	777	10.3
6	KA1C	195	10.3
7	NP2MR/4	296	10.1
8	CT5JZX/MM	204	8.8
9	CE3OVE	349	8.6
10	WA2FBN	118	8.5

2013 CW -- Most Busts
Position	Call	QSOs	Busts	% Busts
1	D4C	23,538	230	1.0
2	TM1A	9,146	211	2.3
3	ZW8T	2,506	180	7.2
4	3DA0ET	7,389	179	2.4
5	PI4CC	10,715	175	1.6
6	PJ2T	15,457	155	1.0
7	V25A	4,675	144	3.1
8	EF8U	6,097	142	2.3
9	4U1ITU	6,919	134	1.9
10	G3V	4,786	133	2.8

2013 CW -- Most Reverse Busts
Position	Call	QSOs	Reverse Busts	% Reverse Busts
1	ES9C	16,168	390	2.4
2	HA30S	10,233	361	3.5
3	JS3CTQ	3,628	354	9.8
4	DF0HQ	11,570	324	2.8
5	DR1A	13,086	272	2.1
6	ED6A	15,019	264	1.8
7	W2FU	10,865	255	2.3
8	P40L	13,180	243	1.8
9	HF9Q	5,885	216	3.7
10	K3LR	12,302	203	1.7

2013 CW -- Highest Percentage of Busts (≥100 QSOs)
Position	Call	QSOs	% Busts
1	AD7XG	207	21.7
2	EA2GC	228	21.5
3	I4MNY	180	21.1
4	JA1LJF	137	20.4
5	W1TW	119	18.5
6	YO7LYM	206	18.0
7	K2SE	400	16.2
8	UR4LIN	260	16.2
9	EA3BES	180	15.6
10	OK1WMJ	214	15.0

2013 CW -- Highest Percentage of Reverse Busts (≥100 QSOs)
Position	Call	QSOs	% Reverse Busts
1	SB0A	541	14.6
2	US0HB	107	14.0
3	PI800GTB	108	10.2
4	JS3CTQ	3,628	9.8
5	JH8DBI	254	9.4
6	YO4CAH	967	9.3
7	XE2ST	525	9.1
8	PD7DB	100	9.0
9	YO7LYM	206	8.7
10	EE3C	284	8.5

Replacing a Linux Software RAID1 Bootable Disk

I recently started to receive SMART e-mails indicating that one of the two drives in a bootable Linux software RAID1 array on my main desktop machine was likely to fail before too long. An Internet search produced several pages that included what were purported to be complete instructions on how to replace the failing drive. However, all seemed to be missing one or more steps, so I thought that I had better document exactly the steps that I followed to perform the replacement.

Note that my system has two md devices, each using two disks, and each disk  has two partitions (see step 2 below).

1. Identify the device

The important part of one of the SMART e-mails looked like this:

This message was generated by the smartd daemon running on:

   host name:  homebrew
   DNS domain: [Empty]

The following warning/error was logged by the smartd daemon:

Device: /dev/sda [SAT], 1024 Currently unreadable (pending) sectors

Device info:
ST2000DM001-1CH164, S/N:Z340BANG, WWN:5-000c50-064a4855d, FW:CC27, 2.00 TB

For details see host's SYSLOG.

You can also use the smartctl utility for further investigation.
The original message about this issue was sent at Sun Mar 19 13:12:58 2017 MDT
Another message will be sent in 24 hours if the problem persists. 

It's a good idea to check that the identification of the device in /dev (in this case, /dev/sda) matches the information in the e-mail. This can be done with:
  ls -al /dev/disk/by-id

In this case, the failing device is /dev/sda, so that is the drive that will be replaced.

2. Determine which md devices use the failing drive

Execute:
  cat /proc/mdstat

$ cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda2[3] sdb2[2]
      1937755968 blocks super 1.2 [2/2] [UU]
     
md0 : active raid1 sda1[3] sdb1[2]
      15615872 blocks super 1.2 [2/2] [UU]
     
unused devices: <none>

So the failing device (sda) is in use by md0 (which uses partition sda1) and md1 (which uses partition sda2).

3. Mark the drive as failed in the md devices

Execute:
  mdadm --manage /dev/md0 --fail /dev/sda1
  mdadm --manage /dev/md0 --fail /dev/sda1

4. Mark the drive as removed in the md devices

Execute:
  mdadm --manage /dev/md0 --remove /dev/sda1
  mdadm --manage /dev/md0 --remove /dev/sda1

5. Check the status

Execute:
  cat /proc/mdstat
which should show that md0 and md1 are both no longer using /dev/sda.

6. Shut down

Execute:
  poweroff
or power off the system using a graphical desktop.

7. Replace the failing drive

Open the case; identify the failing drive using the identification information in the SMART e-mail; remove the drive; replace it with a new drive of at least the same size.

Note that drives that are nominally the same size may in fact have different usable amounts of space. When I created my RAID1 array, the instructions contained no warning about this (I don't know whether this omission is still true). Consequently, I partitioned the disks to use the entirety of the disks. This is a mistake, as I found: none of my spare drives were as large as the removed drive, even though they were all nominally the same size. Eventually, I ordered a new drive that was nominally 50% larger than the one I had removed (3TB instead of 2B), as all my spare drives of the same nominal size were all in fact too small. Using such a large replacement drive wastes a lot of space, but does ensure that the new drive will not fail at step 10 below.

8. Power up the computer

9. Determine the new mapping between device identifiers and device letters

Execute:
  cat /proc/mdstat
and note the sd<x> identifier for the good drive that is currently the (sole) drive in the arrays. We do this now because the sd<x> identifiers of the drives might have changed across the reboot (and in fact on my system, they did change).

I will call this drive sd<good>.

Execute:
  ls -al /dev/disk/by-id 
and write down the /dev/sd<x> identifier of the new drive. I will call this drive sd<new>.

10. Copy the partition table from /dev/sd<good> to /dev/sd<new>

Execute:
  sfdisk -d /dev/sd<good> | sfdisk /dev/sd<new>
If the drive sd<new> is not at least as large as the disk that it replaced, this command will fail. There are ways to work around a failure at this point, but none of them is pleasant, and, frankly, the best thing to do is to do as I did and obtain (quickly) a drive that is known to be large enough.

11. Check that things look correct

Execute:
  fdisk -l
This lists the partition information for all the drives on the system. Check that the partition tables for /dev/sd<good> and /dev/sd<new> are identical.

12. Add the new drive to the RAID arrays

Execute:
  mdadm --manage /dev/md0 --add /dev/sd<new>1
  mdadm --manage /dev/md1 --add /dev/sd<new>2

13. Check that things look correct

Execute:
  cat /proc/mdstat
At this point, one of the RAID drives will be resilvering; the other will either already be completed (if the second one is resilvering) or marked as DELAYED, which means that it waiting for the first RAID drive to complete its resilver.

14. Make the new drive bootable 

Execute:
  dd bs=512 count=1 if=/dev/sd<new> 2> /dev/null | strings

This should produce no output, as /dev/sd<new> is a clean disk.

Now execute:
  grub-install /dev/sd<new>
to install the boot loader on the new disk.

15. Check that the new drive is bootable

Execute:
  dd bs=512 count=1 if=/dev/sd<new> 2> /dev/null | strings

This should now produce output that includes one or more references to GRUB, such as:

   ZRr=
`|f
\|f1
GRUB
Geom
Hard Disk
Read
 Error

At this point the replacement is complete and the md devices are functional. It usually takes a few hours for the resilvering to complete (i.e., to make the two drives in the RAID arrays identical), but this happens behind the scenes, and the system is fully usable in the meantime.

15a. Actually boot from the new drive

If you are paranoid (as I am), it is a good idea, after resilvering has completed, to reboot, changing the BIOS settings temporarily so that the system is forced to boot from the new disk. Once this has been confirmed, the BIOS settings can be returned to the original configuration.