We continue to rationalise the data by removing fields that are unused or provide no useful information, and marking invalid values.
Stage 16: remove records with invalid values of Rad_Re
The documentation for Rad_Re (for both the BDD instrument and the CXD instrument) is:
| Column | Variable name | type | Dim. | description |
| 4 | Rad_Re | double | 1 | (radius of satellite)/Rearth |
By definition, any value less than unity for this field must be an error (actually, any value less than about four is an error, but it turns out that are no records with values between one and four). We therefore remove records with invalid values for this field:
for file in ns*
do
awk '$4 > 1 {print $0}' $file > ../gps-stage-16/$file
done
At this point, the number of records in the dataset for each satellite is:
| Satellite | Stage 16 Records |
|---|---|
| ns41 | 1,990,340 |
| ns48 | 1,105,300 |
| ns53 | 1,331,017 |
| ns54 | 1,938,603 |
| ns55 | 1,054,569 |
| ns56 | 1,680,369 |
| ns57 | 1,082,028 |
| ns58 | 1,174,629 |
| ns59 | 1,515,843 |
| ns60 | 1,495,007 |
| ns61 | 1,469,803 |
| ns62 | 774,976 |
| ns63 | 651,078 |
| ns64 | 343,643 |
| ns65 | 480,017 |
| ns66 | 446,139 |
| ns67 | 327,174 |
| ns68 | 304,964 |
| ns69 | 262,021 |
| ns70 | 110,260 |
| ns71 | 220,694 |
| ns72 | 181,519 |
| ns73 | 145,292 |
Stage 17: remove proton_flux_fit from the CXD data
The documentation for the CXD field proton_flux_fit is:
| Column | Variable name | type | Dim. | description |
| 97-127 | proton_flux_fit | double | 31 | intended to be proton flux at 31 energies, not filled currently |
As the documentation indicates that this field is not filled with useful data, we remove it:
for file in ns[567]*
do
awk \ '{$97=$98=$99=$100=$101=$102=$103=$104=$105=$106=$107=$108=$109=$110=$111=$112=$113=$114=$115=$116=$117=$118=$119=$120=$121=$122=$123=$124=$125=$126=$127=""; print $0}' \
$file | tr -s " " | sed 's/ $//' > ../gps-stage-17/$file
done
The data table for the CXD satellites (ns53 to ns73) now looks like this:
| Column | Variable name | type | Dim. | description |
| 1 | decimal_day | double | 1 | GPS time, a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years. |
| 2 | Geographic_Latitude | double | 1 | Latitude of satellite (deg) |
| 3 | Geographic_Longitude | double | 1 | Longitude of satellite (deg) |
| 4 | Rad_Re | double | 1 | (radius of satellite)/Rearth |
| 5-15 | rate_electron_measured | double | 11 | Measured rate (Hz) in each of the 11 CXD electron channels |
| 16-20 | rate_proton_measured | double | 5 | Measured rate (Hz) in each of the 5 CXD proton channels (P1-P5) |
| 21 | LEP_thresh | double | 1 | LEP threshold in E1 channels (0 means low, 1 means high) |
| 22 | collection_interval | int | 1 | dosimeter collection period (seconds) |
| 23 | year | int | 1 | year (e.g. 2015) |
| 24 | decimal_year | double | 1 | decimal year = year + (decimal_day-1.0)/(days in year) |
| 25 | SVN_number | int | 1 | SVN number of satellite |
| 26 | b_coord_radius | double | 1 | radius from earth's dipole axis (earth radii) |
| 27 | b_coord_height | double | 1 | height above the earth's dipole equatorial plane (earth radii) |
| 28 | magnetic_longitude | double | 1 | Magnetic longitude (degrees) |
| 29 | L_shell | double | 1 | L shell: McIlwain calculation according to model with T89 External Field, IGRF Internal Field. |
| 30 | L_LGM_TS04IGRF | double | 1 | LanlGeoMag L-shell McIlwain calculation, TS04 External Field, IGRF Internal Field. |
| 31 | L_LGM_OP77IGRF | double | 1 | LanlGeoMag L-shell McIlwain calculation, OP77 External Field, IGRF Internal Field (not currently filled) |
| 32 | L_LGM_T89CDIP | double | 1 | LanlGeoMag L-shell McIlwain calculation, T89 External Field, Centered Dipole Internal Field |
| 33 | bfield_ratio | double | 1 | Bsatellite/Bequator |
| 34 | local_time | double | 1 | magnetic local time (0-24 hours) |
| 35 | utc_lgm | double | 1 | UTC (0-24 hours) |
| 36 | b_sattelite | double | 1 | B field at satellite (gauss) |
| 37 | b_equator | double | 1 | B field at equator (on this field line I think) (gauss) |
| 38-48 | electron_background | double | 11 | estimated background in electron channels E1-E11 (Hz) |
| 49-53 | proton_background | double | 5 | estimated background in proton channels P1-P5 (Hz) |
| 54 | proton_activity | int | 1 | =1 if there is significant proton activity |
| 55 | proton_temperature_fit | double | 1 | characteristic momentum -- R0 in the expression given above (MeV/c) |
| 56 | proton_density_fit | double | 1 | N0 parameter in fit to proton flux ((protons/(cm2 sec sr MeV)) |
| 57 | electron_temperature_fit | double | 1 | electron temperature from a one Maxwellian fit (MeV) |
| 58 | electron_density_fit | double | 1 | electron number density from a one Maxwellian fit (cm-3) |
| 59-69 | model_counts_electron_fit_pf | double | 11 | E1-E11 rates due to proton background based on proton flux fit -- currently not filled (all -1's) |
| 70-74 | model_counts_proton_fit_pf | double | 5 | P1-P5 rate from proton fit (using proton_temperature_fit, proton_density_fit) |
| 75-85 | model_counts_electron_fit | double | 11 | E1-E11 rates from the 9-parameter electron flux model |
| 86-90 | model_counts_proton_fit | double | 5 | P1-P5 rates from electron background -- currently not filled (all -1's) |
| 91-96 | proton_integrated_flux_fit | double | 6 | integral of proton flux (based on fit) above 10, 15.85, 25.11, 30, 40, 79.43 MeV (proton kinetic energy) |
| 97-126 | integral_flux_instrument | double | 30 | (based on 9 parameter fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec) |
| 127-156 | integral_flux_energy | double | 30 | energies for the integral of integral_flux_instrument (MeV) |
| 157-171 | electron_diff_flux_energy | double | 15 | energies for the fluxes in electron_diff_flux_energy (MeV) |
| 172-186 | electron_diff_flux | double | 15 | (based on 9 parameter fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec)) |
| 187-195 | Efitpars | double | 9 | fit parameters for 9 parameter electron fit |
Stage 18: remove model_counts_proton_fit from the CXD data
The documentation for the CXD field model_counts_proton_fit is:
| Column | Variable name | type | Dim. | description |
| 86-90 | model_counts_proton_fit | double | 5 | P1-P5 rates from electron background -- currently not filled (all -1's) |
As the documentation indicates that this field is not filled with useful data, we remove it:
for file in ns[567]*
do
awk '{$86=$87=$88=$89=$90=""; print $0}' $file \
| tr -s " " | sed 's/ $//' > ../gps-stage-18/$file
done
The data table for the CXD satellites (ns53 to ns73) now looks like this:
| Column | Variable name | type | Dim. | description |
| 1 | decimal_day | double | 1 | GPS time, a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years. |
| 2 | Geographic_Latitude | double | 1 | Latitude of satellite (deg) |
| 3 | Geographic_Longitude | double | 1 | Longitude of satellite (deg) |
| 4 | Rad_Re | double | 1 | (radius of satellite)/Rearth |
| 5-15 | rate_electron_measured | double | 11 | Measured rate (Hz) in each of the 11 CXD electron channels |
| 16-20 | rate_proton_measured | double | 5 | Measured rate (Hz) in each of the 5 CXD proton channels (P1-P5) |
| 21 | LEP_thresh | double | 1 | LEP threshold in E1 channels (0 means low, 1 means high) |
| 22 | collection_interval | int | 1 | dosimeter collection period (seconds) |
| 23 | year | int | 1 | year (e.g. 2015) |
| 24 | decimal_year | double | 1 | decimal year = year + (decimal_day-1.0)/(days in year) |
| 25 | SVN_number | int | 1 | SVN number of satellite |
| 26 | b_coord_radius | double | 1 | radius from earth's dipole axis (earth radii) |
| 27 | b_coord_height | double | 1 | height above the earth's dipole equatorial plane (earth radii) |
| 28 | magnetic_longitude | double | 1 | Magnetic longitude (degrees) |
| 29 | L_shell | double | 1 | L shell: McIlwain calculation according to model with T89 External Field, IGRF Internal Field. |
| 30 | L_LGM_TS04IGRF | double | 1 | LanlGeoMag L-shell McIlwain calculation, TS04 External Field, IGRF Internal Field. |
| 31 | L_LGM_OP77IGRF | double | 1 | LanlGeoMag L-shell McIlwain calculation, OP77 External Field, IGRF Internal Field (not currently filled) |
| 32 | L_LGM_T89CDIP | double | 1 | LanlGeoMag L-shell McIlwain calculation, T89 External Field, Centered Dipole Internal Field |
| 33 | bfield_ratio | double | 1 | Bsatellite/Bequator |
| 34 | local_time | double | 1 | magnetic local time (0-24 hours) |
| 35 | utc_lgm | double | 1 | UTC (0-24 hours) |
| 36 | b_sattelite | double | 1 | B field at satellite (gauss) |
| 37 | b_equator | double | 1 | B field at equator (on this field line I think) (gauss) |
| 38-48 | electron_background | double | 11 | estimated background in electron channels E1-E11 (Hz) |
| 49-53 | proton_background | double | 5 | estimated background in proton channels P1-P5 (Hz) |
| 54 | proton_activity | int | 1 | =1 if there is significant proton activity |
| 55 | proton_temperature_fit | double | 1 | characteristic momentum -- R0 in the expression given above (MeV/c) |
| 56 | proton_density_fit | double | 1 | N0 parameter in fit to proton flux ((protons/(cm2 sec sr MeV)) |
| 57 | electron_temperature_fit | double | 1 | electron temperature from a one Maxwellian fit (MeV) |
| 58 | electron_density_fit | double | 1 | electron number density from a one Maxwellian fit (cm-3) |
| 59-69 | model_counts_electron_fit_pf | double | 11 | E1-E11 rates due to proton background based on proton flux fit -- currently not filled (all -1's) |
| 70-74 | model_counts_proton_fit_pf | double | 5 | P1-P5 rate from proton fit (using proton_temperature_fit, proton_density_fit) |
| 75-85 | model_counts_electron_fit | double | 11 | E1-E11 rates from the 9-parameter electron flux model |
| 86-91 | proton_integrated_flux_fit | double | 6 | integral of proton flux (based on fit) above 10, 15.85, 25.11, 30, 40, 79.43 MeV (proton kinetic energy) |
| 92-121 | integral_flux_instrument | double | 30 | (based on 9 parameter fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec) |
| 122-151 | integral_flux_energy | double | 30 | energies for the integral of integral_flux_instrument (MeV) |
| 152-166 | electron_diff_flux_energy | double | 15 | energies for the fluxes in electron_diff_flux_energy (MeV) |
| 167-181 | electron_diff_flux | double | 15 | (based on 9 parameter fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec)) |
| 182-190 | Efitpars | double | 9 | fit parameters for 9 parameter electron fit |
Stage 19: remove model_counts_electron_fit_pf from the CXD data
The documentation for the CXD field model_counts_electron_fit_pf is:
| Column | Variable name | type | Dim. | description |
| 59-69 | model_counts_electron_fit_pf | double | 11 | E1-E11 rates due to proton background based on proton flux fit -- currently not filled (all -1's) |
As the documentation indicates that this field is not filled with useful data, we remove it:
for file in ns[567]*
do
awk '{$59=$60=$61=$62=$63=$64=$65=$66=$67=$68=$69=""; print $0}' \
$file | tr -s " " | sed 's/ $//' > ../gps-stage-19/$file
done
The data table for the CXD satellites (ns53 to ns73) now looks like this:
| Column | Variable name | type | Dim. | description |
| 1 | decimal_day | double | 1 | GPS time, a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years. |
| 2 | Geographic_Latitude | double | 1 | Latitude of satellite (deg) |
| 3 | Geographic_Longitude | double | 1 | Longitude of satellite (deg) |
| 4 | Rad_Re | double | 1 | (radius of satellite)/Rearth |
| 5-15 | rate_electron_measured | double | 11 | Measured rate (Hz) in each of the 11 CXD electron channels |
| 16-20 | rate_proton_measured | double | 5 | Measured rate (Hz) in each of the 5 CXD proton channels (P1-P5) |
| 21 | LEP_thresh | double | 1 | LEP threshold in E1 channels (0 means low, 1 means high) |
| 22 | collection_interval | int | 1 | dosimeter collection period (seconds) |
| 23 | year | int | 1 | year (e.g. 2015) |
| 24 | decimal_year | double | 1 | decimal year = year + (decimal_day-1.0)/(days in year) |
| 25 | SVN_number | int | 1 | SVN number of satellite |
| 26 | b_coord_radius | double | 1 | radius from earth's dipole axis (earth radii) |
| 27 | b_coord_height | double | 1 | height above the earth's dipole equatorial plane (earth radii) |
| 28 | magnetic_longitude | double | 1 | Magnetic longitude (degrees) |
| 29 | L_shell | double | 1 | L shell: McIlwain calculation according to model with T89 External Field, IGRF Internal Field. |
| 30 | L_LGM_TS04IGRF | double | 1 | LanlGeoMag L-shell McIlwain calculation, TS04 External Field, IGRF Internal Field. |
| 31 | L_LGM_OP77IGRF | double | 1 | LanlGeoMag L-shell McIlwain calculation, OP77 External Field, IGRF Internal Field (not currently filled) |
| 32 | L_LGM_T89CDIP | double | 1 | LanlGeoMag L-shell McIlwain calculation, T89 External Field, Centered Dipole Internal Field |
| 33 | bfield_ratio | double | 1 | Bsatellite/Bequator |
| 34 | local_time | double | 1 | magnetic local time (0-24 hours) |
| 35 | utc_lgm | double | 1 | UTC (0-24 hours) |
| 36 | b_sattelite | double | 1 | B field at satellite (gauss) |
| 37 | b_equator | double | 1 | B field at equator (on this field line I think) (gauss) |
| 38-48 | electron_background | double | 11 | estimated background in electron channels E1-E11 (Hz) |
| 49-53 | proton_background | double | 5 | estimated background in proton channels P1-P5 (Hz) |
| 54 | proton_activity | int | 1 | =1 if there is significant proton activity |
| 55 | proton_temperature_fit | double | 1 | characteristic momentum -- R0 in the expression given above (MeV/c) |
| 56 | proton_density_fit | double | 1 | N0 parameter in fit to proton flux ((protons/(cm2 sec sr MeV)) |
| 57 | electron_temperature_fit | double | 1 | electron temperature from a one Maxwellian fit (MeV) |
| 58 | electron_density_fit | double | 1 | electron number density from a one Maxwellian fit (cm-3) |
| 59-63 | model_counts_proton_fit_pf | double | 5 | P1-P5 rate from proton fit (using proton_temperature_fit, proton_density_fit) |
| 64-74 | model_counts_electron_fit | double | 11 | E1-E11 rates from the 9-parameter electron flux model |
| 75-80 | proton_integrated_flux_fit | double | 6 | integral of proton flux (based on fit) above 10, 15.85, 25.11, 30, 40, 79.43 MeV (proton kinetic energy) |
| 81-110 | integral_flux_instrument | double | 30 | (based on 9 parameter fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec) |
| 111-140 | integral_flux_energy | double | 30 | energies for the integral of integral_flux_instrument (MeV) |
| 141-155 | electron_diff_flux_energy | double | 15 | energies for the fluxes in electron_diff_flux_energy (MeV) |
| 156-170 | electron_diff_flux | double | 15 | (based on 9 parameter fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec)) |
| 171-179 | Efitpars | double | 9 | fit parameters for 9 parameter electron fit |
Stage 20: mark invalid values of L_shell for the BDD experiment
I have been informed in a private communication that, for ns41 and ns48, all calculated values of L_shell greater than 25 should be set to 25, and this value is to be interpreted as "too large to calculate". Therefore, for these two satellites, we set all values of L_shell that currently have the value 25 or greater, to "NA".
for file in ns4*
do
awk '$28 >= 25 {$28="NA"}; {print $0}' $file > ../gps-stage-20/$file
done
Time to checkpoint the data again. The checkpointed data file has the MD5 checksum 830dfdcc200fd4b2c0ba41be9e6edf0e.
The data table for ns41 and ns48 now looks like this:
| Column | Variable name | type | Dim. | Description |
| 1 | decimal_day | double | 1 | GPS time -- a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years |
| 2 | Geographic_Latitude | double | 1 | Latitude of satellite (deg) |
| 3 | Geographic_Longitude | double | 1 | Longitude of satellite (deg) |
| 4 | Rad_Re | double | 1 | (radius of satellite)/Rearth |
| 5-12 | rate_electron_measured | double | 8 | Measured rate (Hz) in each of the 8 BDD electron channels (E1-E8) |
| 13-20 | rate_proton_measured | double | 8 | Measured rate (Hz) in each of the 8 BDD proton channels (P1-P8) |
| 21 | collection_interval | int | 1 | dosimeter collection period (seconds) |
| 22 | year | int | 1 | year (e.g. 2015) |
| 23 | decimal_year | double | 1 | decimal year = year + (decimal_day-1.0)/(days in year) |
| 24 | svn_number | int | 1 | SVN number of satellite |
| 25 | b_coord_radius | double | 1 | radius from earth's dipole axis (earth radii) |
| 26 | b_coord_height | double | 1 | height above the earth's dipole equatorial plane (earth radii) |
| 27 | magnetic_longitude | double | 1 | Magnetic longitude (degrees) |
| 28 | L_shell | double | 1 | L_shell (earth radii) -- I do not clearly understand the origin of the calculation, but it seems to be a dipole field/T-89 |
| 29 | bfield_ratio | double | 1 | Bsatellite/Bequator |
| 30 | local_time | double | 1 | magnetic local time (0-24 hours) |
| 31 | b_sattelite | double | 1 | B field at satellite (gauss) |
| 32 | b_equator | double | 1 | B field at equator (on this field line I think) (gauss) |
| 33-40 | electron_background | double | 8 | estimated background in electron channels E1-E8 (Hz) |
| 41-48 | proton_background | double | 8 | estimated background in proton channels P1-P8 (Hz) |
| 49 | proton_activity | int | 1 | =1 if there is significant proton activity |
| 50 | electron_temperature | double | 1 | electron temperature from a one Maxwellian fit (MeV) |
| 51 | electron_density_fit | double | 1 | electron number density from a one Maxwellian fit (cm-3) |
| 52-59 | model_counts_electron_fit | double | 8 | E1-E8 rates from the 2-parameter Maxwellian fit to the electron data |
| 60-67 | dtc_counts_electron | double | 8 | Dead time corrected electron rates (from data, not fit) |
| 68-97 | integral_flux_instrument | double | 30 | (based on 2 parameter Maxwellian fit) integral of electron flux above integral_flux_energy[i] particles/(cm2sec) |
| 98-127 | integral_flux_energy | double | 30 | energies for the integral of integral_flux_instrument (MeV) |
| 128-142 | electron_diff_flux_energy | double | 15 | energies for the fluxes in electron_diff_flux_energy (MeV) |
| 143-157 | electron_diff_flux | double | 15 | (based on 2 parameter Maxwellian fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec)) |
And for the remaining satellites:
| Column | Variable name | type | Dim. | description |
| 1 | decimal_day | double | 1 | GPS time, a number from 1 (1-Jan 00:00) to 366 (31-Dec 24:00) or 367 in leap years. |
| 2 | Geographic_Latitude | double | 1 | Latitude of satellite (deg) |
| 3 | Geographic_Longitude | double | 1 | Longitude of satellite (deg) |
| 4 | Rad_Re | double | 1 | (radius of satellite)/Rearth |
| 5-15 | rate_electron_measured | double | 11 | Measured rate (Hz) in each of the 11 CXD electron channels |
| 16-20 | rate_proton_measured | double | 5 | Measured rate (Hz) in each of the 5 CXD proton channels (P1-P5) |
| 21 | LEP_thresh | double | 1 | LEP threshold in E1 channels (0 means low, 1 means high) |
| 22 | collection_interval | int | 1 | dosimeter collection period (seconds) |
| 23 | year | int | 1 | year (e.g. 2015) |
| 24 | decimal_year | double | 1 | decimal year = year + (decimal_day-1.0)/(days in year) |
| 25 | SVN_number | int | 1 | SVN number of satellite |
| 26 | b_coord_radius | double | 1 | radius from earth's dipole axis (earth radii) |
| 27 | b_coord_height | double | 1 | height above the earth's dipole equatorial plane (earth radii) |
| 28 | magnetic_longitude | double | 1 | Magnetic longitude (degrees) |
| 29 | L_shell | double | 1 | L shell: McIlwain calculation according to model with T89 External Field, IGRF Internal Field. |
| 30 | L_LGM_TS04IGRF | double | 1 | LanlGeoMag L-shell McIlwain calculation, TS04 External Field, IGRF Internal Field. |
| 31 | L_LGM_OP77IGRF | double | 1 | LanlGeoMag L-shell McIlwain calculation, OP77 External Field, IGRF Internal Field (not currently filled) |
| 32 | L_LGM_T89CDIP | double | 1 | LanlGeoMag L-shell McIlwain calculation, T89 External Field, Centered Dipole Internal Field |
| 33 | bfield_ratio | double | 1 | Bsatellite/Bequator |
| 34 | local_time | double | 1 | magnetic local time (0-24 hours) |
| 35 | utc_lgm | double | 1 | UTC (0-24 hours) |
| 36 | b_sattelite | double | 1 | B field at satellite (gauss) |
| 37 | b_equator | double | 1 | B field at equator (on this field line I think) (gauss) |
| 38-48 | electron_background | double | 11 | estimated background in electron channels E1-E11 (Hz) |
| 49-53 | proton_background | double | 5 | estimated background in proton channels P1-P5 (Hz) |
| 54 | proton_activity | int | 1 | =1 if there is significant proton activity |
| 55 | proton_temperature_fit | double | 1 | characteristic momentum -- R0 in the expression given above (MeV/c) |
| 56 | proton_density_fit | double | 1 | N0 parameter in fit to proton flux ((protons/(cm2 sec sr MeV)) |
| 57 | electron_temperature_fit | double | 1 | electron temperature from a one Maxwellian fit (MeV) |
| 58 | electron_density_fit | double | 1 | electron number density from a one Maxwellian fit (cm-3) |
| 59-63 | model_counts_proton_fit_pf | double | 5 | P1-P5 rate from proton fit (using proton_temperature_fit, proton_density_fit) |
| 64-74 | model_counts_electron_fit | double | 11 | E1-E11 rates from the 9-parameter electron flux model |
| 75-80 | proton_integrated_flux_fit | double | 6 | integral of proton flux (based on fit) above 10, 15.85, 25.11, 30, 40, 79.43 MeV (proton kinetic energy) |
| 81-110 | integral_flux_instrument | double | 30 | (based on 9 parameter fit) integral of electron flux above integral_flux_energy[i] particles/(cm2 sec) |
| 111-140 | integral_flux_energy | double | 30 | energies for the integral of integral_flux_instrument (MeV) |
| 141-155 | electron_diff_flux_energy | double | 15 | energies for the fluxes in electron_diff_flux_energy (MeV) |
| 156-170 | electron_diff_flux | double | 15 | (based on 9 parameter fit) electron flux at energies electron_diff_flux[i] (particle/(cm2 sr MeV sec)) |
| 171-179 | Efitpars | double | 9 | fit parameters for 9 parameter electron fit |
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.