Obtaining Magnetometer Data

The Problem  

 

Until a few days ago, it was possible to obtain near-real-time ("NRT") magnetometer data files from directories housed below http://magweb.cr.usgs.gov/data/magnetometer/. Attempts to reach that location, and the subordinate files that held the actual NRT data, now return:

The requested service is temporarily unavailable. Please try later.

After waiting for several days for something more useful to be returned, I finally realised that temporarily here seems to be government-speak for permanently.

There is still a way to obtain NRT magnetometer data, but it's vastly more complicated than simply downloading data from a URL. I do not know why the sites that make the data available do not hide all the complexity from users and simply provide a URL as an interface, but the fact is that, at least for now, they do not, and we now have to go through a rather tortuous process to obtain data that was, until a few days ago, essentially trivial to obtain.

Obtaining the data in the absence of a URL

These are the steps that I had to take on debian stable; other Linux distributions and other operating systems will likely differ in details, but the basic idea should be the same.

First, one needs to install python, since the data are accessed via a python script. Debian, like most  Linux distributions installs python by default.
Next, install the packages python-numpy, python-scipy and python-flake8 from the distribution repositories.

Now, install obspy by following the instructions at: https://github.com/obspy/obspy/wiki/Installation-on-Linux-via-Apt-Repository. If using a different distribution or operating system, follow the pertinent instructions at https://github.com/obspy/obspy/wiki.

Now that python and the necessary libraries are available, we need to download the actual software package that does the work; we get this from: https://github.com/usgs/geomag-algorithms.

I note that the installation instructions on the site read, in their entirety:
First time install. Walk through dependencies and other considerations.

If one looks inside the downloaded package, there is no sign of the standard INSTALL file. Neither is there a README or README.FIRST file. There is a README.md file, but its contents merely repeat the above "instructions" from the site. So I simply unpacked the package in a reasonable location and used it as-is. I am sure that that is not what one is supposed to do, but in the absence of explicit, detailed installation instructions, it will have to do.

After unpacking that software somewhere suitable, we have access to the script we need, which is geomag.py in the .../bin directory.

The documentation is... well, the politest word I can think of is "sparse". As unfortunately seems to be so often the case when trying to use government data, it is hard to escape the conclusion that they don't really want it to be used. They certainly don't seem to go to any effort to explain the details of the package in a way that is useful to users. So we're pretty much on our own. 

The command geomag.py -h offers a lot of output, most of it incomprehensible without further documentation. However, the following command (typed on one line) produces output in a format very similar to that stored at the URL that in the past provided one-minute Boulder magnetometer data for a particular day:

geomag.py --inchannels H D Z F --observatory BOU --starttime 2016-01-23T00:00:00Z --endtime 2016-01-23T23:59:00Z --input-edge cwbpub.cr.usgs.gov --output-iaga-file output-file-name

From this point one can process the file output-file-name appropriately, in a way similar to the manner in which the old URL-based data could be processed. For example, I have a widget on my desktop that displays the last couple of hours of data from the Boulder magnetometer, using the above method to obtain data every five minutes: