Missing GPS

This page summarizes notes on filling in missing GPS fixes from submitted
D files from WHOI ARGO floats. This comprises floats with an entry in Pelle's file
"/data1/argo/metadata/float_sum.mat" AND which hold D files in our GDAC mirror
as of October 19,2020.

float_sum.mat held 1601 entries. Of those, 1247 held D files in our GDAC mirror,
with a total of 172318 D files.

From here on, I saved information of my searches in the structure "argoinfo_fb"
with one entry per float.

I defined five float "types", based on the float_sum.comm and float_sum.inst entries:

"ARGO float type 1": SOLO1 ORBCOMM : comm=ORBCOM
"ARGO float type 2": SOLO1 ARGOS : comm=ARGOS (also check that inst is NOT "MRV" or "ALT" which it is not)
"ARGO float type 3": SOLO1 IRIDIUM : comm=IRIDIUM inst(1:2) = "SS"
"ARGO float type 4": S2A : comm=IRIDIUM inst(1:2) = "MR"
"ARGO float type 5": ALTO : comm=IRIDIUM inst(1:2) = "AL"

I also defined 4 position types based on the first letter of the D*.nc variable "POSITIONING_SYSTEM":

"position type 1: "O" for "ORBCOMM"
"position type 2: "A" for "ARGOS
"position type 3: "G" for "GPS
"position type 4: "I" for "IRIDIUM"
"position type 0: neither

The "0" type was included in case there was another type I had not considered, but
none were found; neither was type 1, "ORBCOMM". So all POSITIONING_SYSTEMS were either
ARGOS, GPS, or IRIDIUM.

The first overview figures show the total number of missing positions (below left) as well
as the size of the largest contiguous gap (right); colors indicate the float type.
(164 floats missed positions for a total of 1073 D files.)

Looking for currently already interpolated positions (position QC flag = 8) indicated
many for ORBCOMM floats, also a few for S2A's, but none for ALTO floats:

Iridium positions (edited based on their own quality parameter) had been routinely
substituted for missing GPS fixes during SOLO1 Iridium float DMQC work, so I was surprised
to not see more SOLO2 show up in the above figure. Pelle reminded me that we did not
mark Iridium fix substitutes for missing GPS as "8", since they were not based on
interpolation.

Adding IR fixes (those with POSITIONING_SYSTEM = 'IRIDIUM ') has thinner lines
indicated a large number of them for SOLO1s (2321 D files had them), a few for
S2As (69 D's) and none for ALTOs.

Pelle had suggested that some of the remaining GPS gaps in SOLO1 IR floats could
relate to gaps to contact with the float. Checking one example, I compared the
available Iridium fixes, already combined into a file by Alex, with the position
record, which showed that the large GPS gaps for this float did indeed coincide
with a break in SBD transmissions.

Looking closer at how to add position information:

1) Can we squeeze more out of SOLO1 Iridium? (Yes.)

There is anecdotal evidence (by Frank) that perhaps some of the
earliest SOLO1s did not receive IR substitutions during DMQC. To find out,
I checked missing SOLO1 positions against the Iridium position files.
Note that for SOLO1, Alex already combined the IR fixes from individual
SBD messages into a single profile IR positions (".srf files"), considering
the IR position QC info for a weighted average (I believe). The IR QC info
is contained in the original email message, but not in the individual
SBD messages or their listing in /data1/solo1/data/surface/[WHOI #]/[WHOI #].sbd
files.

Working with the .srf files, that turned up an additional 92 fixes for 18 floats,
reducing the number of total missing fixes from 1073 to 981. The additions were
checked in plots of lat/lon time series, a map, and a "speed check" derived as
the the ratio of spatial separation and elapsed time between successive profiles.

In the figures below, the original gap counts were plotted in thin lines
to help visualize the improvement. The axis ranges are the same as in the
earlier gap summary figures.

2) Can we squeeze more out of SOLO2 Iridium? (Yes.)

The SOLO2 real-time processing already considers using IR positions if GPS
is unavailable. However, it seemed worthwhile to check the few S2A gaps
against available IR fixes. This identified six floats with a total of
12 missing positions. Of those, five could be filled by available IR/GPS data
from the files in /data1/solo2/data/surf/[WHOI #].sfc files. These files typically
list two GPS and one IR fix per profile:
G 11 2 1 2013/11/10 04:04 +25.25431 -91.58239 190 5 30 35 43 1.2
G 11 1 1 2013/11/10 04:22 +25.25389 -91.58333 50 5 35 38 44 1.1
I 12 0 2013/11/20 02:48 +25.04630 -91.66490 3.1 13 2.7 0.1

In most of the five successful "rescues", the first GPS fix was "bad" (99999), but
the second one appeared to be fine. In one case, only the Iridium fix was good.
I did some exploring to understand patterns, such as in the 6 QC numbers following
the GPS lat/lon data, but it was not obvious to me. Given the small number
of cases, I decided to not go further (e.g., trying to get QC definitions from
Alex and/or Breck, etc), but instead checked the substitute fix in figures
of lat/lon time series, a map, and with a speed check as for SOLO1 IR
substitutions earlier.

3) Related: positioning QC flag of 4

I do not know the extend to which our routines consider the QC flag of
profile positions, but it would seem that an existing position marked as
bad (4) may have the same result as a missing position.

QC=4 was found for 115 profiles from 100 floats - most floats had only one
occurance. Starting with late ARGOS floats launched by Nov 2007 and
continuing for SOLO1 IR and SOLO2, it was basically only cycle 0 was marked.
In all but one instance, lat/lon time series and speed checks determined to
"re-activate" the fix by setting QC=1.

4) Missing first position:

Interpolation also does not work if the gap starts with the first profile.
Pelle's meta_sum structure in (/shared/argo/metadata/meta_sum.mat includes a
launch position and date that could be substituted in these cases. Its on the
list to fix 12 floats with this concern. All of them have a launch position
in meta_sum.mat, with 7 having launch- and missing profile time within a day.
For 5 others, its 10 days (4 floats) or 4 days (1 float) .

5) Interpolation - over what gap size?

The previous plots included position gaps of up to 45 profiles. At what point
is a gap too large to interpolate over?

First: how many large gaps are there? Below is th distribution of gap sizes,
dominated by single-fix gaps. A zoom-in on the right shows the number of larger
gaps.

One "hitch" to deal with before some stats: what if a position gap includes
the last profile - i.e., there is no end point for an interpolation.

This issue in fact affects the longest gap found here (a SOLO1 IR float at
60 degrees south, 3900758. After sending 14 profiles, the float "disappeared"
for 2.5 years (898 days) to send home 47 additional profiles. However, it
proved unrealistic to use the Iridium positions from those late transmissions.

But aside from this one and a second one with 16 fixes, the "no-ender" gaps are all
less than 4 positions long, with most only being the last fix.

I repeated the earlier distribution plots after excluding the "no-enders", shown below:

6) Finally: interpolating to fill GPS gaps

As described above, most gaps were fairly small, and interpolation posed no
issues. However, some of the remaining larger gaps cause at least some pause.
I assembled overview figures for the largest gaps to discuss during out Tuesday
meeting. In addition to the speed check described earlier, I included a distance
for each interpolated position for the closest pre- and post-gap good fix.
Following a conversation with Susan the following day, I added T and S plots to
avoid GPS interpolation across fronts or other hydrographic features with large
changes along the interpolated path. For this web page, the floats were ordered
by largest contiguous GPS gap size, and only the first 10 were included to allow
for reasonable upload speed. The full record of all floats with GPS gaps is found
here.

The interpolated record was accepted for nearly all floats, the most notable exception
being WMO# 3900758 with a large position gap across the Gulf Stream . Please see
this listing for details.

These efforts generated 772 interpolated positions. Together with
SOLO1 and SOLO2 IR substitutions (92 and 5, respectively) and using launch
positions (7), we reduced the number of missing positions from 1073 to 197.

Note that we also "re-activated" an additional 114 positions with QC=4 by reverting
them to QC=1 following time series- and speed checks.

My processing notes contain additional details on this job.

This page is maintained by Frank Bahr, and was last updated Dec 17,2020.