8,137
edits
Changes
→Trail Running and GPS
{{DISPLAYTITLE:GPS Accuracy of Garmin, Polar, and other Running Watches}}<div style="float:right;">__TOC__</div>
I evaluated the real -world accuracy of GPS watches while running over 612,000 miles/919,600Km 000Km and recording over 2550,000 data points as part of my evaluation of the [[Best Running Watch]]es. Under good conditions most of the watches are remarkably good, but when things get a little tough the differences become more apparent. However, '''none of the watches have GPS accuracy that is good enough to be used for displaying your current pace'''. For current paceAs a result, I've added the only viable option is to use a test results for various [[Footpod]]s as they can be far more accurate than GPS, and my [[Best Running Watch| review of running watches]] lists those that but more importantly they tend to have far less moment-to-moment variation so they can give a far better display of your current pace from a Footpod while still using GPS for your course. (Note that my accuracy is tests focus on the ability to measure distance, not the moment in time position, though the two are obviously related.)
[[File:GPS Accuracy.png|none|thumb|800px|An infographic of the accuracy of the GPS running watches. The top right corner represents the most accurate watches. (This graphic uses ISO 5725 terminology.)]]
The table below is a simplified summary of the results, where a '10' would be a perfect device. (For an explanation of the ISO 5725 terms 'trueness', 'precision' and 'accuracy', see below.)
The table below shows summary data for each device. The count field is how many measurements I have for that combination of condition and device, with each measurement being a quarter mile distance. I generally aim for over 1,000 data points to even out the effects of weather, satellite position and other factors. The Trueness is the absolute of the mean, though nearly all watches tend to read short. The standard deviation is provided based on the variance from the mean and the variance from the known true value. The average pace error is shown to give a sense of how much error you're likely to see in the display of current pace. This is an average error not a worst case. The data shown below is a summary the accuracy based on all the sections. If you'd like more detailed information, I've split off the [[Detailed Statistics for GPS Running Watches]] for the results under different conditions.
{{:GPS Accuracy-statistics}}
The "Accuracy (Combined)" column has an indication of statistical significance compared with the most accurate entry. The key to this indication is: † p<0.05, * p< 0.01, ** p< 0.001, *** p< 0.0001, **** p< 0.00001, ***** p< 0.000001
==Progress of newer watches==
I expected GPS watches to improve with time, but the opposite appears to be happening. With the Garmin devices especially, you can see that the older watches generally do far better than the newer ones. I suspect this is due to compromises to get better battery life and smaller packaging and the cost of GPS accuracy.
==Smartphone Accuracy==
There are various things you will need to do in order to get the level of accuracy I found with Smartphones. See [[Running With A Smartphone#Optimizing GPS Accuracy| Optimizing Smartphone GPS Accuracy]] for details.
==Interpretation and Conclusions==
What do these statistics mean? This is my interpretation:
=Footpod Accuracy=
The accuracy of a Footpod is far higher than GPS, as well as more consistent and quicker to react to changes in pace. For any given run, the average pace error from the Footpod is only 7 seconds/mile (at a 9:00 min/mile pace) or 5 seconds/Km (at a 5:30 min/Km pace). In practical terms, I've found that I always have to use a Footpod to pace a marathon or for critical speedwork. For details of how the Footpod calibration was done, see [[GPS Testing Methodology]].
=Trail Running and GPS=
Trail running tends to be rather harder for a watch to measure accurately. There are far more twists and turns, and for a [[Footpod]] your footsteps tend to be uneven. I realized how bad the problem was when running some mountain bike trails and my GPS watch said I'd only been traveling at walking pace. This prompted me to survey and evaluate the accuracy of various devices on these mountain bike trails. The table below is preliminary data, but you'll notice how the results are dramatically worse than my usual GPS testing. The Polar V800, which does really well on my greenway tests has serious problems on trails, though it's still one of the better watches I've tested so far. The Suunto Spartan Trainer shows its strength more clearly on the mountain bike trails, coming in far ahead of other GPS watches. The [[Stryd]] footpod is vastly more accurate than GPS, and unlike GPS it could be calibrated to improve its accuracy even further.
{{:GPS Accuracy-TrailSummary}}
=Which Chipset? =
While the specific chipset used in a GPS watch will impact its accuracy, there are many other factors that come into play. The physical packaging of the chipset, the antenna used, the particular features that are implemented, and the software that interprets the raw data will influence the overall accuracy. It's important to note that the SiRF chipsets such as "SIRFstarIV" are not a single chipset, but rather an overall architecture with several specific chipsets bearing the same name.
|[[File:GPS MarketSt.jpg|none|thumb|x300px|Here's another example of running down Market Street in San Francisco, where you can see the errors that would add to the distance. ]]
|}
=GPS Accuracy and Weather=
GPS Accuracy is slightly better with clear skies than with cloud cover. The difference between completely clear and fully overcast is generally less than 0.1% and my testing includes a similar mix of cloud cover for each watch, so I ignore this difference. However, rain can degrade accuracy by 0.3-3.1%, with the better watches being impacted the least. Because it does not rain that frequently where I test, this has created some potential bias in my testing so I now ignore measurements taken during the rain. This has only made a slight difference to the results, but it ensures consistence.
=GPS Accuracy and Seasons=
I run in a wooded area with mostly deciduous trees, so the foliage varies by season. This foliage can have a noticeable impact on GPS accuracy, with better accuracy during the bare winter months than the rest of the year. This difference is mostly 0.1-1.5%, but in some cases can be as large as 2.5%. Because of this, my testing now ignores data from the winter months when the trees are bare. The short winter here in the south of the US means that the impact on the overall results are small, but like the weather impacts noted above, this does ensure greater consistency.
=GPS Accuracy and Pace=
[[File:AccuracyAndPace.jpg|none|thumb|500px| A plot of GPS precision against pace. The red line is the correlation.]]
There have been reports of GPS accuracy changing with pace, but as you can see from the graph above, my testing does not show this.
=GPS and GLONASS=
I have found that GPS plus GLONASS produces less accuracy than GPS alone, something that is a little counterintuitive. I have no definitive explanation for this, and I do have a working hypothesis. My thought is that enabling both GPS and GLONASS will increase the number of satellites above the horizon, and a modern chipset can have over 50 channels. This means the chipset will have access to far more satellites with both systems enabled. However, I don't believe that the chipset will use all the available satellites when calculating its position. In an urban, or wooded environment, the satellites nearest the horizon will have the weakest signal, and the satellites closest to directly overhead will have the strongest signal. If the chipset were to use only the strongest 5-6 signals, then it's likely to choose the satellites that are closest to being directly overhead. That means the satellites chosen are relatively close together, which is a poor geometry that reduces accuracy. (In GPS terms this is called Dilution of Precision, or DoP.) I've talked to a GPS specialist who tells me that they have seen this in GPS systems they've tested (though not necessarily consumer grade systems.) What this means in the real world is that if you're in an environment with a partial view of the sky due to tree cover for low buildings then GPS on its own is likely to provide better accuracy. If you're in an environment with a clear view of the sky from horizon to horizon, then it's less clear to me which system is likely to provide better accuracy, and I've not tested this in practice. Given that the theoretical accuracy of GLONASS is not quite as good as GPS I'm not sure that enabling both systems will improve matters. It's possible that GLONASS will do relatively better at extreme polar latitudes due to its different orbital patterns.
==Garmin 920XT and GLONASS==
The [[Garmin 920XT]] is significantly worse with GLONASS enabled.
{| class="wikitable"
!Device
!Accuracy
!Trueness
!Precision
!Repeatability
|-
|Garmin 920XT
|style="background-color: #FAE983;"|6.6
|style="background-color: #D2DE81;"|7.5
|style="background-color: #FED680;"|5.9
|style="background-color: #D1DD81;"|7.5
|-
|Garmin 920XT (GLONASS)
|style="background-color: #FEC77D;"|5.5
|style="background-color: #FAE983;"|6.6
|style="background-color: #FCA777;"|4.6
|style="background-color: #E1E282;"|7.2
|}
==Suunto Spartan Ultra and GLONASS==
The [[Suunto Spartan Ultra]] seems to do particularly poorly with GLONASS enabled.
{| class="wikitable"
!Device
!Accuracy
!Trueness
!Precision
!Repeatability
|-
|Suunto Spartan Ultra 1.6.14
|style="background-color: #E2E282;"|7.1
|style="background-color: #79C47C;"|9.5
|style="background-color: #FED881;"|6.0
|style="background-color: #D8DF81;"|7.4
|-
|Suunto Spartan Ultra 1.6.14.GLONASS
|style="background-color: #FCB179;"|4.9
|style="background-color: #D4DE81;"|7.5
|style="background-color: #F9756E;"|3.3
|style="background-color: #FDB57A;"|5.0
|}
==Garmin Epix and GLONASS==
The [[Garmin Epix]] has slightly better accuracy with WAAS than without it, and GLONASS didn't degrade the accuracy the way it does with other devices. My belief is that enabling WAAS effectively disables GLONASS, as WAAS is GPS specific (and only available in North America.) There is EGNOS Ground Segment is the equivalent of WAAS for GLONASS/GPS/Galileo in Europe.
{| class="wikitable"
|- valign="top"
!Device
!Accuracy
!Trueness
!Precision
!Repeatability
|-
|Garmin Epix with GLONASS+WAAS
|style="background-color: #FFE082;"|6.2
|style="background-color: #D6DF81;"|7.4
|style="background-color: #FDBE7C;"|5.3
|style="background-color: #D6DF81;"|7.4
|-
|Garmin Epix with WAAS
|style="background-color: #FFDF82;"|6.2
|style="background-color: #C8DB80;"|7.7
|style="background-color: #FDB77A;"|5.1
|style="background-color: #F7E883;"|6.7
|-
|Garmin Epix
|style="background-color: #FDC37D;"|5.4
|style="background-color: #F3E783;"|6.8
|style="background-color: #FB9C75;"|4.4
|style="background-color: #F2E783;"|6.8
|}
==Garmin Fenix 5X and GLONASS==
Continuing the theme of poor accuracy with GLONASS enabled, the [[Garmin Fenix 5X]] demonstrates even worse performance than its peers. The values shown below are rather dramatically worse with GLONASS enabled than without. My anecdotal observation is that sometimes the Fenix 5X does a little worse with GLONASS than normal, possibly in line with other Garmin devices, and sometimes it seems to just get lost and produce dramatically worse results.
{| class="wikitable sortable"
!Device
!Accuracy
!Trueness
!Precision
!Repeatability
|-
|Fenix 5X 4.30
|style="background-color: #FEC97E;"|5.6
|style="background-color: #EDE683;"|6.9
|style="background-color: #FCA377;"|4.5
|style="background-color: #F3E783;"|6.8
|-
|Fenix 5X 4.30 GLONASS
|style="background-color: #F97B6F;"|3.5
|style="background-color: #FFE082;"|6.2
|style="background-color: #F8696B;"|1.6
|style="background-color: #F97B6F;"|3.5
|}
Below you can see a visual representation of the problems. Many of the tracks are a little worse than normal, but you generally follow the path. A small subset of the tracks are dramatically worse, either showing and offset from the actual path, or sometimes it looks like the sampling frequency has dropped, suggesting that the watch is only periodically able to get a location fix. This latter phenomenon is rather surprising to me and goes against my hypothesis of why GLONASS has worse accuracy. I would expect there would be more satellites available with GLONASS enabled, which would result in the watch selecting the subset with the strongest signal that are more likely to have a narrow angle of separation, which would result in increased Dilution of the Precision. These tracks suggest that the Fenix 5X is unable to get any location fix for brief periods.
{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"
|- valign="top"
|[[File:BridgeFenix 5X 4.30 GLONASS.jpg|center|thumb|x300px| The GPS tracks from the Fenix 5X with GLONASS enabled. This diagram has tracks color coded with green indicating good accuracy through to red indicating poor accuracy, and the lap markers as blue dots.]]
|- valign="top"
|[[File:BridgeFenix 5X 4.30.jpg|none|thumb|x300px|Here's the tracks from testing with GLONASS disabled for comparison.]]
|}
=GPS Accuracy and Sampling Rate=
GPS watches default to recording a sample frequently enough that accuracy is not compromised. However, several devices offer the option of recording less frequently to improve battery life at the cost of accuracy. These devices actually turn off the GPS receiver, turning it on periodically for just long enough to get a fix. The images below are from the [[2014 Badwater 135]] using the [[Suunto Ambit2| Suunto Ambit2 R]] with recording set to one minute intervals. As you can see, accuracy suffers on curves, but is fine on the straights. For a course like Badwater, the one minute recording interval was fine as the course has few turns.
{| class="wikitable"
|- valign="top"
|[[File:GPS Sampling Curve.jpg|none|thumb|x300px|On a curve, the infrequent samples tend to 'cut the corners' and are quite inaccurate.]]
|[[File:GPS Sampling Straight.jpg|none|thumb|x300px|On the straight sections, the one minute sampling does not lose any accuracy.]]
|[[File:GPS Sampling Comparison.jpg|none|thumb|x300px|Here's a comparison of 1 minute sampling (red) with 1 second sampling (blue). On my GPS testing course the 1 minute sampling lost nearly 2 miles over a 16 mile run.]]
|}
=GPS Accuracy and Recording Rate (Smart/1-Second)=
While the GPS sampling rate mentioned above has a huge impact on GPS accuracy, the same isn't true for recording rate. These two ideas seem to get confused. GPS sampling rate allows a watch to turn off the GPS receiver for short periods to conserve battery life while sacrificing GPS accuracy. Some Garmin watches can be configured to either record every second, or only record when something happens, such a change in heart rate or change in direction, something they call "smart recording." With a smart recording in normal GPS mode, the GPS system is continually active, so there's no loss in accuracy. To verify this, I tested the [[Garmin Fenix 5X]] in both the smart recording mode I normally use, and one second recording mode for comparison. As you can see, the two modes are virtually identical, and the differences are most likely due to chance (p=0.72).
{| class="wikitable sortable"
!Device
!Accuracy
!Trueness
!Precision
!Repeatability
|-
|Fenix 5X 4.30 Smart Recording
|style="background-color: #FEC97E;"|5.6
|style="background-color: #EDE683;"|6.9
|style="background-color: #FCA377;"|4.5
|style="background-color: #F3E783;"|6.8
|-
|Fenix 5X 4.30 One Second Recording
|style="background-color: #FDBF7C;"|5.3
|style="background-color: #E5E382;"|7.1
|style="background-color: #FB9073;"|4.0
|style="background-color: #FBE983;"|6.6
|}
=Device Specific Notes=
For those interested in some of the details of how devices are configured for testing, here are some additional notes.
* Garmin devices are set to 'smart recording'. I did try an informal test with the 620 using 1-second recording, but it appeared to make no difference.
* For details of the calibration of the [[Footpod]] see [[GPS Testing Methodology]].
* The Fenix 2 was tested with and without WAAS support activated; WAAS helped slightly.
* The [[Garmin 920XT]] was tested with Watch Firmware 2.50, GPS Firmware 2.70 using smart recording.
=Garmin 620 Issues=
The Garmin 620 had some notorious problems with its GPS accuracy. The table below shows the changes with various firmware versions, culminating in the GPS-3.30 firmware that resolved the issues. I've including some testing I did without EPO data (NoEPO row below) and with a Footpod (+FP row below).
|[[File:Fenix2 Getting Lost6.jpg|none|thumb|x400px|This GPS track looks reasonable until marker #54, and then the track gets offset, but strangely it stays offset until the last marker.]]
|}
=Next Steps=
This is an initial analysis of the data I have, and there are a number of further evaluations to do.