Revision as of 07:28, 23 April 2014

I evaluated the real world accuracy of GPS watches while running over 3,500 miles/5,600Km and recording over 14,000 data points as part of my evaluation of the Best Running Watches. Under good conditions most of the watches are remarkably good, but when things get a little tough the differences become more apparent.

An infographic of the accuracy of the GPS running watches. The top right corner represents the most accurate watches. (This graphic uses ISO 5725 terminology.)

The table below is a simplified summary of the results, where a '10' would be a perfect device. (For an explanation of the ISO 5725 terms 'trueness', 'precision' and 'accuracy', see below.)

1 Methodology

Simply taking a GPS watch on a single run does not provide sufficient data to reasonably evaluate its accuracy. So to gather the data for this test I marked my usual running route at quarter-mile intervals, using a Jones counter to provide an accurate measure of distance. The Jones counter is the device used to certify courses, and I followed the USATF course certification process. The course I run along is a little challenging for GPS, with lots of twists, tree cover, power lines, and one bridge that I go under. The bridge carries a 4 lane road, so it's wide enough to cause the watches to briefly loose GPS signal. However, I believe that it's reasonably representative of real-world conditions, and probably less challenging than running in the city with skyscrapers. At both ends of the course there is a turnaround, and I set the mark an eighth of a mile from the end. That way I can evaluate how well the watches handle an about turn.

This is the course I use to evaluate the accuracy of GPS Running Watches.

To prevent startup problems each device was turned on, satellites acquired, and then left for around 5 minutes before the run starts. This is to counter the problem of a device claiming to have acquired the satellites when it only has a minimal lock. The watches are worn on the left wrist or held in the left hand in roughly the same orientation as it would be on the left wrist. (The iPhone was also on the left wrist.)

2 Accuracy, Trueness and Precision

For this evaluation I'll use the ISO 5725 definition of Accuracy as the combination of trueness and precision.

This is an example of high precision, as all the hits are tightly clustered. However, the trueness is poor as all the hits are off center, so accuracy is low.

This shows good trueness, as all the hits are around the center. On average they are on target, but there is poor precision, as the hits are scattered.

We can look at trueness by measuring the average lap length and precision by measuring the standard deviation. (It is more common in many fields to use "accuracy" to mean closeness to true value and "validity" to mean the combination of accuracy and precision. However, I feel that the meanings used by ISO 5725 are closer to the common usage.)

3 Accuracy

I've divided the laps into those that go under the bridge, those that are immediately after a lap that goes under the bridge, those that go next to or under the power lines, those that turn around and then all the rest (normal). I've also included a 'mixed' category that reflects all the sections and a category for the straightest segment. The count field is how many measurements I have for that combination of condition and device. I generally aim for over 1,000 data points to even out the effects of weather, satellite position and other factors.

3.1 Progress of newer watches

I expected GPS watches to improve with time, but the opposite appears to be happening. With the Garmin devices especially, you can see that the older watches generally do far better than the newer ones. I suspect this is due to compromises to get better battery life and smaller packaging and the cost of GPS accuracy.

3.2 Interpretation and Conclusions

What do these statistics mean? This is my interpretation:

• Under normal conditions the GPS accuracy is quite good for most devices.
• The Garmin 620 and Garmin 10 are noticeably poorer than the other devices. I found the accuracy of the 10/620 in general usage to be rather grim, and I did some testing pairing them up with the 610 or the 310XT. In all cases the 10/620 would have poor accuracy compared with the 610 or 310XT on the same run. On one run, the 620 lost over a mile over a 28 mile distance. When Garmin replaced my 620, the new watch would lose over a mile on a 16 mile run!
• The results of the Garmin 610 indicate the problems with the 620 & 10 are not inherent in a smaller device.
• The accuracy of all devices is better in a straight line than on curves or bendy routes. My course is a tough test for GPS devices with many curves and only a few relatively straight sections.
• Not surprisingly, accuracy drops going under the bridge.
• More interestingly the trueness just after the bridge is even lower, suggesting that the GPS watches are struggling to reacquire the satellites.
• The turnarounds are even less accurate than going under a bridge, but Power Lines do not seem to impact accuracy noticeably.
• The Footpod improves the accuracy of the 310XT.
  • Note that I'm intentionally using an uncalibrated Footpod (factor = 1.000) to gather data for a comparison of Foodpod and GPS.
• The Garmin 205 does remarkably well.

4 Recommendations

Here are some recommendations for GPS watches.

• GPS watches are accurate enough for casual running.
• The better devices are accurate enough for most runners if their limitations are understood.
• None of the devices were accurate enough for a runner to trust the display of current pace for training or race pacing.
  • For interval training, use a track or measure out the distance using some other mechanism.
  • For general training or for races, use a device that supports displaying pace from the Footpod while using GPS for distance.
• Adding a Footpod to the Garmin 310XT improves its GPS accuracy.
• For the Garmin 610 there was no difference with and without the Footpod. (Trueness was 3.33%/3.32%, Precision was 3.54%/3.68%, with/without).
• It takes time for the GPS watches to acquire the satellites. Some watches tended to say they are ready to go before they have an optimal lock. Therefore, to improve accuracy try to give them a little more time. Note that some newer GPS watches such as the Garmin 620 have the ability to be preloaded with the satellite positions, reducing this startup time and start up in accuracy dramatically.

5 Even GPS Watches have Bad Days

While it's tempting to take the various GPS watches on a single run and simply compare the totals, this is a flawed approach. Evaluating the devices GPS accuracy on the basis of a single sample does not tell you much. It's a bit like evaluating an athlete's ability on the basis of one event; everyone has good days and bad days, and that applies to GPS watches as well. To illustrate this, the images below are from two runs, recorded on 9/20 and 9/22. In each run I recorded data on both the 310 and 910 watches, hitting the lap button on both at as close to the same time as is humanly possible. On 9/20 the 910XT was far more accurate than the 310XT, but on 9/22 the situation is reversed. If you were to have evaluated the two watches on the basis of a single run, you would conclude that one is much better than the other. But which device would win would depend on the particular day. This is why I've accumulated a lot of data to do a statistical analysis to work out which is really better.

The Garmin 310XT having a bad day. You can see on the upper half of the course where it got a little confused and off track.	The Garmin 910XT on the same run having no problems, and only the standard, expected level of inaccuracy.
Two days later and it's the turn of the Garmin 310XT to have a good day.	Again, this track is recorded on the same run as the image to the left. The Garmin 910XT gets a little confused at the start, and then again around lap 27.

6 Some Devices Are Better Than Others

Below is a section of two runs showing the same section of the course, both taken at the same time, one from the Garmin 310XT and the other from the Garmin 620. These give a good indication of the accuracy problems I've seen with the Garmin 620.

You can see the GPS tracks (thin red line) are close together and the lap markers (yellow diamonds) are clustered nicely. The blue dots on the GPS tracks are the actual GPS recordings.

By contrast, the 620 has much wider GPS tracks and dispersed lap markers.

7 GPS Short and long measurements

As you can see from the images below, the GPS track tends to take shortcuts around bends, reducing the length of the measured track. This cutting of the corners indicates the devices are doing some post-hoc smoothing to try to overcome the GPS errors. The more smoothing they do, the better the accuracy is likely to be in a straight line and the worse it is around corners or twisty courses.

The GPS tracks in red with blue dots representing each GPS point recorded. The arrow points to the actual path taken.

Often GPS measurements of races, especially marathons record a longer distance than the race. This is partly because the USATF technique for measuring the distance takes a path that is no more than 12 inches away from the tangent (corner), and few runners are able to run that close. In a large marathon you can be forced to take a line that is a long way from the tangent. The other factor is that on a straight line, the GPS error tends to give a slightly longer measurement.

Here you can see the GPS line is not following the straight road, giving a longer reading on the Thunder Road Marathon. Notice that the GPS is also cutting the corner at the top (we didn't run through the building).

8 Detailed Breakdown by Condition

Below are more detailed statistics for each device under different conditions. This is rather a lot of data, but it allows you to see where various devices seem to have problems.

8.1 Devices then Condition