8,153
edits
Changes
Created page with "I evaluated the real world accuracy of GPS watches using the {{Garmin 310XT}} and the {{Garmin 910XT}} while running over 1,100 miles/1,800Km and recording over 4,000 data poi..."
I evaluated the real world accuracy of GPS watches using the {{Garmin 310XT}} and the {{Garmin 910XT}} while running over 1,100 miles/1,800Km and recording over 4,000 data points. On average, the watches were remarkably good, but this is because the errors even out.
=Methodology=
To gather the data for this test I marked my usual running route at quarter-mile intervals, using a [http://www.jonescounter.com/ Jones counter] to provide an accurate measure of distance. The Jones counter is the device used to certify courses, and I followed the [http://www.usatf.org/Products-/-Services/Course-Certifications/USATF-Certified-Courses/Certify-Your-Course.aspx USATF course certification process]. The course I run along is a little challenging for GPS, with lots of twists, tree cover, and one bridge that I go under. The bridge carries a 4 lane road, so it's wide enough to cause the watches to briefly loose GPS signal. However, I believe that it's reasonably representative of real-world conditions, and probably less challenging than running in the city with skyscrapers. At both ends of the course there is a turnaround, and I set the mark an eighth of a mile from the end. That way I can evaluate how well the watches handle an about turn. So far I have recorded 4,658 GPS points, 3,881 using a {{Garmin 310XT}} and 777 using a {{Garmin 910XT}}. This represents 1,164 miles/1,874 Km of running.
[[File:Course Overview.jpg|none|thumb|500px|This is the course I use to evaluate the accuracy of [[Best Running Watch| GPS Running Watches]].]]
=Accuracy, Trueness and Precision=
For this evaluation I'll use the definition of [http://en.wikipedia.org/wiki/Accuracy_and_precision Accuracy as the combination of trueness and precision]:
{| class="wikitable"
|- valign="top"
|[[File:High precision Low accuracy.svg|none|thumb|x300px|This is an example of high precision, as all the hits are tightly clustered. However, the trueness is poor as all the hits are off center, so accuracy is low.]]
|[[File:High accuracy Low precision.svg|none|thumb|x300px|This shows good trueness, as all the hits are around the center. On average they are on target, but there is poor precision, as the hits are scattered.]]
|}
We can look at trueness by measuring the average lap length and precision by measuring the standard deviation.
=Accuracy=
I've divided the laps into those that go under the bridge, those that are immediately after a lap that goes under the bridge, those that turn around and then all the remainder (normal).
{| class="wikitable"
! Type
! Count
! Trueness
! Precision (Standard Deviation)
! 95% within +/-
! Dropouts
|-
| Normal
| 3,452
| 0.19%
| 3.01%
| 12.0%
| 1.8%
|-
| Under Bridge
| 252
| 1.27%
| 3.51%
| 14.0%
| 100.0%
|-
| After Bridge
| 448
| 2.24%
| 3.18%
| 12.7%
| 1.8%
|-
| Turn Around
| 217
| 4.01%
| 3.76%
| 15.0%
| 2.3%
|}
=Interpretation=
What do these statistics mean? Is my interpretation:
* Away from the bridge and the turnaround, the average distance recorded (trueness) is quite good, averaging out to 0.19%, which is 30"/74cm per quarter mile.
* Not surprisingly, the trueness drops going under the bridge and is 1.27% (201"/500cm). More interestingly, the accuracy just after the breach is even lower at 2.24% (355"/885cm). This suggests that the GPS watches are struggling to reacquire the satellites.
* The turnarounds are even less accurate than going under a bridge, resulting in an average distance that is out by 4.01% (635"/1,582cm).
* The precision of the GPS watches is 3-4%, which means that 95% of samples will fall within 12-15% of the correct distance. For normal laps, that is 159'/4.8m per quarter mile.
* As expected, the GPS watches have dropouts (use satellite signal) on every lap that goes under the bridge. The watches also have dropouts 1.8% of the time on normal laps.
=Recommendations=
This data leads me to a few conclusions.
* GPS watches are accurate enough for most general running.
* It takes time for the GPS watches to acquire the satellites. The watches tended to say they are ready to go before they have an optimal lock. Therefore, to improve accuracy try to give them a little more time. Note that some newer GPS watches such as the [http://www.amazon.com/Garmin-010-01128-30-Forerunner-620-Bundle/dp/B00FBYYC90 Garmin 620] have the ability to be preloaded with the satellite positions, reducing this startup time and start up in accuracy dramatically.
* The GPS watches are not accurate enough to perform any type of [Practical Interval Training| interval training]. Instead, use a track or measure out the distance using some other mechanism.
=Next Steps=
This is an initial analysis of the data I have, and there are a number of further evaluations to do.
* Separate out the data from the {{Garmin 310XT}} and the {{Garmin 910XT}} to see how they compare accuracy.
* Gather data from some other [[Best Running Watch| GPS Running Watches]].
** I am in the process of evaluating the [http://www.amazon.com/Polar-Heart-Rate-Monitor-Black/dp/B00BV81U7W Polar RC3 GPS], but I don't think it will make my recommendation for [[Best Running Watch]es. It may however provide a useful example of poor quality GPS reception, and as I've seen some rather erratic tracks (note to self, check Oct 10 run).
** I will evaluate the [http://www.amazon.com/Garmin-010-01128-30-Forerunner-620-Bundle/dp/B00FBYYC90 Garmin 620] when it becomes available as it has some promising looking functionality.
* Examine and compare specific runs where GPS quality is lower. I have some examples where I recorded the run with two watches, and only one had poor accuracy (and not the same one each time).
* Check how GPS accuracy changes over the course of a run, as I've seen a distinct tendency for the watches to say they are good to go when they don't really have an optimal lock on the satellites.
* Look at how accurate the GPS watches are for measuring innovation, and compare with barometric data.
* Check for any correlation between accuracy and running pace.
* Look for any correlation between accuracy and the use of heart rate monitor. The radio signal from the heart rate monitor could interfere with accuracy.
* Evaluate how a [[Footpod]] influences accuracy. Most of my runs are with a Footpod, and this may improve the accuracy if GPS is lost.
=Methodology=
To gather the data for this test I marked my usual running route at quarter-mile intervals, using a [http://www.jonescounter.com/ Jones counter] to provide an accurate measure of distance. The Jones counter is the device used to certify courses, and I followed the [http://www.usatf.org/Products-/-Services/Course-Certifications/USATF-Certified-Courses/Certify-Your-Course.aspx USATF course certification process]. The course I run along is a little challenging for GPS, with lots of twists, tree cover, and one bridge that I go under. The bridge carries a 4 lane road, so it's wide enough to cause the watches to briefly loose GPS signal. However, I believe that it's reasonably representative of real-world conditions, and probably less challenging than running in the city with skyscrapers. At both ends of the course there is a turnaround, and I set the mark an eighth of a mile from the end. That way I can evaluate how well the watches handle an about turn. So far I have recorded 4,658 GPS points, 3,881 using a {{Garmin 310XT}} and 777 using a {{Garmin 910XT}}. This represents 1,164 miles/1,874 Km of running.
[[File:Course Overview.jpg|none|thumb|500px|This is the course I use to evaluate the accuracy of [[Best Running Watch| GPS Running Watches]].]]
=Accuracy, Trueness and Precision=
For this evaluation I'll use the definition of [http://en.wikipedia.org/wiki/Accuracy_and_precision Accuracy as the combination of trueness and precision]:
{| class="wikitable"
|- valign="top"
|[[File:High precision Low accuracy.svg|none|thumb|x300px|This is an example of high precision, as all the hits are tightly clustered. However, the trueness is poor as all the hits are off center, so accuracy is low.]]
|[[File:High accuracy Low precision.svg|none|thumb|x300px|This shows good trueness, as all the hits are around the center. On average they are on target, but there is poor precision, as the hits are scattered.]]
|}
We can look at trueness by measuring the average lap length and precision by measuring the standard deviation.
=Accuracy=
I've divided the laps into those that go under the bridge, those that are immediately after a lap that goes under the bridge, those that turn around and then all the remainder (normal).
{| class="wikitable"
! Type
! Count
! Trueness
! Precision (Standard Deviation)
! 95% within +/-
! Dropouts
|-
| Normal
| 3,452
| 0.19%
| 3.01%
| 12.0%
| 1.8%
|-
| Under Bridge
| 252
| 1.27%
| 3.51%
| 14.0%
| 100.0%
|-
| After Bridge
| 448
| 2.24%
| 3.18%
| 12.7%
| 1.8%
|-
| Turn Around
| 217
| 4.01%
| 3.76%
| 15.0%
| 2.3%
|}
=Interpretation=
What do these statistics mean? Is my interpretation:
* Away from the bridge and the turnaround, the average distance recorded (trueness) is quite good, averaging out to 0.19%, which is 30"/74cm per quarter mile.
* Not surprisingly, the trueness drops going under the bridge and is 1.27% (201"/500cm). More interestingly, the accuracy just after the breach is even lower at 2.24% (355"/885cm). This suggests that the GPS watches are struggling to reacquire the satellites.
* The turnarounds are even less accurate than going under a bridge, resulting in an average distance that is out by 4.01% (635"/1,582cm).
* The precision of the GPS watches is 3-4%, which means that 95% of samples will fall within 12-15% of the correct distance. For normal laps, that is 159'/4.8m per quarter mile.
* As expected, the GPS watches have dropouts (use satellite signal) on every lap that goes under the bridge. The watches also have dropouts 1.8% of the time on normal laps.
=Recommendations=
This data leads me to a few conclusions.
* GPS watches are accurate enough for most general running.
* It takes time for the GPS watches to acquire the satellites. The watches tended to say they are ready to go before they have an optimal lock. Therefore, to improve accuracy try to give them a little more time. Note that some newer GPS watches such as the [http://www.amazon.com/Garmin-010-01128-30-Forerunner-620-Bundle/dp/B00FBYYC90 Garmin 620] have the ability to be preloaded with the satellite positions, reducing this startup time and start up in accuracy dramatically.
* The GPS watches are not accurate enough to perform any type of [Practical Interval Training| interval training]. Instead, use a track or measure out the distance using some other mechanism.
=Next Steps=
This is an initial analysis of the data I have, and there are a number of further evaluations to do.
* Separate out the data from the {{Garmin 310XT}} and the {{Garmin 910XT}} to see how they compare accuracy.
* Gather data from some other [[Best Running Watch| GPS Running Watches]].
** I am in the process of evaluating the [http://www.amazon.com/Polar-Heart-Rate-Monitor-Black/dp/B00BV81U7W Polar RC3 GPS], but I don't think it will make my recommendation for [[Best Running Watch]es. It may however provide a useful example of poor quality GPS reception, and as I've seen some rather erratic tracks (note to self, check Oct 10 run).
** I will evaluate the [http://www.amazon.com/Garmin-010-01128-30-Forerunner-620-Bundle/dp/B00FBYYC90 Garmin 620] when it becomes available as it has some promising looking functionality.
* Examine and compare specific runs where GPS quality is lower. I have some examples where I recorded the run with two watches, and only one had poor accuracy (and not the same one each time).
* Check how GPS accuracy changes over the course of a run, as I've seen a distinct tendency for the watches to say they are good to go when they don't really have an optimal lock on the satellites.
* Look at how accurate the GPS watches are for measuring innovation, and compare with barometric data.
* Check for any correlation between accuracy and running pace.
* Look for any correlation between accuracy and the use of heart rate monitor. The radio signal from the heart rate monitor could interfere with accuracy.
* Evaluate how a [[Footpod]] influences accuracy. Most of my runs are with a Footpod, and this may improve the accuracy if GPS is lost.