Revision as of 17:03, 28 December 2016

Don't buy the Suunto Spartan Ultra. It's sad to be able to write such a simple review of a complex and expensive watch like the Spartan Ultra, but the product as it stands in October 2016 is far from acceptable. It's possible that over time the software updates will correct many of its deficits, but I think it may be a long road with an uncertain outcome. This review is based on my initial impressions, and I'll expand it as I gain more experience. I wouldn't normally write a review this early, but I believe the clear issues with the Spartan justify some early warning. For a watch to be this bad is rather grim, but it this price it seems more like a cruel joke.

1 Spartan Pros

So far, there are no reports of the Spartan catching fire, so it's ahead of the Samsung Note 7. Beyond that, it's hard to find much I like. The watch strap is quite soft, and the watch is made of quality materials but the Garmin Fenix 3 seems nicer to me, and it's a lot cheaper (though still expensive.) As I expand this review, I'll dig into some of the features that have promise, like the Navigation options, but even there other watches do it better for far less.

2 Spartan Cons

This is far from an exhaustive list, but it gives you the highlights.

• The Spartan is clearly incomplete, with only a subset of the features completed. There's no option to configure the display, no support for the Suunto Apps from the Ambit range, no way of using the display with inverted colors, etc. I realize the idea of "release early, release often" has value, and that it can be important to release a "minimum viable product". However, when a $700+ watch like the Spartan seems bare bones when compared with a $150 Garmin Vivoactive then the release should have been delayed.
• Another consequence of the unfinished nature of the Spartan is there are quite a few bugs. So far, I've not had it crash, but lots of things just don't work.
• The GPS Accuracy is looking really bad. It's way too early to release any figures as I only have ~150 data points, but I think the Spartan stands a reasonable chance of being the worst watch I've tested. Initial testing indicates it's just below the Garmin 10, but the lack of data means its scoring may improve with further testing. I normally consider 500 data points to be the minimum for GPS Accuracy, though I know some authors that will make assumptions based on a few trial runs. It's possible that Suunto will improve the accuracy with software updates, but it's hard to imagine they can reach even a mediocre performance from here. I thought the Garmin 620 had grim GPS accuracy when it was first released, and it was widely panned in the forums (though not by reviewers). To their credit, Garmin listened and managed to improve the 620's performance to a reasonable level. However, the 620's initial performance doesn't look so bad when I compare it to the Spartan.
• I found the Spartan to be surprisingly uncomfortable. It's a large watch by modern standards, but it's tiny compared with the old Garmin 310XT which worked for me nicely. The Spartan is lighter than the Garmin Fenix 3 but feels a lot heavier. I think part of the problem is that the watch strap is not articulated where it reaches the body of the watch, so it doesn't conform to your wrist.
• The display on the Spartan has great resolution, the highest by far of any watch I've tested. This makes it sharp and allows for a lot of data on the screen. However, I find that the display is rather dim, with the lightest color a dingy gray rather than white. In most lighting conditions I find it hard to read, and it's worse than competitors color displays. I've found that color displays are attractive, but so far no one's used color to provide meaningful functionality. I'd much rather have a monochrome display than a color LCD. The back light makes it quite easy to read, but having to hit the backlight each time you want to check the display is a pain. (The Fenix 3 trick of turning the backlight on when you raise the watch to look at it is pretty cool, and badly needed on the Spartan.)
  
  

  The Garmin Fenix 3, the Spartan Ultra, Garmin 920XT, Garmin Vivoactive, and the Suunto Ambit3.
• The price starts at $700 and goes up to nearly $1,000. That seems rather high for an outstanding and groundbreaking watch, and even if the Suunto worked perfectly and had more features, there's nothing that is likely to break new ground other than the price. Even there you could argue the Apple Watch has gone to even greater extremes of conspicuous consumption.

Normally I give a link to each product I review on Amazon, and I get a commission from the sales. I buy all the gear I test using the same outlets as any other runner, which allows me to write critical reviews free from the concern that the manufacturer won't give me freebies in the future. I don't review the products for the financial benefits, but it does provide the money to buy new gear to test. (I do buy gear to test that isn't available on Amazon, like RunScribe. There's no benefit to me, but it fits in with my goal of supporting the running community.) However, I can't in good conscience give a link for you to buy the Spartan Ultra. If you really want the Spartan, I'd recommend holding off for a while to see if they can improve things. If you've already got a Spartan, I can recommend a MilestonePod so that you'll have a better idea of how far you've run.

3 The Spartan Ultra in Context

To understand my angst with the Suunto Spartan, let me compare it to another device, the TgForce. Neither device is ready for prime time as they both have missing features and significant bugs. However, the problems I'm seeing with the TgForce are exactly what you'd expect from a small startup, while the Spartan comes from a major company. But more importantly, the TgForce sensor is a new and novel device that has the potential to improve a runner's form and reduce injury rates. This means that even though TgForce is not really ready for prime time, it's still an exciting device to use. On the other hand, even if Suunto can fix the problems with that the Spartan, all you'll have is a rather overpriced watch that doesn't do anything that existing devices don't already do, often better.

4 GPS Accuracy

The Spartan has the worst GPS Accuracy of any watch I've tested, but to back up the numbers and to visualize things in a more qualitative way I've included the images below of the GPS tracks from one part of the testing course. Below are the tracks from the latest firmware (1.4.6), along with the earlier firmware I tested (1.2.4) and from the Suunto Ambit3 which is one of the most accurate watches I've tested. You can see that Suunto have improved the accuracy with the 1.4.6 firmware, but the Spartan is still the least accurate watch I've tested. The firmware release says "A fix to GPS signal filtering, reducing the number of cases with large location offsets in GPS tracking experienced by some users" and you can see that the new firmware doesn't have the problems with offset tracks in the same way that 1.2.4 did. However, though the accuracy has improved it's still appalling. The blue lap markers are scattered, highlighting positional inaccuracy, though that's improved with this firmware (my analysis focuses on the distance measurement.) Most watches have a problem where the path curves in the middle section, and the Spartan does badly here, though there are a few green lines indicating that occasionally the Spartan is close. The Spartan does badly in the sharp turn on the right side of the picture, an area where most watches do surprisingly well. The section under the bridge doesn't show any obvious signs that the Spartan is failing to reacquire the GPS signal, though there the shape of the tracks to the right of the bridge is a little odd. Overall, it's good to see that Suunto have improved the GPS accuracy, and it's a statically significant improvement (p<0.05) but they have a very long way to go. The tracks look better, so if the problems won't be quite so obvious to a casual user, but the distance measured can be wildly off.

4.1 SGEE and Accuracy

There are some suggestions online that syncing the Spartan before you run will improve GPS accuracy. The idea is that the sync will update the cache of satellite information (SGEE), which in turn will improve accuracy. This satellite information is a prediction of the GPS Ephemeris data, which gives information needed to use a satellite's signal for calculating the watch's position. Because it takes about 30 seconds to get a satellite's Ephemeris, having it preloaded reduces the time to get a usable location (Time To First Fix, TTFF.) This predicted Ephemeris data is not typically used to improve accuracy, but a publication from Telit suggests otherwise for the chipset used in the Spartan. The document is rather ambiguous, but I interpret it to mean that for the time between the watch getting an initial fix using the SGEE data and getting the true Ephemeris from the satellite the accuracy will be compromised. Given this should only be for a minute or so at the start of a run, this shouldn't have a practical impact. It's possible the situation may reoccur mid-run as satellites drop below the horizon and others rise up, as the watch will use SGEE data until the new satellites Ephemeris has been downloaded. To test how this worked in practice I did a few runs with the SGEE data expired (>7 days since the last sync) and I found that the GPS accuracy was not impaired. If anything the accuracy might have been better, but I didn't collect enough data for statistical analysis.

5 Comparison Table

I evaluate running watches in three distinct ways. Firstly, you can use a watch on its own, without any kind of Footpod. This is probably the most common way runners use their watch, but you miss out on a lot. The second rating is with a standard Footpod that is available quite cheaply. These Footpod's can be reasonably accurate once the calibrated, but calibration is a little tedious. The final evaluation is with the Stryd Footpod, which is vastly more accurate than any other type of Footpod, or and more accurate than GPS. The table below looks at the score, and the value for money of each watch for each of the three conditions. (I’ve also tested the Apple Watch 3, but I’ve not included it in these tables as it’s not really a running watch.)

The score is the sum of how well each watch can answer the four basic questions (how far, how fast, where are you, what's your cadence), plus some bonus points.

1. The "How far you've run" will be based on GPS only for "without Footpod" and "with Standard Footpod", but based on Stryd if supported in the "with Stryd Footpod" table..
2. How fast you're running assumes you're using a Footpod if it's supported, otherwise the rating is 0-2 based on GPS accuracy.
3. The "Where are you?" is based on various navigation features such as back to start, breadcrumbs, and preloaded maps. For some watches, you have to turn GPS off to get the benefit of Stryd, so those watches have worse "where are you scores" with Stryd than without.
4. The cadence score uses 1 point for an internal cadence sensor, 2 points for footpod support, 1 point for support from chest strap cadence, and 1 point for cadence alerts.
5. I give 1-2 bonus points for application support, 1-2 bonus points for data upload, 1-2 bonus points for Optical Heart Rate Monitoring, and 0-1 bonus points for battery life.
6. Value for money is the score divided by the price (at the time I last updated the table.) Your needs may be different, so you might weight the different aspects of the watches differently, or be basing your decision on different criteria totally. Hopefully this table will give you a good starting point for your decision.

5.1 Score Breakdown without a Footpod

5.2 Score Breakdown with a Standard Footpod

5.3 Score Breakdown with a Stryd Footpod

5.4 Basic Features

6 Navigation Features

For "navigation":

• Color Maps gives you full color maps, rather like a smart phone, with roads and paths marked out.
• Track Outline is a display of where you've run, rather like a breadcrumb trail. If there are maps, the outline is superimposed otherwise this is just the outline on its own without any context.
• Course Outline is an outline of a route that can be downloaded. I've found this useful during ultras or in unfamiliar cities where I've needed to know where to go.
• Back To Start is a simple arrow point to your starting point, so it won't help you backtrack.
• Back To Waypoint returns you to a previously marked location using a simple arrow to point.
• Compass. A magnetic compass can help you orient yourself or the map. Without a magnetic compass you have to be moving for the GPS to give you a sense of direction.

(Older Reviews: Polar RC3 GPS, Soleus 1.0, Motorola Motoactv.)