WHOOP Review: a good idea fatally flawed?

The current iteration of WHOOP is deeply, possibly fatally, flawed. There may be some usefulness for some athletes, if you understand the problems. However, of the five WHOOP devices I've had, only one has worked even moderately well, so you may also need to be lucky.

1 What is WHOOP, and why might you want one?

This is not a full review of WHOOP, and I'd recommend checking out something like the 5K runner review as background. For our purposes, we going to assume that you are an athlete, and you are looking to optimise your training and recovery. Knowing how hard to push your training, and how much to recover is one of the objectives of good training. and this is the primary purpose of WHOOP, so you'd hope they do a better job than alternatives like Garmin. (I've found little to no value in Garmin's training advice, though it's general recovery metrics are useful.)

2 Accuracy Issues

The principle of WHOOP is to measure things about your body and use that to understand your training stress and recovery. WHOOP primarily uses heart rate, combined with movement. It also seems to use skin temperature and blood oxygen. The details of how it uses this data is a black box, proprietary model. Anyone who has used to heart rate monitor for any length of time will know that bad readings are inevitable. even a good quality chest strap will sometimes give bad data. Optical heart rate monitors, like WHOOP, are especially vulnerable to bad data. The latest generation of optical heart rate monitors, like the Polar Verity Sense, have become good enough for most non-critical usage. Unfortunately, WHOOP has a number of compounding problems.

• Quality Control. I have had five different WHOOP devices. Two of them had appalling problems with heart rate accuracy, and even slight movement would throw them off. Two devices had the removable strap jammed on, meaning they could only be used on the wrist. Finally, I got a fifth device which achieved moderately good heart rate accuracy when worn on the upper arm. It's not as good as the best in class, but it tends to mostly have bad readings in the first few minutes. It's even given accurate readings on the wrist when using a stationary bike. This suggests to me that WHOOP has significant quality control problems. But even if you're lucky, you're still not getting a best in class heart rate monitor.
• Band Design. The design of the band and its locking mechanism seems quite clever at first glance. However, I found it made it quite tricky to get the tension right, the mechanism tended to pinch my skin, and it was hard to open and close. The band needs to be taken off to change the length, making it tedious to get right and impractical to move it around as each new position requires readjustments. The band isn't stretchy, so arm movements change the pressure, which makes it both uncomfortable and unreliable. If you compare this with the Polar Verity Sense, the best in class, you can see how it should be done. The polar has a stretchy band so it works with arm flexing, and it has an attachment that means you can just close it with the right level of tension. It's amazing that after decades of watch straps, WHOOP can create something so much worse.
• Sensor Design. The focus of the sensors appears to be long battery life, rather than data quality. The internal components talk about "Ultra Low-Power" rather than quality. The device only has three green HR LEDs in a line, close together. Modern optical HRMs use more LEDs and have them in a circle around the sensor. If you look at the firmware updates, there are several updates that mention "Improved strap stability", and one "Improved heart rate estimation algorithm", suggesting that WHOOP are fully aware of the problems and trying to fix them in software.
• Cheap Not Cheerful. WHOOP seems cheaply made, rather than a quality product. While the bands are expensive, they feel poorly made to me. It doesn't seem like WHOOP has put the effort into creating a premium product, just premium marketing.
• Placement. WHOOP seems to have been created to be worn on the wrist, which is the worst place on the arm for heart rate sensing. The company does recommend you wear the band high up on your bicep, just below the deltoid muscle. This is a strange recommendation as the flexing of your bicep tends to increase and decrease the tension and make accurate readings even harder. Positioning an optical heart rate monitor on the inside of the upper arm just above the elbow seems to be the optimum position, as it is for most optical HRMs.

3 The possibly fatal flaws

I'd really like WHOOP to work, and I was hoping that the flaws would be small enough that there was still value. Sadly, I think I was wrong.

• As noted above, I only had one device out of five that worked. Such poor data means any algorithm WHOOP uses will suffer from "garbage in, garbage out."
• Having used many different optical heart rate monitors, the data I get from the fifth WHOOP device is comparable with an older generation, not the latest technology. WHOOP seems especially sensitive to positioning and tension on the strap. Like older Optical HRMs, WHOOP really struggles at the start of a run, often reading 20-80 BPM too high.
• Because WHOOP doesn't have a display, it can be far less obvious when a reading is bad. You can overcome this by having broadcasting the WHOOP heart rate and displaying it on a watch. However, that means the bad heart rate data from WHOOP will be stored in the watch. My solution was to have a chest strap heart rate monitor sending good data to my main watch, then have a second watch that displays the WHOOP heart rate and keep an eye on the difference.
• Sadly, WHOOP has no ability to use a better heart rate monitor. There's no way of linking it to a chest strap or getting it to take heart rate data that's been stored in Strava. So even when you have good data, WHOOP can't make use of it.
• If the bad heart rate data WHOOP detects is above your maximum heart rate, it will reset your heart rate zones and then underestimate the impact of any future activities until you notice. You can minimise the impact of this by setting manual heart rate zones, but it doesn't seem to be possible to get WHOOP to ignore excessively high heart rate data.
• Where all this becomes a potentially fatal flaw is there is no way of telling WHOOP that a workout contains bad data to ignore it. So, if you do an easy run where WHOOP records it with a wildly high heart rate, the strain will be massive. This will then confuse the model, as it looks like a very, very hard day rather than an easy day. WHOOP claims that bad data will only impact their model for three weeks. It strikes me as very strange that data only impacts their model for three weeks, as most training models have a much longer time horizon. But even three weeks can render WHOOP useless, as it's hard to get a three week stretch without bad data.

4 Additional Flaws

In addition to the main flaws above, there are other issues to be aware of.

• Because WHHOP has a subscription model, there's no residual value. If you decide you don't like it, you can't give it away or sell it.
• The lack of display means you're probably going to have to use another device for displaying metrics. So, WHOOP is only ever going to be an add-on.

5 The Benefits of WHOOP

Given all the problems, it's worth pointing out there are still some residual benefits to WHOOP.

• The sleep tracking seems reasonable, and better than my Garmin. Even having a full sleep clinic analysis has some subjectivity to working out sleep stages, so any device that shows sleep stage can only be treated as a rough estimate. Looking at the simple awake and asleep data WHOOP seems more plausible than Garmin.
• The recovery metric is nice and simple and seems broadly reasonable. I get the impression that it's heavily based on overnight HRV, rather than modelling strain and performance. The recovery metric is easier to understand than what's provided by Garmin but doesn't seem to add much.
• I really like the idea of the target strain, where you're told how hard you should work out. But the values given don't seem to reflect my actual training status, and translating their strain numbers into a training run is unclear at best. In theory you can use the app and run until it says you've had enough strain, but that doesn't help you plan a training session.

6 Other reviews of WHOOP

Lots of reviewers seem to respect WHOOP and think it's worthwhile. Are they able to get much better data than I am? Are they getting "reviewer special" devices that perform much better than retail units? Is WHOOP providing incentives that bias reviews? I'm not sure, and I read enough reviews from people I believe are credible to pay for WHOOP. There are some contrary views, such as this analysis Data Scientist Breaks Down why WHOOP doesn't Work, but they are few and far between. I should have listened to the many individuals that don't write reviews, but found WHOOP to be unhelpful at best, rather than the bigger reviewers. Caveat Emptor, Your Milage May Vary, etc.

7 Should you get a WHOOP?

Maybe. It seems like the sleep tracker is reasonable, and maybe if you're a cyclist the data quality issues won't be so bad. The sleep score and recovery metrics seems plausible, though given all the issues, it's hard to trust WHOOP. Personally, I'm finding WHOOP like a sick cat that I need to look after rather than a valuable training tool.