Can mobile phones help people avoid getting COVID-19? That’s the concept behind the Government of Canada’s free and voluntary COVID Alert app. Designed to send out randomly generated codes via Bluetooth to other app users, the app lets Canadians know if they’ve been in contact with someone who tested positive for COVID-19. While concerns about the app’s privacy and accuracy dominate the media, Keatext decided to dig into the data to see what app users are saying about their experience with COVID Alert.
Since launching, COVID Alert has been downloaded more than 4.3 million times, with that number on the rise. Using Keatext’s AI text analytics technology, Keatext’s team analyzed COVID Alert’s publicly available unsolicited reviews from the Google Play Store and Apple’s App Store, where customers offer their opinions candidly in their own words, usually with the expectation that their feedback will be heard. User data could very well be the key to more people downloading the app and using it — we already know that the more people who use the app, the better it will work to keep Canadians safe and healthy.
Keatext analyzed:
- 2,505 comments from 2,496 commenters
- 1,896 comments from the Google Play Store
- 609 comments from the Apple’s App Store
- Posted between July 31 and October 5, 2020
The objective of Keatext’s analysis was to learn more about how people are using the app, what their concerns were before and after downloading it and what they think about various aspects of the app, from ease of installation to privacy concerns.
What the data shows about COVID Alert
Keatext’s first high-level level analysis of the data shows the following distribution of comments, with each comment categorized according to the emotion attached to it:
- 1,787 problems (45.19%)
- 1,570 praises (39.71%)
- 390 suggestions (9.86%)
- 207 questions (5.24%)
The average numeric rating of the app is 3.6 out of 5, with a slightly lower rating from Apple’s App Store commenters (3.5) than the Google Play store commenters (3.6).
The comments submissions over time show a peak during the month of August, from which the submissions decreased gradually until October.
With this high-level information in mind, Keatext dove deeper into six main areas that COVID Alert users were concerned with – privacy, ease of use, Bluetooth connectivity, battery life, support for phones, notifications – to reveal several insights about users’ experiences of the app.
Privacy and Canadian health
As with any app that connects people through their data, issues of privacy arise. Yet in the case of the COVID Alert app, users expressed far less negativity (19%) than positivity (80%) around the issue of privacy.
Users expressed far less negativity (19%) than positivity (80%) around the issue of privacy.
For many, comments about public health went hand in hand with comments about privacy, such as in this comment: “I was initially concerned about privacy, but this app provides a balance between preserving privacy and factual information to help Canadians help each other.”
For most users, once they downloaded the app and understood that it uses only Bluetooth technology to connect people, with no tracking of location or identity, their concerns about privacy dissolved. As one person comments: “Many people worry about privacy but I think this app really protects your privacy and has one purpose, to protect and serve the Canadian community health and welfare.”
Meanwhile, some users would like more information shared, such as dates and times when contact with COVID-19 occurs: “[Date and time] is not a privacy issue as no ID of positive contact and helps to rule out where one might have been exposed. We need more useful and therefore impactful information!”
Ease of use
For an app that is meant to be used by all Canadians, regardless of their level of tech know-how, ease of use is paramount. Alongside that, people expect a clear explanation of how the app works, especially valuable for users concerned about privacy.
Looking at the comments, 227 comments (9%) praised the app for being easy to use, without any complicated steps for installation or for use, as this comment illustrates: “App does a great job at introducing the issues around privacy and explaining them. Wish more did such a thorough job at explaining why they are using privacy impacted features.” Meanwhile, another comment points out that ease of use is often tied to ease of mind: “Super easy to understand and no complicated steps! I feel more calm now that people will be notified.”
Bluetooth connectivity
The technology the COVID Alert app uses is well-known to most users: Bluetooth’s common usage has been around since the ‘90s. The world of technology and connectivity has changed vastly since then and along with it people’s expectations, so it’s no surprise that Keatext uncovered 130 comments about Bluetooth, a main concern among users.
The majority of these comments were negative (61%): some related to battery life (“Downside: on all but the newest, most expensive phones, Bluetooth is a battery killer”), others related to the conflict that the app creates with other Bluetooth devices (“Its Bluetooth usage seems to conflict with/interrupt the Bluetooth connectivity between my Continuous Glucose Monitor (aka CGM) and my phone, causing that connection to drop 10-15 times daily”).
61% of comments about Bluetooth were negative. Some of them highlighted the conflict that this app creates with other Bluetooth devices.
Contrary to this, 29% of comments about Bluetooth were positive, particularly around the issue of privacy, as this comment shows: “Awesome app for society, and using Bluetooth not only helps with privacy it also allows for data to be stored in the app for long term use.”
Some of the comments (4%) included useful suggestions for further app updates: “Easy to use. Hope everyone downloads it. Message for the developers: Send a push notification whenever Bluetooth is disabled advising users that the app will not properly operate if Bluetooth is disabled!”
Effect on battery life
While the COVID Alert app doesn’t take much power to run on its own, if Bluetooth is constantly left on, it will use power. Of the 116 comments on battery use, most tended to skew negatively (63%), illustrating an area that needs improvement either with the app itself or instructions on how to use it.
Most of the comments on battery use got straight to the point (“Huge battery drainer.”), while other comments expressed a need for more clarity, with questions such as: “If my spouse and I are normally together, and we both download the app, it must constantly send codes for the two devices. Will this impact battery?”
Support for all phones
Another area of the negativity that Keatext’s analysis reveals concerned the app not working on all phones, especially on older phones such as the iPhone 6. Of the 100 comments about phones, many skewed negative (76%), while also revealing that users wanted to use the app but could not due to technology limitations: “This app should be usable no matter what, and Wifi and outdated phones should not be working against people wanting to be safe and know. I truly love this idea and I hope you make it usable for everyone.”
Notifications
The Covid Alert app lets users turn on notifications if they wish. Users who wanted that option expressed some frustration, with some not able to enable notifications at all, while others found the exposure report disconcerting and didn’t like getting notifications that weren’t exposure related.
From the 101 comments about notifications, 74% were negative, reporting problems such as notifications enablement issues or alerts simply not working on their particular phone.
From the 101 comments about notifications, 74% were negative, reporting problems such as notification enablement issues or alerts simply not working on their particular phone. Users also had questions and concerns about notifications and how they relate to testing: some commenters said they were alerted that they had been exposed to COVID-19, got tested and received negative result, yet continued to receive exposure notifications. Some comments focused on the app’s weekly push notifications being too distressing and alarmist, especially because they featured the “freaky” red COVID icon and weren’t proximity warnings, while others suggested that the title of the “Exposure Report” be changed to something less disturbing.
From user comments to actionable insights
Every app review tells a piece of the story about user experience: added up, they can paint a bigger picture of user behaviours and needs. Yet too often those comments aren’t analyzed for the valuable insights they can provide to decision-makers. Through AI text analysis of COVID Alert app feedback – which shows that users are primarily concerned with privacy, ease of use, Bluetooth connectivity, battery life, support for phones, and notifications – Keatext aims to add further insight on the available data that could lead to more Canadians using the app.
Positive or negative, every review provides more data that can be used to augment the COVID Alert app itself or aspects surrounding the app, such as help guides and public relations and marketing materials. With AI analysis, app developers are able to look at the true scope of their app’s impact and identity points of concern and points of positivity.
Unlike a simple star system, AI analysis of user comments provides the kinds of insights that can make a big difference on the success of the app itself as well as on app adoption across a broad cross-section of users – in the case of COVID Alert, a user base doesn’t get much broader or more diverse than all Canadians.
