Study Reveals Inaccuracy of Smartphone Health Apps

Performance of smartphone applications in assessing melanoma risk is highly variable and 3 of 4 applications incorrectly classified 30 percent or more of melanomas as unconcerning, according to a study published in JAMA Dermatology ("Diagnostic Inaccuracy of Smartphone Applications for Melanoma Detection"; jamadermatol.2013.2382). The authors suggest that reliance on these applications, which are not subject to regulatory oversight, and not seeking medical consultation can delay the diagnosis of melanoma and potentially harm users.

This is the first example I have seen of a study evaluating the accuracy of health apps, but it may not be the last. Sooner or later, apps developed by pharma companies will also be subject to such scrutiny by academics, and also by the FDA.

I myself have reviewed several health apps developed by pharma companies that may be "buggy" and deserve to be rigorously tested for accuracy (see "Some Unregulated Physician Smartphone Apps May Be Buggy"). Some pharma apps may be collecting personal information without adequately notifying users (see "Checking Under the Hood of Pharma Mobile Apps") or may be performing diagnostic calculations using undocumented and uncertified algorithms (see "FDA Promises Still More Guidance! This Time It's Mobile. Janssen's Psoriasis iPhone App May Need It").

The authors of the study cited above looked at 4 apps that are designed to help consumers evaluate photographs of skin lesions and provide the user with feedback about the likelihood of malignancy. Three of the apps studied used an automated analysis algorithm to evaluate images. These apps were free or very inexpensive (e.g., $4.99) and allowed unlimited evaluations at no additional charge. The fourth app required images to be sent to a board-certified dermatologist for evaluation and an assessment is returned to the user within 24 hours (for a fee).

Although the authors did not identify the apps, I used one of their search terms ("skin cancer") in the Apple App Store and quickly found two representative apps, which are displayed in the image below:

The authors used the apps to evaluate photos of 60 melanoma and 128 benign control lesions. It's no surprise that human experts were much better at correctly evaluating skin lesions. Whereas the 3 apps that used algorithms evaluated 30% or more of melanomas as "unconcerning," the remote dermatologist app incorrectly evaluated only 1 of the 53 melanomas as typical (ie, benign).

The authors note that "more than 13,000 health care applications marketed to consumers are available in the largest online application store alone... Two-thirds of physicians use smartphone applications in their practice. Some [my emphasis] of these applications have been evaluated in the peer reviewed literature,...However, this type of evaluation is not common for applications marketed directly to consumers," noted the authors. And I wouldn't say this type of evaluation is "common" either for apps marketed directly to physicians by pharma.

The FDA may regulate some medical apps, but it is unclear which apps will be subject to regulation and what the regulatory review will entail. Sooner or later, IMHO, Congress is going to be investigating mobile health apps to see if more stringent regulations are required.

IMHO, the pharmaceutical industry should differentiate itself from "wild west" apps developers by being pro-active in issuing Guidelines for Mobile Health Apps Developed by the Pharmaceutical Industry in much the same manner as it developed other self-regulatory guidelines such as PhRMA's DTC Guiding Principles and Code on Interactions With Healthcare Professionals.

What Do You Think?

The Regulation of Pharma Health Apps Survey asks whether or not you agree that it is in the drug industry's best interest to police itself and develop best practices or self-regulatory guidelines for developing trustworthy health/medical apps for consumers and physicians.

It also asks if you agree or disagree with the following specific suggestions:

• Apps must include full disclosure regarding the company that has created the app or the sponsoring pharma company. This includes contact information. BRANDED apps MUST include ISI (important safety information) up front in an easily accessible manner (e.g., on start-up screen).
• Apps that are BRANDED (i.e., mention drug brand names) must be available ONLY from the appropriate U.S. app site (e.g., Apple App Store) even if all the FDA-required ISI (important safety information) is included.
• Apps intended to be used by healthcare professionals in the U.S. must be HIPAA compliant.
• If an app collects personal information, it should include a privacy policy that explains how such data is protected (security), who owns the data, how users can access the data, where data is stored (on device or on remote web site) and instructions for opting out of data collection.
• The pharmaceutical industry has to police itself with regard to development of all health apps regardless of what regulations FDA may impose.
• The app should include appropriate disclaimers and terms of use that the user MUST agree to before the app will run.
• All health/medical apps should be certified by third parties such as Happtique.
• If an app relies on algorithms or formulas, it must be validated through rigorous testing and documentation to ensure it works properly (i.e,. calculations are correct).

Please take a few minutes to respond to this survey here. The summary of results will be published in a future issue of Pharma Marketing News. You may remain anonymous or you may provide your name and contact information if you wish to be quoted in the published summary. If so, I may contact you for more details and allow you to review your responses prior to publication.