HealthTrack AI - Predictive Healthcare System
Niraj Sthapit
Introduction
Artificial intelligence is transforming healthcare at an unprecedented pace, offering tools that can identify disease risks earlier, optimise clinical pathways, and support overstretched health services. HealthTrack AI represents one such application: a predictive system deployed within NHS Trusts that combines medical records, wearable technology data, genetic information, and smartphone lifestyle data to forecast a patient's risk of developing Type 2 diabetes up to 18 months before diagnosis. With an 87% accuracy rate across a 50,000-patient trial cohort, its headline results are striking. Yet, as computing professionals and future technologists, we are obliged to look beyond the headline. An 87% accuracy rate means 13% of cases are misclassified — and in a system operating at national scale within a healthcare context, those numbers represent thousands of real patients. Our group explored HealthTrack AI through five focused analytical lenses: Responsible AI and Ethics, Cybersecurity and Privacy, Policy and Regulation, Social Impact, and Governance and Accountability. Together, these perspectives reveal a system that, while technically sophisticated, raises profound concerns about bias, consent, legal compliance, social equity, and accountability. This article presents our findings, aiming to contribute to the critical conversation about how AI should be deployed responsibly in high-stakes public services.
Responsible AI and Ethics
The most urgent ethical issue of HealthTrack AI is the false positive rate of Black and Asian patients being 15 percent higher than that of the general population. This is no statistical exception. It is an expression of structural algorithmic bias (Obermeyer et al., 2019), in which machine learning models trained on historically biased datasets replicate and intensify existing inequalities at scale. The direct clinical implications are unnecessary referrals, unjustified medical anxiety, and the undermining of trust in NHS services among already underserved communities. These are the very communities already burdened with disproportionately high rates of Type 2 diabetes, meaning the system is failing most severely where precise prediction matters most.
The mechanism is well understood. Where HealthTrack AI's training data disproportionately reflects white patient health trajectories, the model learns a version of normal that is not well calibrated for patients of other ethnic backgrounds. Systematic gaps in ethnicity data completeness have been documented in UK primary care databases (Mathur et al., 2022), and Rajpurkar et al. (2022) demonstrated that such gaps consistently worsen model performance for underrepresented groups. The issue is not the algorithm itself, but the uncritical use of historically biased data as a neutral training ground.
Compounding this is the opacity of the system. Pilot clinicians stated they could not explain why certain patients received certain risk scores. This is not a technical inconvenience. It represents a fundamental incompatibility with evidence-based medicine. Rudin (2019, p.206) argues that in high stakes domains, the time for black box models is over. Explainable AI frameworks such as SHAP and LIME already exist to address this, allowing clinicians to understand which variables drove a specific risk score. Their absence from HealthTrack AI appears to be a design decision that favours predictive performance over clinical accountability.
The BCS Code of Conduct (2022) is unambiguous. Principle 1 requires members to have due regard for the public interest, act with integrity, and avoid actions they know will cause harm. The IEEE/ACM Software Engineering Code of Ethics (2018) reinforces this, requiring practitioners to be fair and avoid discrimination. These obligations apply to everyone in the development and deployment chain. The BCS Code does not permit a defence of following instructions. Professional judgment must be exercised at every level. Responsible AI in healthcare is achievable, but only when ethical rigour is built in from the start, not retrofitted after harm has occurred.
Cybersecurity and Privacy
HealthTrack AI collects some of the most sensitive personal data imaginable: NHS medical histories, genetic test results, continuous biometric monitoring, GPS location tracking, sleep patterns, and behavioural data drawn from smartphones. Fifty thousand patients handed over this information trusting it would be used to improve their healthcare. What many did not appreciate is how inadequately that trust is protected.
The consent process is where the framework begins to fail. Under UK GDPR, consent must be informed, specific, and freely given (Solove, 2013). In practice, patients were presented with a 12-page legal document in busy clinical settings, already anxious about their health and subject to the implicit pressure of a medical environment. Signing a form is not the same as understanding what you have agreed to, and that distinction carries significant legal weight.
There is also a largely unacknowledged infrastructure risk. HealthTrack AI stores patient data on Amazon Web Services servers in Ireland. Post-Brexit, cross-border data flows between the UK and EU depend on an adequacy arrangement that is temporary and politically fragile (Kuner, 2022). If that arrangement breaks down, NHS Trusts could find themselves in immediate legal jeopardy with no contingency in place.
The most serious concern is the reported interest from insurance companies in accessing patient risk scores to adjust premiums. This would be unlawful. Data collected under NHS consent cannot be repurposed for commercial use without fresh explicit consent that was never sought (NHS England, 2023). If insurers can use algorithmic risk scores to price or restrict coverage, the foundational NHS principle of universal access begins to erode. This is acutely concerning given the known bias in the system: patients from ethnic minority backgrounds, already disproportionately flagged as high risk, could face higher premiums as a direct consequence of a flawed algorithm. These are not isolated oversights. They are deep structural failures that require enforceable legal obligations and a genuine institutional commitment to placing patient welfare first.
Policy and Regulation
The UK Government's AI Regulation White Paper (DSIT, 2023) adopts a pro-innovation approach that deliberately avoids prescriptive legislation in favour of sector-led, context-specific guidance. The rationale is defensible in principle: rigid frameworks risk stifling innovation at an early stage of technological development, and domain-specific regulators may develop more proportionate rules. For general-purpose commercial AI, this flexibility has some merit.
Healthcare AI is a different matter entirely. The EU Artificial Intelligence Act (European Parliament, 2024) classifies AI systems that influence clinical diagnosis or treatment as high-risk, subjecting them to mandatory pre-deployment conformity assessments, compulsory human oversight, and enforceable transparency obligations. Under the UK framework, no equivalent requirements apply. HealthTrack AI can be integrated into NHS clinical pathways without independent pre-deployment review, without mandatory disclosure of known performance disparities, and without any binding obligation to demonstrate safety across the full population it serves. Developers operate largely on a self-regulatory basis under voluntary schemes with no legal force (DSIT, 2023).
The practical consequence is significant. HealthTrack AI, as currently configured, would require substantial modification before it could lawfully operate under EU rules. The EU model requires high-risk AI systems to prove safety before deployment. The UK model relies on detecting and remedying harm after it has occurred. In a medical context, this sequencing is deeply problematic: algorithmic errors cause immediate patient harm, and post hoc remediation cannot undo that damage.
This amounts to regulatory arbitrage in which the costs of inadequate design are externalised onto patients while commercial benefits accrue to developers. The absence of mandatory pre-deployment requirements is a conscious policy decision that prioritises market freedom over patient protection (Cath et al., 2018). In a publicly funded healthcare system, that prioritisation requires urgent reconsideration.
Social Impact
HealthTrack AI does not operate in a vacuum. It enters a healthcare system already marked by entrenched structural inequality, where health outcomes correlate strongly with socioeconomic status and ethnicity (Marmot, 2020). Type 2 diabetes illustrates this clearly: in the UK, people from South Asian communities are diagnosed three to five times more frequently than white British people, and African and Caribbean communities face significantly elevated rates (Cronjé et al., 2023). A predictive AI system deployed in this context should, in principle, offer its greatest benefit to the communities most at risk. The evidence suggests the opposite is occurring.
The documented 15% higher false positive rate for Black and Asian patients means the system produces its least reliable outputs for precisely those who need accurate prediction most (Haider et al., 2024). False positives are not clinically benign. A patient incorrectly identified as high risk faces unnecessary anxiety, unwarranted lifestyle interventions, and potential overmedicalisation. For communities that already experience institutional mistrust rooted in historical discrimination (Armstrong et al., 2007), an unreliable algorithmic verdict can deepen that mistrust and deter future engagement with preventative care.
The system's reliance on wearable devices introduces a further concern. Continuous collection of GPS location data, sleep patterns, and daily activity logs goes well beyond clinical measurement. For communities with historical reasons to be wary of institutional data collection, participation may not feel genuinely voluntary even where formal consent has technically been obtained. Engagement under institutional pressure does not constitute free choice, and healthcare technology that cannot be freely refused risks becoming a source of harm rather than support.
The social impact of HealthTrack AI ultimately depends on whether its known failures are addressed before scale deployment. A system that is technically impressive but systematically less accurate for already disadvantaged communities does not reduce health inequality. It entrenches it.
Governance and Accountability
When an AI system causes harm in a healthcare setting, establishing responsibility is rarely straightforward. This is the problem of many hands (Nissenbaum, 1994): moral agency is distributed across developers, engineers, procurement officers, clinicians, and institutions, diluting accountability to the point where no single party bears clear responsibility for collectively foreseeable and avoidable harm.
HealthTrack AI illustrates this precisely. Developers cite adherence to specification. NHS Trusts point to vendor assurances. Clinicians note they used an institutionally approved tool. The vendor references signed consent forms. Each position is individually defensible, but together they produce an accountability vacuum in which a system with documented racial bias continues operating at clinical scale without meaningful oversight.
NHS Trusts are data controllers under UK GDPR (EU GDPR, 2016) and bear primary statutory responsibility for compliance. Article 35 mandates a Data Protection Impact Assessment before any processing likely to pose a high risk to individuals' rights. The ICO identifies three clear triggers: large-scale processing of health data, profiling with significant effects, and deployment of new technologies. HealthTrack AI satisfies all three (ICO, 2021). A properly conducted DPIA should have identified the racial bias in false positive rates, the inadequacy of consent, the data sovereignty risks of cloud storage, and the purpose limitation risks posed by insurance industry interest. The public emergence of these issues without evidence of prior DPIA documentation indicates either a direct legal violation or that identified risks were disregarded. Either constitutes a serious governance failure.
Beyond legal compliance, the BCS Code of Conduct (BCS, 2022) requires members to act honestly and raise concerns when professional standards are compromised. A professional who identified a 15% racial disparity and did not escalate it would be in clear breach. Accountability gaps in AI are structural. Closing them requires enforceable frameworks and a professional culture in which individuals act on what they know.
Conclusion
HealthTrack AI demonstrates the transformative potential of artificial intelligence in preventative healthcare. Predicting Type 2 diabetes risk 18 months in advance could enable earlier interventions, reduce long-term NHS costs, and improve patient outcomes. However, our analysis reveals that technical capability alone is insufficient justification for deployment at scale. The system exhibits a 15% higher false positive rate for Black and Asian patients, groups already disproportionately affected by Type 2 diabetes. This is not a minor statistical quirk; it is a systemic bias with real consequences for individuals' health, wellbeing, and financial circumstances. Consent processes are inadequate, data governance is unclear, and accountability frameworks are insufficiently defined. The UK's flexible AI regulatory approach, while designed to promote innovation, leaves healthcare AI in a regulatory grey zone that prioritises speed over safety. We recommend the following: first, the rollout of HealthTrack AI should be paused until a rigorous independent bias audit is completed. Second, explainable AI tools should be integrated so clinicians and patients can understand and challenge risk scores. Third, consent processes must be simplified, accessible, and genuinely voluntary. Fourth, patient data must not be shared with third parties, including insurers, for commercial purposes. Fifth, a clear governance framework must assign responsibility across developers, NHS Trusts, and regulators. Innovation in healthcare AI is welcome and necessary, but it must be built on a foundation of fairness, transparency, and accountability. Responsible computer science demands nothing less.
This research report has been prepared by students of The Westminster College, Kupandole, as part of their module requirements. Students are required to write a research paper and publish it on their LinkedIn profiles.