Robotics, artificial intelligence and distributed ledgers in surgery: data is key!

‘Robotic Surgery’ is increasingly debated in surgical circles. The reality is, however, that we are nowhere near the era of true robotic surgery, and what we are actually debating are advanced laparoscopic devices or ‘tele-manipulators’. Whichever English definition one chooses for the term ‘robot’, the consistent qualification is a machine that is able to undertake tasks ‘automatically’, whether this be programmed or independently. The current iterations are robotic platforms which do not fulfil this most basic of criteria to be called robots.

The path to real robotic surgery involves artificial intelligence (AI) techniques so that a machine is able to recognise, process, predict and ultimately execute a task, either under supervision or unsupervised from human control. And to do this, the algorithms require data….and lots of it! To collect such masses of data means we must be able to share information in a far more honest, transparent and collaborative manner, but this brings about the understandable concerns of misuse of confidential patient information. If we are able to surmount these initial barriers, we are next faced with the problem that the information we have is more often than not, unstructured, heterogenous and in the wrong format to be used meaningfully. This applies to national longitudinal datasets such as those derived from socialised healthcare systems such as the National Health Service (NHS) in the UK. Formal establishment of multidisciplinary data science groups within healthcare institutions is encouraging us to present patient data in a format which can be used more effectively. Nevertheless, organising this data into ‘usable’ machine learning data remains a huge task.

A surge in the electronics industry over the past few years has driven an incredible growth in our computational capabilities [1]. At the current rate of progress and squeezing of transistors into integrated circuits, scientists predict that by 2050 the size of components needed to build a transistor would have to be smaller than an atom. A gaming system today has almost the same computer power as those that navigated the first satellite launch into orbit and the moon landings; and it is this rapid growth of computational capability that has given rise to the recent rebirth of AI and specifically techniques in AI involving ‘deep learning’ or ‘deep neural networks’—machine learning techniques based on learning data representation rather than task-specific algorithms. While the foundational concepts in deep neural networks have been around since the 1980s, it is current computers and especially architectures with hugely parallel design originally designed for gaming, that are unlocking true AI capabilities.

Alongside computational power, the past decade has seen an exponential growth in digital data; and for AI, especially using deep learning techniques, the access to data is one of the keys. The neural network architecture takes examples of matched data that demonstrates an input and desired output, and it number-crunches through trillions of possibilities to estimate the parameters of its architecture that best estimate the input to output mapping. Effectively training this system requires large volumes of data that captures almost all the different possibilities that can be observed. Examples of how AI can benefit healthcare include:

Patient records Electronic records that contain the history of patient treatments/conditions may be used to find patterns that link the risk of condition incidence to historical patterns. This makes the assumption that some patterns may not be easily perceived by human observers. Potentially, the data can be used to choose optimal treatment choice in a patient specific manner [2]. The structure and density of the records data is a significant practical challenge but this will become alleviated by modern integrated software for record keeping. Additionally, AI algorithms may be used to transcribe digital scans of hand written notes.

Diagnostics We have already seen huge strides in healthcare diagnostics in particular radiology [3]. Digitization of both radiology and pathology have led to exciting developments in automated image analysis which have been further accelerated by AI. Radiology and pathology both require identification of specific patterns and are ideal for deep learning techniques whereby the more images are seen and analysed, the more accurate the machine becomes. A machine is invariably better than a human at extracting information from an image leaving the purpose of the radiologist or pathologist to provide medical context to that information. For example, machine learning has already been shown to be more accurate than humans at predicting non-small cell lung cancer [4] whilst Bejnordi et al. developed an algorithm to detect sentinel lymph nodes in histopathology sections of breast cancer patients, which did better than a panel of 11 pathologists [5].

Intraoperative sensors The modern integrated operating room has ample sensors for monitoring patient vitals, cameras to monitor the theatre, cameras inside the patient, additional imaging modalities and sensors to monitor activity and/or function [6]. Embedded within this rich data could be indicators about surgical performance, operational risk, outcome precursors or other signs of higher level understanding. Surgical data science is an emerging field looking at harnessing the potential of this type of surgical data [7] and most modern companies are now beginning to utilise such data analysis to better position their products. For example, a surgeon will be able to logon to his or her surgical platform whether that be a robotic console or laparoscopic hub in the theatre at the beginning of a procedure and enter a personalised online portal containing vast quantities of their surgical performance data. The console will be able to recognise the surgeon and display historical data including performance metrics. By linking this data across an institution or geographical area, one could compare performance metrics between surgeons and/or ‘learn’ from peers and their performance. Such application of performance data would have the inevitable effect of driving standards upwards.

These data sources combined with AI algorithms to either automate data analysis and synthesise higher level information, or to elucidate new inference from the underlying data, can have a significant impact to the future of surgery.

The concept of computer aided diagnosis (CAD) has gained popularity in endoscopy to increase polyp detection rates [8]. The premise is to provide real-time guidance rather than assessment of static images. Reports combining optical biopsy and narrow band imaging (NBI) [9] with CAD have shown promise and as process speed increases we will see more real-time application. CAD must be able to accurately detect and characterise polyps. A recent study by Urban et al. analysed 8641 hand-selected colonoscopy images from over 2000 patients consisting of 4088 unique polyp images and 4553 images without polyps [10]. Using deep learning techniques they detected polyps with a cross-validation accuracy of 96.4% and area under the receiver operating characteristic curve (ROC-AUC) value of 0.991. Similarly, Misawa et al. performed a study of 73 colonoscopy video sequences running from caecal intubation to withdrawal including 155 polyps [11]. Each frame containing a polyp was retrospectively annotated by two expert endoscopists acting as the reference for polyp presence. The system achieved a sensitivity, specificity and accuracy of 90.0%, 63.3% and 76.5%, respectively, on an image-frame based analysis using a test set of 135 short videos. However, CAD technology must overcome a number of challenges before it enters routine clinical practice. The key stages for the implementation of CAD technology into routine colonoscopy have been highlighted by Mori et al.—feasibility studies, clinical trials, and regulatory approval [12].

We must also consider how data are acquired, stored, validated and released for analysis. At a time when privacy laws are getting tougher and there is a climate of scepticism regarding data security, it would be foolhardy to ignore the challenges faced by secure handling of patient data.

Distributed ledger technology (DLT) has been touted as a potential solution to address these issues. Distributed ledgers are a superset of technologies which include the more well-known blockchain (eg Bitcoin) and newer variants such as Directed Acyclic Graphs (DAGs eg IOTA). To understand their value proposition, we must look more closely at some of the potential problems we face within data protection and how DLT may help.

Recent controversies [13] have raised public awareness regarding centralised data collection and control, and how algorithms working on this data can potentially exert significant untoward influence. There is growing public concern regarding an emerging data oligopoly which may affect the future of society as a whole, as reflected in a report [14] from the Nesta group, for the DECODE project [15]. Healthcare data and records are not immune from this and there is understandable concern about misuse of personal patient data.

Distributed ledgers promise a more decentralised approach to data management as no single entity controls the network. Control may be completely decentralised in a ‘permissionless’ ledger or be distributed amongst many trusted parties in a ‘permissioned’ ledger. The latter approach offers tighter control and higher scalability but is more prone to the oligopoly control. If data will serve as the ‘ground truth’ for the algorithms, then it is imperative that its provenance and integrity is maintained. We must ensure that data is not corrupted unintentionally, or indeed tampered with intentionally, as this could have potentially devastating effects. Distributed ledgers provide an immutable tamper-proof history of events. This history can be made either transparent in the case of public data or obfuscated by storing only the mathematical hash of the data, when dealing with private data [16].

There is a natural tension between the need for free sharing of data to improve care, versus the right to privacy and the need for data-sharing consent. The MedRec project [17] looks at how this could be managed in a decentralised manner using DLT. The number of devices collecting data is expected to rise exponentially. By 2020, it is envisaged that there may be up to 20 billion connected devices [18]. Many devices that exist “in the wild” outside the normal clinical environment (such as activity trackers), may provide valuable data for the clinician and the clinical algorithms. The identity of, and the data from these devices may be maintained in either a centralised process controlled by a small group of identity providers, or in a more decentralised fashion. DLT may provide a way to handle this data safely in a decentralised manner [19].

So how can this all be applied to surgery itself? If one reconsiders the route to developing accurate algorithms, it is reliant on providing sufficient data samples which have been responsibly acquired, stored and processed, to account for all eventualities during a surgical procedure—computer processing power and ethical data management are integral to this. The specific difficulties in applying this concept to a surgical procedure are that there many ‘moving parts’ unlike radiology or pathology where there is commonly a single image to analyse. Acquiring the requisite number of data in surgery is in itself a challenge but beyond that, the video data must be ‘coded’ or ordered in a way that can expedite analysis. This often means ‘segmenting’ the video into smaller steps and standardizing the actual procedure to facilitate learning. Computer vision techniques are currently being tested and developed to validate various surgical procedures. A recent report of application in bariatric and colorectal surgery at Imperial College and University College London, respectively [20], described how analysis of video footage of recorded surgical procedures has allowed for future ‘prediction’ of procedural steps and to notify the surgeon of deviation from routine. Furthermore, many large data firms are interested in acquiring repositories of video content to enable development of such computer vision techniques but perhaps unlike the diagnostics, analysis of surgery remains firmly in the hands of the surgeon. This is an important issue as the process of developing effective AI in surgery will also lead to standardisation of technique and optimal outcomes which can only be of benefit for the patient.

Once we are able to share data responsibly and without the genuine concerns of misuse through data ledgers, we will be able to harness the expanding computational power which surrounds us to build accurate AI algorithms and finally develop true robotic surgery. Remember—data is key and surgeons must be fully engaged to ensure it is used wisely.