The recent work by the École Polytechnique Fédérale de Lausanne proposes an approach based on artificial neural networks to optimize image processing within the field of retinal prosthesis.
Discussing retinal prosthesis and artificial images brings us to the research field of neuroengineering, which applies engineering techniques and methodologies to study the functions of the central and peripheral nervous system (including the visual system), as well as to develop solutions aimed at repairing or replacing damaged parts, employing – precisely – artificial devices such as robotic systems or prostheses [source: “Neuromodulation” – ScienceDirect].
One of the challenges of neuroengineering has always been precisely to do with sensory prosthesis and, more specifically, “the construction of the process known as “artificial sensory coding“, which plays a crucial role in those prostheses designed to restore sensory perception in patients with disabilities“, as noted by a team of scientists from the École Polytechnique Fédérale de Lausanne (EPFL), in “An actor-model framework for visual sensory encoding“, a study published in Nature Communications on 27 January 2024. The greatest difficulty – the researchers from the Swiss university explain – lies in “determining the appropriate artificial input to a sensory system deputed to reproduce the desired perception“. But let’s go deeper, taking a small step back.
TAKEAWAYS
Understanding artificial sensory encoding
In the human being, the sensory organs such as the sight, the hearing or the touch receive information (input) from the external environment and translate it into ‘neural signals‘ that are processed by the brain, in a process known as ‘sensory coding‘.
In sensory prosthesis – including the retinal prosthesis and the acoustic prosthesis, to name but the best-known examples – inputs come from sensors and not from the environment and are converted into ‘artificial stimulation parameters‘, in such a way as to replace sensory coding natural with artificial coding.
In such prostheses, however,” recalls the ETH Lausanne team, “the input range is much shorter than in biological sensory systems. For example, the number of electrodes in the neuroprosthesis is usually several orders of magnitude smaller than the number of sensory neurons. Therefore, artificial sensory coding is a form of dimensionality reduction‘.
What does this mean? It means that the high-dimensional information from the sensors in the prosthesis is reduced to a few stimulation parameters, so that a few electrodes can transmit the information in a format that the human brain can actually understand and encode.
In this regard, the working group cites the example of auditory coding in hearing aids, where sound is converted into electrical stimulation of certain frequency regions within the auditory nerve, enabling those with deafness to perceive sounds. Similarly, prosthetic limbs provide amputees with tactile feedback to improve manual dexterity.
Retinal prosthesis and artificial image encoding: The challenges in replicating the function of the biological retina
Artificial sensory coding also has a pivotal task in the retinal prosthesis, transmuting high-resolution images taken by an external camera ‘into a spatio-temporal pattern of artificial stimuli‘. The study team specifies, however:
“The visual coding of artificial images is crucial in improving the perception of patients with retinal diseases, but it is not simple. In the retina, information flows from about 120 million photoreceptors to about 1.2 million retinal ganglion cells divided into different classes, which project to different brain nuclei, including the lateral geniculate nucleus, and then to the visual areas, where further image processing takes place. The complexity of the visual information process therefore requires advanced coding strategies to ensure effective stimulation of visual neurons and lead to useful computer vision.”
This is to say that the retina is by no means a simple ‘camera’ (and cannot be replaced by it), but performs a function with intricate and non-linear mechanisms in the processing of visual stimuli. And reproducing it is not an easy job.
“The performance of the visual processing mechanism embedded in retinal prostheses depends on a full understanding of how the retina works, as well as on the evolution of computer vision models” warn the authors of a research paper made public in the Journal of Neural Engineering -“Artificial intelligence techniques for retinal prostheses: a comprehensive review and future direction“-in February 2023. Let us see how such performance can be achieved.
The Argus II Retinal Prosthetic System
To date, the most widely implanted retinal prosthesis, designed exclusively for the treatment of severe cases of retinitis pigmentosa and approved by the US Food and Drug Administration (2013), remains the Argus II, the brainchild of American scientist Mark Humayun, who developed it based on an invention by engineer Robert Greenberg.
By January 2021, there were 300 implanted models worldwide, of which 55 in Italy [source: Saint Camillus International University of Health and Medical Sciences – UniCamillus]. This is the latest data available. In fact, on 15 February 2022, an article appeared in IEEE Spectrum – “Their bionic eyes are now obsolete and unsupported” – warning of the discontinuation of production of Argus II and, indeed, of the cessation of care for patients still wearing the device.
From a technical point of view, this prosthetic system is designed to replace – in the centre of the retina, right on the macula – the function of photoreceptors degenerated as a result of retinitis pigmentosa.
Its operation involves a micro-camera connected to a pair of visor glasses, whose images are converted by the system into a series of electrical impulses that travel from the optic nerve to the visual area of the brain.
The visual model it is able to recreate is partial, made up of areas of light and shadow, which, however, help patients to identify obstacles, recognise zebra crossings and distinguish the silhouette of people [source: “Argus II: The ‘Bionic Eye’ An Incredible Breakthrough for People with Retinitis Pigmentosa” – American Academy of Ophthalmology].
Retinal prosthesis and artificial imaging: new approaches to sensory coding
On the subject of retinal prostheses, researchers at the École Polytechnique Fédérale de Lausanne point out that most of the devices designed in the last decade have been trained to encode artificial images using very simple techniques, including – for example – pixel averaging.
In the Argus II itself, they point out, ‘pixel averaging was used in conjunction with video filters, in order to subsample the images captured by the camera‘. In essence, the prosthetic system reduces the number of pixels in the images to be transmitted by compressing them.
More recent studies, on the other hand, have led to the definition of new approaches to the encoding of artificial images, through the development of new processing algorithms, some of which involve the detection of objects within the scene, the edge detection and the content-based ‘retargeting‘ method. As the authors illustrate:
“… in general, such methods aim to reduce the complexity of the image, highlighting only the contents and features that are useful to the wearer. For example, edge detection can identify a discontinuity of brightness within the image, which helps to identify the outline of an object. By reducing the amount of information, the patient could perceive the environment better. However, despite the progress, the coding potential of these algorithms is still limited, as they still cannot fully approximate the reproduction of the complexity of the human visual system, in which information processing is a two-step process: from the photoreceptors to the retinal ganglion cells“
In short, since the demise of the Argus II model, the process of encoding artificial images into prostheses designed to restore visual perception in those with retinal diseases seems to be failing to make further decisive steps towards the biological retina model.
Convolutional neural networks in the encoding of artificial visual information
The Lausanne Polytechnic University team, in its work on retinal prosthesis and artificial image coding, opted for a convolutional neural network-based approach, a type of artificial neural network most commonly used in the analysis of video data.
The reason for this choice,’ he explains, ‘lies in the potential of Convolutional Neural Nertworks (CNN) applied to the definition of a complex retinal pattern, closer to the characteristics of the biological one, thus improving the artificial visual coding of the prosthesis.
In detail, the study involved the development of a machine-learning algorithmto fine-tune the ‘acor-model framework‘ (‘acor-model framework‘) – as defined by the authors – specifically designed ‘to learn to undersample images‘ based on the weaknesses relative to contrast identified in the video data captured by the micro-camera embedded in the outer part of the retinal prosthesis.
To explain the function of the framework, the example given by the researchers comes from the use of Photoshop in calibrating the light and dark parts (the contrast, in fact) within images, in order to make them as usable as possible by the human eye. In brief:
“The framework acts as a digital twin of the artificial retina. The latter is first trained to receive a high-resolution image and to emit a binary neural code as similar as possible to the neural code generated by a biological retina. Theactor-model framework is, then, trained to subsample a high-resolution image, capable of eliciting in the artificial retina a neural code as close as possible to that produced by the biological retina in response to the original image”
Using this methodological scheme, the team tested the images subsampled by their framework on both the digital twin of the artificial retina and on the retina explanted from a deceased animal (ex vivo experimentation), demonstrating that the approach adopted succeeded in producing “images that elicit a neuronal response more similar to the response of the original image than an image generated by a computational approach without learning as pixel averaging“.
Glimpses of Futures
On the subject of retinal prosthesis and artificial images, the study described presents a novel approach to sensory coding, in which artificial neural networks and machine learning techniques play a strategic role.
In the years to come, what its authors expect from this approach is certainly the development of even more accurate and effective image coding methods. Indeed – in their opinion – the actor-model framework “could guide future artificial image coding strategies for visual prosthesis”.
But – even more importantly – its use could be of interest in other areas of artificial sensory coding, for example in reference to sensory prostheses such as cochlear (for the hearing impaired) or limb prostheses.
We now try to anticipate future scenarios, analysing – thanks to the STEPS matrix – the impacts that the evolution of the described framework for encoding images in retinal prosthesis could have from a social, technological, economic, political and sustainability perspective.
S – SOCIAL: in a future scenario that could see the actor-model framework validated by clinical studies, approved at an institutional level and, therefore, applicable to retinal prosthesis that can be implanted in humans (just as happened, as of 2013, with the Argus II model), the result would be improved visual perception by patients suffering from severe retinal diseases, enabling them to acquire greater autonomy and safety in daily life.
T – TECHNOLOGY: the evolution of the method proposed by the Lausanne Polytechnic Institute, in addition to the development of an artificial neural network even deeper than the one adopted, could in future also rely on more powerful techniques for compressing images, so that the artificial retinal system can process several visual dimensions simultaneously, just as happens in the biological retina.
E – ECONOMY: from an economic point of view, a very broad reflection opens up, related to the cost of the Argus II retinal prosthesis, produced in the USA (about $150,000, i.e. about 140,000 Euro, excluding the cost of surgery under general anaesthesia and the cost of training to learn how to use it). In a future in which we imagine implantable retinal prostheses with the actor-model framework on board, the healthcare system of each country will have to bear part or all of the expected costs, especially in cases where the socio-economic level of the patients does not allow them to bear them.
P – POLITICAL: The future impact, from a political point of view, given the evolution of the framework for image coding in retinal prostheses, would be on the need for the institutions of each country to have a specific regulation on the subject, aimed at establishing – for the manufacturers – the safety requirements, from a health point of view, of implantable retinal systems.
S – SUSTAINABILITY: a retinal prosthesis such as the one described by the Swiss team, intended for those whose retina is damaged by an established disease, should be available and accessible to all, globally, and not be a luxury of the few due to the overall costs. The negative impact, as far as social sustainability is concerned, could, in the future, be precisely related to the unfairness of a private rather than a public healthcare provision.
