Large scale AI in digital pathology without the heavy lifting

Pictured: Tissue sample stained with H&E and immunofluorescence (IF) markers side by side.

This is a guest post by Moritz Widmaier, product manager; Martin Shulze, principal consultant for technology and IT; and Florian Leiss, vice president (VP) of product strategy and corporate development, at Ultivue.

Pathology is currently undergoing a transformation. While microscopes still dominate many workflows, digital pathology combined with artificial intelligence (AI) is disrupting the space. Together they unlock the pathway to precision medicine and will be among our strongest weapons to fight cancer. Pathologists have been looking at morphological patterns in patients’ tissue sections highlighted by Hematoxylin and Eosin (H&E) staining for more than a century. They accumulated a comprehensive body of knowledge on how combined insights from tissue and cell morphologies characterize disease and guide treatment selection. AI tools can complement expert assessment with quantitative measurements to enable data-driven medicine. AI companies have formed on the idea that AI can complement the expert assessment of H&E slides by a pathologist.

Ultivue is one such healthcare technology (HealthTech) company that provides high-quality multiplex immunofluorescence assays and large-scale, AI-based computational pathology—built on Amazon Web Services (Amazon Web Services).

Tissue analysis in color

With the advent of immune therapy, a more granular characterization of the immune system (and tumor cells) has become necessary to support clinical pathology and diagnostics.

At a high level, our immune system comprises multiple different cell types (T cell, B cell, etc.), which are difficult to distinguish in H&E slides. To complicate things, most of these cell types can be subdivided into a vast number of “specializations” and states of activities that cannot be distinguished from H&E slides anymore. Neither pathologists nor AI can reveal the required information from H&E sections since it was not captured when the tissue section was stained and digitized. Trying to understand the tumor-immuno microenvironment through the eyes of ‘Digital Pathology’ (i.e. only looking at H&E stained sections) can be compared to using black-and-white photography to spot yellow tulips in a red tulip field.

Immunofluorescence is the color photography equivalent of pathology. It allows us to see many more cell phenotypes by adding highly specific staining channels, and with the help of AI, their spatial relationships, or spatial biology.

Ultivue combines expertise in the development of high-quality multiplex immunofluorescence assays and AI-based computational pathology. We aim to deeply characterize immune cells and tumor cell and their spatial relationships in the context of intact tissue, using AI to analyze tissue images at scale. This presents multiple challenges. Our multiplex immunofluorescence images can have five billion pixels and more than five, sometimes 13 channels. These images can get as large as 100 GB. Images at the scale of 10 to 100 GB present a different order of magnitude to what many systems are built for – including systems built to analyze H&E images in pathology. Experts developing these tissue assays and understanding the scientific intricacies in the rich image data mostly aren’t AI engineers and cloud architects. They want algorithms to support the interpretation of the images to generate results quickly, without having to write or read code.

Handling over 10 GB images on Amazon Web Services

Amazon Simple Storage Service ( Amazon S3 ) provides a cost-effective and scalable solution for storing this data, and in combination with Amazon Web Services Storage Gateway , traditional block-storage-like access is achievable. Amazon Web Services Storage Gateway is an important part of our solution, as many of the common tools that pathologists and scientists rely on would cease to function without it. However, these commonly used tools that run on a Windows Desktop do not scale very well. That’s where Amazon SageMaker comes in.

Figure 1. The Ultivue architecture using Amazon S3, Amazon Web Services Storage Gateway, Amazon SageMaker, and Amazon SageMaker Studio as key components.

For this solution, we use Amazon S3 with Intelligent-Tiering to store data in a cost-effective manner with low-latency access. Storage Gateway provides file shares via the SMB network file system that Windows clients can use. Amazon WorkSpaces provides a pre-configured desktop working environments to our users, including the Storage Gateway-backed network drives.

Amazon SageMaker Studio enables cloud working environments for AI engineers, as well as pathologists. We use Amazon SageMaker’s processing, training, and inference APIs to analyze images at any scale.

Amazon SageMaker Studio as a user interface for non-technical users

When handling data at the scale that Ultivue does, the need for utilizing the scalability of cloud services is omnipresent. However, the bio-technology space requires medical and biological experts that cannot be experts in software and IT at the same time. Our user interfaces must simplify the use of highly scalable compute services, such as Amazon SageMaker, and at the same time, offer flexibility to a wide range of users, like AI engineers.

Utilizing custom images and lifecycle configurations allows Ultivue to provide an internal app-store-like experience on top of the highly flexible Amazon SageMaker Studio interface.

Figure 2. Screenshots of the user experience.

In the past, image analysis experts and AI engineers worked with software engineers to integrate algorithms into custom processing environments and connect them to different data sources. Now, processing at scale and data pipelines are reduced to API calls, and image analysis experts can make their algorithms available to pathologists without any intermediary needed.

From pixel to object

Without image analysis, information-rich multi-channel images are just images. Humans, even trained experts, are limited by–in the best case–the presence of only three-color receptors in our retina. Consequently, it becomes increasingly difficult to visually process the information coming from the 5-13 channels that are typically used with Ultivue technology, let alone considering the interactions of potentially dozens, if not a hundred, different cell types across millions of cells present in a tissue section.

AI-based image analysis gives us the opportunity to classify millions of cells and interrogate their frequency and spatial relationships in a data-driven way. This can help find out which specific immune cells need to talk to other immune cells or interact with a subset of tumor cells to predict therapy outcome, and more.

Visit Ultivue to learn more about this solution.

Learn more about how healthcare and life science organizations use Amazon Web Services at Amazon Web Services for Health . Do you want to learn more about how to use Amazon Web Services for digital pathology and other healthcare solutions? Contact the Amazon Web Services Public Sector Team here .

Pictured: Classified cells overlayed on IF image as image analysis detected them.

Read more about Amazon Web Services for healthcare:

• How Digithurst and Telepaxx built a secure and scalable radiology solution chain using Amazon Web Services
• AMILI helps advance precision medicine by building microbiome library on Amazon Web Services
• Helping prevent sudden cardiac arrest in young athletes with AI
• How KHUH built a long-term storage solution for medical image data with Amazon Web Services
• Solving medical mysteries in the Amazon Web Services Cloud: Medical data-sharing innovation through the Undiagnosed Diseases Network

Subscribe to the Amazon Web Services Public Sector Blog newsletter to get the latest in Amazon Web Services tools, solutions, and innovations from the public sector delivered to your inbox, or contact us .

Please take a few minutes to share insights regarding your experience with the Amazon Web Services Public Sector Blog in this survey , and we’ll use feedback from the survey to create more content aligned with the preferences of our readers.