Large scale AI in digital pathology without the heavy lifting

by Moritz Widmaier, Martin Schulze, Florian Leiss, Ruben Cardenes, and Razvan Ionasec | on

Pictured: Tissue sample stained with H&E and immunofluorescencemarkers side by side.

Pictured: Tissue sample stained with H&E and immunofluorescence (IF) markers side by side.

This is a guest post by Moritz Widmaier, product manager; Martin Shulze, principal consultant for technology and IT; and Florian Leiss, vice president (VP) of product strategy and corporate development, at Ultivue.

Pathology is currently undergoing a transformation. While microscopes still dominate many workflows, digital pathology combined with artificial intelligence (AI) is disrupting the space. Together they unlock the pathway to precision medicine and will be among our strongest weapons to fight cancer. Pathologists have been looking at morphological patterns in patients’ tissue sections highlighted by Hematoxylin and Eosin (H&E) staining for more than a century. They accumulated a comprehensive body of knowledge on how combined insights from tissue and cell morphologies characterize disease and guide treatment selection. AI tools can complement expert assessment with quantitative measurements to enable data-driven medicine. AI companies have formed on the idea that AI can complement the expert assessment of H&E slides by a pathologist.

Ultivue is one such healthcare technology (HealthTech) company that provides high-quality multiplex immunofluorescence assays and large-scale, AI-based computational pathology—built on Amazon Web Services (Amazon Web Services).

Tissue analysis in color

With the advent of immune therapy, a more granular characterization of the immune system (and tumor cells) has become necessary to support clinical pathology and diagnostics.

At a high level, our immune system comprises multiple different cell types (T cell, B cell, etc.), which are difficult to distinguish in H&E slides. To complicate things, most of these cell types can be subdivided into a vast number of “specializations” and states of activities that cannot be distinguished from H&E slides anymore. Neither pathologists nor AI can reveal the required information from H&E sections since it was not captured when the tissue section was stained and digitized. Trying to understand the tumor-immuno microenvironment through the eyes of ‘Digital Pathology’ (i.e. only looking at H&E stained sections) can be compared to using black-and-white photography to spot yellow tulips in a red tulip field.

Immunofluorescence is the color photography equivalent of pathology. It allows us to see many more cell phenotypes by adding highly specific staining channels, and with the help of AI, their spatial relationships, or spatial biology.

Ultivue combines expertise in the development of high-quality multiplex immunofluorescence assays and AI-based computational pathology. We aim to deeply characterize immune cells and tumor cell and their spatial relationships in the context of intact tissue, using AI to analyze tissue images at scale. This presents multiple challenges. Our multiplex immunofluorescence images can have five billion pixels and more than five, sometimes 13 channels. These images can get as large as 100 GB. Images at the scale of 10 to 100 GB present a different order of magnitude to what many systems are built for – including systems built to analyze H&E images in pathology. Experts developing these tissue assays and understanding the scientific intricacies in the rich image data mostly aren’t AI engineers and cloud architects. They want algorithms to support the interpretation of the images to generate results quickly, without having to write or read code.

Handling over 10 GB images on Amazon Web Services

Amazon Simple Storage Service ( Amazon S3 ) provides a cost-effective and scalable solution for storing this data, and in combination with Amazon Web Services Storage Gateway , traditional block-storage-like access is achievable. Amazon Web Services Storage Gateway is an important part of our solution, as many of the common tools that pathologists and scientists rely on would cease to function without it. However, these commonly used tools that run on a Windows Desktop do not scale very well. That’s where Amazon SageMaker comes in.

Figure 1. The Ultivue architecture using Amazon S3, AWS Storage Gateway, Amazon SageMaker, and Amazon SageMaker Studio as key components.

Figure 1. The Ultivue architecture using Amazon S3, Amazon Web Services Storage Gateway, Amazon SageMaker, and Amazon SageMaker Studio as key components.

For this solution, we use Amazon S3 with Intelligent-Tiering to store data in a cost-effective manner with low-latency access. Storage Gateway provides file shares via the SMB network file system that Windows clients can use. Amazon WorkSpaces provides a pre-configured desktop working environments to our users, including the Storage Gateway-backed network drives.

Amazon SageMaker Studio enables cloud working environments for AI engineers, as well as pathologists. We use Amazon SageMaker’s processing, training, and inference APIs to analyze images at any scale.

Amazon SageMaker Studio as a user interface for non-technical users

When handling data at the scale that Ultivue does, the need for utilizing the scalability of cloud services is omnipresent. However, the bio-technology space requires medical and biological experts that cannot be experts in software and IT at the same time. Our user interfaces must simplify the use of highly scalable compute services, such as Amazon SageMaker, and at the same time, offer flexibility to a wide range of users, like AI engineers.

Utilizing custom images and lifecycle configurations allows Ultivue to provide an internal app-store-like experience on top of the highly flexible Amazon SageMaker Studio interface.

Figure 2. Screenshots of the user experience.

Figure 2. Screenshots of the user experience.

In the past, image analysis experts and AI engineers worked with software engineers to integrate algorithms into custom processing environments and connect them to different data sources. Now, processing at scale and data pipelines are reduced to API calls, and image analysis experts can make their algorithms available to pathologists without any intermediary needed.

From pixel to object

Without image analysis, information-rich multi-channel images are just images. Humans, even trained experts, are limited by–in the best case–the presence of only three-color receptors in our retina. Consequently, it becomes increasingly difficult to visually process the information coming from the 5-13 channels that are typically used with Ultivue technology, let alone considering the interactions of potentially dozens, if not a hundred, different cell types across millions of cells present in a tissue section.

AI-based image analysis gives us the opportunity to classify millions of cells and interrogate their frequency and spatial relationships in a data-driven way. This can help find out which specific immune cells need to talk to other immune cells or interact with a subset of tumor cells to predict therapy outcome, and more.

Visit Ultivue to learn more about this solution.

Learn more about how healthcare and life science organizations use Amazon Web Services at Amazon Web Services for Health . Do you want to learn more about how to use Amazon Web Services for digital pathology and other healthcare solutions? Contact the Amazon Web Services Public Sector Team here .

Pictured: Classified cells overlayed on IF image as image analysis detected them.

Pictured: Classified cells overlayed on IF image as image analysis detected them.

Read more about Amazon Web Services for healthcare:

  • How Digithurst and Telepaxx built a secure and scalable radiology solution chain using Amazon Web Services
  • AMILI helps advance precision medicine by building microbiome library on Amazon Web Services
  • Helping prevent sudden cardiac arrest in young athletes with AI
  • How KHUH built a long-term storage solution for medical image data with Amazon Web Services
  • Solving medical mysteries in the Amazon Web Services Cloud: Medical data-sharing innovation through the Undiagnosed Diseases Network

Subscribe to the Amazon Web Services Public Sector Blog newsletter to get the latest in Amazon Web Services tools, solutions, and innovations from the public sector delivered to your inbox, or contact us .

Please take a few minutes to share insights regarding your experience with the Amazon Web Services Public Sector Blog in this survey , and we’ll use feedback from the survey to create more content aligned with the preferences of our readers.

Moritz Widmaier

Moritz Widmaier

As product manager at Ultivue, Moritz Widmaier is responsible for the development of artificial intelligence based software solutions and services connected to the companies’ cutting edge tissue staining technology. Moritz holds a PhD in biochemistry and has more than seven years experience with the development of image and data analysis solutions in the field of digital pathology and spatial biology.

Martin Schulze

Martin Schulze

As principal consultant for Technology and IT, Martin supports Ultivue in conceptualizing and implementing a scalable cloud-native digital pathology infrastructure. He and his team at Slash-m GmbH (/m) routinely help medical technology startups to build sustainable medical software in complex, uncertain and ambiguous environments. Martin has been working for over 13 years in the medical software space from surgical tables and robots to cloud and web application in the pathology and radiology space.

Florian Leiss

Florian Leiss

Florian Leiss is currently a strategy consultant for companies in spatial biology, digital health, and diagnostics. He was previously responsible for digital and data-driven offerings, product development and management, and partnering at Ultivue as VP of Product Strategy & Corporate Development. Florian has been working for over 10 years in software development for healthcare to make AI-powered CDx in oncology a reality, to support radiologists with routine reporting, or to empower pathologists with image analysis at AstraZeneca, Smart Reporting, and Definiens. He holds PhDs in neuroscience and philosophy.

Ruben Cardenes

Ruben Cardenes

Ruben Cardenes is a physicist and holds a PhD in artificial intelligence (AI) and medical image analysis. As Director of AI Engineering at Ultivue, he is responsible for the entire AI development pipeline, including data management, MLOps, model development, testing, and deployment. With over 20 years of experience in computer vision in the medical field, he has previously worked as an AI expert at AstraZeneca and Hensoldt Sensors. He has a strong research background in the academy, with over 20 international journal publications and more than 40 international top peer-reviewed conference publications in the fields of medical image analysis, computer vision, and computational pathology.

Razvan Ionasec

Razvan Ionasec

Razvan Ionasec, PhD, MBA, is the technical leader for healthcare at Amazon Web Services in Europe, Middle East, and Africa. His work focuses on helping healthcare customers solve business problems by leveraging technology. Previously, Razvan was the global head of artificial intelligence (AI) products at Siemens Healthineers in charge of AI-Rad Companion, the family of AI-powered and cloud-based digital health solutions for imaging. He holds 30+ patents in AI/ML for medical imaging and has published 70+ international peer-reviewed technical and clinical publications on computer vision, computational modelling, and medical image analysis. Razvan received his PhD in Computer Science from the Technical University Munich and MBA from University of Cambridge, Judge Business School.


The mentioned AWS GenAI Services service names relating to generative AI are only available or previewed in the Global Regions. Amazon Web Services China promotes AWS GenAI Services relating to generative AI solely for China-to-global business purposes and/or advanced technology introduction.