Empowering Life Science Cloud Giants to Jointly Build the Foundation of Digitalization

Under complex business scenarios and stringent industry standards, the challenges Veeva faces are multidimensional and interconnected. Among them, the upgrade of the core database is particularly challenging and is regarded by the Veeva team as the most impactful challenge to the business.

As the most widely used cloud service for Veeva in China, Amazon RDS supports the core of its CRM system. When the news broke that official support for MySQL 5.7 would soon end, Veeva was faced with the daunting task of upgrading all 200+ Amazon RDS database instances online to version 8.0 within the stipulated time frame. This not only meant a massive amount of work, but also carried significant risks: potential compatibility issues between the old and new versions, unavoidable business interruptions during the upgrade process, and unknown performance impacts post-upgrade, all of which could directly affect the many pharmaceutical enterprise tenants they serve. More strikingly, if the upgrade were not performed, the extended support fees required would amount to 13 times the cost of the instances, which added immense cost pressure to Veeva’s decision-making process and strongly pushed the company to quickly develop an upgrade strategy.

As a company providing SaaS products, accurately calculating the resource cost per tenant and evaluating its Return on Investment (ROI) is crucial. However, Veeva previously had up to 97% of its cloud resources un-tagged, making cost allocation a "black box." The technical team could not clearly correlate cloud bills with specific tenant businesses, making it difficult to determine the business cost and ROI of individual tenants. This not only hampered financial accounting but also obstructed data-driven decisions for refined business operations.

Veeva found that users in certain specific regions or using certain network operators encountered obstacles when accessing its applications, leading to complaints from some end-customers. The root cause was that the IP addresses of the Application Load Balancer (ALB) could change, causing some users' network traffic to be directed to old, invalid addresses. Although the impact was limited, for Veeva, which serves B2B clients, any user experience issue was unacceptable for its brand reputation and customer satisfaction requirements. Simultaneously, the impending IPv4 address charging policy prompted Veeva to proactively consider optimizing its network architecture to avoid unnecessary cost increases.

Faced with the unprecedented scale and high-risk database upgrade task, the TAM and PS teams provided full-chain, systematic support from strategy planning to implementation and post-upgrade optimization. The entire upgrade process took approximately six months, and the close collaboration between both teams was key to the success of the project.

At the project's outset, the TAM team first assisted Veeva in comprehensively reviewing the list of all 200+ instances to be upgraded and organized dedicated meetings. During these meetings, the TAM detailed various Amazon RDS upgrade strategies, including blue-green deployment, and conducted in-depth analysis of the advantages, disadvantages, cost implications, and suitable business scenarios for each approach. Based on Veeva's high requirement for business continuity, both parties jointly selected the blue-green deployment method, which posed the lowest risk and minimal business impact. This early-stage professional consultation and solution selection helped Veeva quickly identify the upgrade strategy most suitable for its needs, laying a solid foundation for the smooth progression of the entire project.

During the six-month upgrade cycle, to ensure every production environment operation was flawless, the TAM team applied for multiple Infrastructure Event Management (IEM) for Veeva. Especially in the initial stages of the upgrade, every critical operation involved the participation of Enterprise Support engineers, jointly executing and providing monitoring. This "escort service" gave Veeva's operations team significant confidence and peace of mind, ensuring the correctness of all operations and effectively avoiding major impacts on the business.

After the upgrade was complete, new challenges emerged. The Veeva team found that MySQL 8.0 exhibited significantly lower performance compared to version 5.7 when executing DDL (Data Definition Language) operations, affecting its new synchronization tasks in a shared environment. Additionally, some instances experienced frequent restarts due to Out-Of-Memory (OOM) issues, directly causing service interruptions.

To address these "after-effects," the TAM promptly organized a performance optimization workshop led by senior PS engineers. This workshop precisely targeted the pain points of Veeva's operations team:

• Migration to Amazon Graviton Instances: To address the DDL performance degradation, the Enterprise Support team proposed migrating instances to those based on the ARM-architecture Amazon Graviton processors. Through actual performance testing, it was demonstrated that Graviton instances could not only deliver an 18% improvement in DDL performance but also achieve a 24% reduction in instance costs. This solution, offering both performance and cost benefits, perfectly aligned with Veeva's needs and was ultimately adopted and widely implemented by the customer.
• Deepening Database Monitoring: PS engineers provided detailed explanations to the Veeva team on using monitoring tools like Amazon RDS Performance Insights, sharing numerous practical monitoring metrics and recommendations relevant to operations. The engineers also provided clear best practices—ensuring available memory always exceeds 500MB through monitoring. This specific and actionable suggestion was adopted by the client, effectively resolving the issue of frequent instance restarts.

Through a comprehensive support system encompassing "pre-support, during-support, and post-support," the Enterprise Support team not only helped Veeva successfully complete the database upgrade but also resolved post-upgrade performance bottlenecks, enhancing system stability.

To tackle the cost allocation challenge, the Enterprise Support team provided a combination of "strategy + tools," helping Veeva move from chaos to clarity.

• Optimizing Cost Models: While transitioning from Reserved Instances (RI) to Savings Plans (SP), Veeva encountered calculation difficulties. The traditional method required running on-demand instances for 7 days to get system recommendations, leading to added costs. The TAM provided a new calculation logic: accurately estimating the required SP quota based on the historical RI utilization rate, helping the client achieve a smooth transition without additional costs.
• Establishing Tagging Standards: The team assisted Veeva in establishing mandatory tagging policies using services like Amazon Config, standardizing resource tagging at the source, and identifying un-tagged legacy resources for remediation.
• Introducing Visualization Tools: More importantly, the TAM team recommended and assisted in deploying the Cost and Usage Dashboard (CUDOS) solution for Veeva. Through this monitoring panel deployed within the customer's account, Veeva could perform multi-dimensional, granular cost analysis based on tags, enabling accurate calculation of cloud costs incurred per tenant. This provided a reliable data foundation for final ROI assessments.

Through in-depth collaboration with Enterprise Support team, Veeva not only successfully resolved its immediate technical challenges but also achieved a qualitative leap in cloud operational excellence.

• Cost Savings: Achieved a 15% reduction in overall cloud platform costs. By adopting Amazon Graviton instances, instance costs were reduced by 24%. Optimizing Amazon OpenSearch configurations saved 214,000 CNY monthly, and optimizing network architecture saved approximately 10,000 CNY monthly in IPv4 fees.
• Performance Improvement: After migrating to Amazon Graviton instances, core database DDL performance improved by 18%.

• Business Insight: It achieved cost accounting down to the tenant level, providing business departments with unprecedented cost insights and enabling accurate evaluation of the ROI for each project.
• Enhanced System Resilience: Successfully completed the smooth upgrade of the core database and resolved performance bottlenecks and stability issues, significantly enhancing the stability and availability of core business systems.
• Team Capability Uplift: Through multiple in-depth workshops and daily communications, Veeva's operations team gained extensive frontline practical experience from PS engineers, significantly improving their hands-on capabilities in database performance tuning, cost management, and cloud monitoring.
• Improved Customer Aatisfaction: By adding a Network Load Balancer (NLB) with a fixed IP, the optimization of the network architecture completely resolved access issues for end users, eliminating related customer complaints and ensuring an excellent user experience.