How to Better Support Testing in Production with CA APM

by November 1, 2017

The term “testing in production” is polarizing. Mention it to some teams and they’ll say it’s sheer genius; a critical method for gaining realistic insight into app performance behavior. On the other hand others might scoff that it’s just dev and QA playing catchup -running tests they didn’t have time for earlier in the development cycle.

But testing in production is more about supporting the realities of modern software development than it is about playing catchup. Yes, the term will probably get business folks and ops teams squirming, but in truth it’s an essential element of successful DevOps and continuous delivery initiatives.

In the modern development world of API’s, cloud, containers, microservice architectures and the Internet of Everything, testing in production isn’t just common sense – it’s really the only way to progress. Massively complex distributed applications will perform in crazy ways, users will encounter problems you never anticipated, and cloud services no longer under your direct control can and will fall. And without a production testing strategy in place these problems challenge even the best operational capabilities. Worse, they’ll remove critical feedback loops and negatively impact decision making.

But in truth, testing in production isn’t new; it’s been happening for years. Every time we slavishly update our Facebook pages and LinkedIn profiles we’re active participants in experimentation – even if we don’t know it.

In enterprise IT, testing in production goes by many names and is also becoming well established. This includes:

  • Blue Green Deployments – is a practice that reduces downtime risk by running two identical production environments (blue and green) only one of which is live (blue). As you prepare an app release for deployment, testing is conducted in the green environment. Once you have the green light (forgive the pun), routing traffic is switched to the green system. It’s not without its challenges, but it’s a great way to roll back quickly when things go wrong.


  • A/B or Split Testing – is used to test new features of an application against a variety of factors like uptake, popularity, conversions, and how these impact revenue and profitability. A/B production testing is mostly involved with the UI, but of course back systems need to be available and performant to do it reliably.


  • Canary Releases – as the name suggests, this practice involves sending out a version of your software (the canary) into the live environment (the coal mine) to see how it performs. Here you check critical integrations, response times, error rates, CPU, memory utilization etc. If the canary app keeps chirping, great. If it dies, no great loss – you have the feedback to quickly rollback the deployment.


Irrespective of method, the one consistent requirement for testing in production are short and fast feedback loops. The faster you gain information about application performance, the quicker you can ‘get out Dodge’ by rolling back the release, or proceed and reap the benefits.

Modern Application Monitoring is absolutely essential for fast feedback and CA APM provides many great features to support it. These include:

  • Visibility into KPIs and SLA’s – CA APM can be easily configured to monitor critical changes to KPIs for business services. Using Experience views, teams have single line-of-sight into app performance, aggregating key performance metrics into overall health scores. With canary releases these provide indisputable evidence that your canary (app) is failing, but, and here’s the rub – in context of business outcomes and customer experience.


  • Change Impact – the what, when and how – It’s one thing rolling back quickly, it’s another being able to figure out what changed caused the problem. As updates are made – be that software, environmental or architecture – we need to ensure that the folks making the changes are alerted as quickly as possible.

CA APM supports this in a number of ways.

– With fully automated transaction tracing, developers can quickly pinpoint problematic code.

– Using Timelines, architecture teams can track back to determine was specific environmental update caused degraded performance.

– By layering infrastructure onto app topology maps, app support teams can quickly correlate symptomatic performance conditions with related infrastructure dependencies.


  • Facing the “unknown unknowns” – complex distributed applications behave in complex ways. They can exhibit production behaviors never anticipated. In the past it was pretty straightforward to check and alert on known failures. Now, however, it becomes difficult to detect severe conditions that build slowly and cumulatively over time – they can fall under the radar.CA APM’s differential analysis can help address these complex issues. Employing proven statistical methods and analytics it distinguishes nuisance alarms from anomalous trends worthy of attention. When combined with blue green deployments, it could provide a reliable and precise roll back trigger. Not least because every alert in these complex environments shouldn’t force teams to hit the panic button.


This isn’t a definitive list of how CA APM supports testing in production. Depending on your own organizational practices and use cases there are many more applications.

For example, CA APM’s open data model and Team Center could be employed to surface key attributes contributing to performance – comparing blue green deployments side by side. In an A/B testing scenario the integration between CA Application Performance Management and CA App Experience Analytics can provide some great insights. Like for example determining that slow customer uptake of a new feature is actually related to back-end performance – not the feature itself. Great insight = better decision making.

Testing in production shouldn’t be feared, castigated and reviled. It’s an essential part of modern software development. To do it effectively we need effective app performance monitoring.