Historically, banks have achieved competitive advantage through distribution. The more branches and ATMs they had on every corner, the more they succeeded. That competence—while still important—gave way to information technology advantage in the 2000s. Banks added networks and technology to run their businesses and serve customers more efficiently. This trend is shifting with the advent of ubiquitous digital access and cloud-based networking computer power.
Today, the only source of competition in the banking world is an obsession with serving and delighting customers. This is something Melbourne, Australia-based ANZ Bank had in mind recently when it launched a new application performance management (APM) program with the hopes of extending it across the entire IT infrastructure, which includes four data centers, mainframes and more than 10,000 servers.
“The primary goal of all my teams is 100 percent availability of services for our customers. It’s simple as that,” explains Adam Cartwright, Head of IT Security and Operations at ANZ Bank.
“That means whatever channel it happens to be—whether it’s a corporation doing a payroll transaction or a private user transferring money using Internet banking—it’s got to be up. It’s got to be seamless. It’s got to perform to their expectations. That is the primary mission of operations within technology.”
The Road to APM
Unfortunately, the complexity of modern distributed applications means that this doesn’t happen all the time, says Cartwright. Applications and systems go down, adversely impacting end users. In 2012, it became apparent that infrastructure monitoring alone—which focused on platform and event monitoring at the infrastructure level—was not enough to give ANZ Bank the insight it needed to fix and prevent incidents within the organization.
“What you really need to do is to understand the transaction flow within the application context you’re going to identify the root cause, or if you’re going to get early signaling of potential problems before a customer actually has [an incident],” he explains.
For example, the bank has a distributed payment application that is used to transact billions of dollars for its highest value customers. The application comprises more than 120 distributed servers and 60 separate Java applications. Whenever there was an issue with that application, it was nearly impossible to figure out what the root cause of the stability issues were. The worst part was that many times ANZ Bank IT didn’t realize there was a problem until its customers notified them about the issue.
These challenges led to many of the goals of ANZ Bank’s APM project, one of which was to bring CA Technologies Application Performance Management (CA APM) into the organization. First and foremost, IT wanted to reduce the number of incidents caused by application releases. This would increase the quality and confidence of application deployment or changes while improving the overall lifecycle of new ANZ applications.
In addition, Cartwright wanted a way to perform deep dives into the application layers to obtain code-level visibility to measure performance and availability. He also wanted to measure and analyze transactions as they move across the distributed and highly diverse ANZ Bank infrastructure technologies. These efforts have helped the bank minimize customer downtime.
Gaining Buy-In, Gaining Benefit
The road to APM adoption wasn’t an easy one, especially on the test-and-delivery side of things. “There was a lot of resistance within those teams to the adoption of it,” Cartwright says. Cost was an issue, he adds, since teams didn’t want to use their own funding to implement the software, and potential implementation time was also an issue. To avoid these arguments, the ANZ Bank IT operations team decided it would fund the licensing for all potential users. Once this happened, it didn’t take long for the other IT departments and users to become advocates of CA APM.
Using CA APM, IT is now proactively alerted when there’s an issue before it affects service. Since CA APM allows the company to not only look at transactions flowing through an application, but also identify the business user behavior attached to those transactions, outages can be found more quickly, and applications can be redesigned so outages don’t repeat themselves.
“Getting that insight in production is fantastic, but what is perhaps more brilliant is getting that insight in the development–and-test environment,” he says. “We have been able to show that by putting APM into development and test—making it part of the process in those areas—you can stop defects from getting into production. And that’s probably the most surprising thing to people new to APM, but the most critical thing from a production support point of view.”
Indeed, says Cartwright, this is one of the more astounding benefits of CA APM. For example, one project team was able to prevent 10 high-severity incidents from happening, saving more than a dozen hours of investigations. Another project team reduced recovery time from more than four hours to less than 30 minutes with no impact to service. “While it’s good to have APM and to use it to diagnose problems once they’ve occurred in production, you’ve got a customer that has been affected,” Cartwright explains. “It’s much better to stop that customer effect from occurring in the first place by not letting that sort of design issue propagate into production under transition.”
Crawling, Then Running
One of ANZ Bank’s first projects in the test–and-development realm was a payments application that caused serious production issues. The bank had five or six major releases each year, and every time a new version went live there was “a raft” of high-severity incidents. “No matter what we did with testing, reviewing, traditional sorts of approaches or performance and volume testing, we always managed to end up in a situation where we had a release go live and we’d have problems in production,” says Cartwright.
Using CA APM, ANZ Bank eliminated between 10 and 15 high-severity incidents, which often stretched out three or four weeks following a release. “Now, we’re down to one or two or, in some cases, zero [incidents],” says Cartwright. IT employees are thrilled with the change.
“Since APM from CA Technologies has been in, we have been able to pinpoint the exact impacted servers and restart these without impacting business and payments processing,” explains Joseph Rocco, Support Transition Analyst at ANZ. “Before APM, the LMS [Limits Management System] recovery was four-plus hours. Post APM, it has been around 30 minutes total to restart impacted servers with no outages.”
This is an example of the fact that, while ANZ’s main focus was on customer satisfaction, a positive side effect of reducing incidents is the boost it gave the IT organization as a whole. “If you can stop having incidents, you track capacity and headcount within the organization to do other things, more proactive things,” Rocco says. “APM essentially means your organization is going to continue to grow as your system footprint grows and the complexity grows.” IT employees are freed up to be proactive rather than reactive.
The use of APM in banking can help firms like ANZ Bank differentiate themselves, says Bill Doyle, VP and Principal Analyst at Forrester Research. “Any tool that can improve customer service and satisfaction is critical in the era we have entered where competitors can easily match your capabilities, especially around IT,” he explains. “Now, the way [companies] are going to win and keep customers is by doing a great job at delighting them.”
APM: Like Money in the Bank
While the initial rollout of CA APM was small—ANZ bank has deployed about five major applications using CA APM to date—Cartwright has teed up the technology to roll out more extensively throughout the organization. “We’ve finished the rollout of what we’re calling the baseball pitch,” he says. “That is, the appropriate number of collectors, patterns and standards that would allow a large number of applications or products coming through just to commit to that infrastructure. That infrastructure is built; it’s connected up to our ticketing systems, our alerting systems and our paging systems.”
This year, ANZ Bank expects to roll out CA APM with a large number of new and existing banking applications in Asia, as well as “six to seven major releases” in the core banking space, especially since there is what Cartwright calls an avalanche of people waiting to adopt the technology. “We have payments applications coming on, and we have critical channels like Internet banking and [mobile banking app] goMoney scheduled.” Combined, ANZ Bank is shooting for at least 20 critical applications to be running with CA APM by the end of this year. The longer-term goal, Cartwright says, is to have between 75 and 100 critical applications using CA APM by 2017.
Theoretically, this will have a positive impact on ANZ Bank’s relationship with its customers. They will see fewer outages and better, more innovative software releases that address their needs today and tomorrow. And once this happens, banks like ANZ earn their ability to keep their customers—and the profits associated with them.
Says Forrester Research’s Doyle: “People are no longer tethered to a local branch. You can move an IRA to a new financial services provider in a matter of moments. That’s what makes incident reduction and elimination of outages so essential.”