Onstage at DevOps Enterprise Summit in both London and Las Vegas, Shaun Norris (then Global Head, Cloud Infrastructure) explained how he and his colleagues at Standard Chartered were transforming operations at their 165-year old bank.
Rundeck played a key role in the transformation and provided a rapid return on investment:
- 28 person-years of effort saved
- 13,000+ manual reviews of production operations activities avoided
- Time to Repair reduced by 25 minutes on average
- 200 self-inflicted operations incidents avoided
Here is a 4:00 minute edit of highlights from Shaun’s talk and his remarks about Rundeck:
Transforming a global bank with 86,000 employees operating in more than 60 countries
Let’s put this transformation story into perspective. First, this effort was driven by the bank’s mainline technology organization that was responsible for the more than 1,000 applications that accounted for virtually all of Standard Chartered’s revenue.
Second, Shaun appreciated the daunting scale and history of Standard Chartered. The bank’s charter was signed by Queen Victoria before the invention of the lightbulb. They have 86,000 employees operating in more than 60 countries (each with its own regulatory requirements ). Finally, this wasn’t a transformation of a special project or an internal startup, but rather an entire institution.
“All of the regulation — and the accompanying compliance that comes along with it — really means that our processes were optimized for compliance and not for speed” -Shaun Norris
While compliance and security should be the highest priorities of any bank, speed and agility are essential to compete in today’s global financial marketplace. Shaun and his colleagues embarked on a journey to optimize Standard Chartered’s processes for both speed and compliance.
Their transformation featured several significant initiatives including streamlining infrastructure delivery, private and public cloud adoption (stressing that “how is more important than where”), the creation of an SRE capability, and the adoption of a standard, modern DevOps toolchain.
Perhaps most importantly, their leadership developed a standard set of tenets and principles (Figure 1) to be adopted across the technology organization. This gave the organization alignment around expectations and clear direction on improvement efforts.
Rundeck expands from solving local pain to delivering global impact like many other Rundeck community members, Rundeck initially entered Standard Chartered through a “grassroots” effort to solve a set of acute problems in a specific group. In this case, it was a middleware and web support team within Standard Chartered that deployed Rundeck to automate tasks like service restarts and failover actions.
A single engineer discovered Rundeck and undertook the not insignificant task of managing the approval process. Once in-house and running, other engineer’ saw the results and started scripting all of the their manual tasks and then created standardized and shareable Rundeck jobs.
“How we started using Rundeck is almost more interesting than the tool itself. One middle seniority engineer was unsatisfied with the status quo of how we did operational tasks. As a result, he went through the considerable effort […] to have this tool introduced to the bank.” -Shaun Norris
When Shaun came in to run Standard Chartered’s retail banking technology group, he noticed the early success of Rundeck and saw an opportunity to spread that success to the rest of the bank. By 2018 Rundeck was used for incident remediation in over 400 applications across the bank.
“We found that TTR [Time to Repair] is reduced by 25 minutes per incident for apps where we use Rundeck than where we don’t. ” -Shaun Norris
In addition to replicating the speed and quality benefits, Shaun found another important way to exploit Rundeck across the bank.
Before Rundeck, all production changes required special access tokens and all terminal sessions had to be recorded for immediate review by the manager of the person making changes. Yes, each and every change, no exceptions. This was no minor feat. Thousands of changes had to be reviewed annually (13,000+ by Shaun’s estimate). Imagine the employee time required for this effort.
Rundeck dramatically reduced that burden by providing a safe way to distribute production access. With Rundeck, each team member has restricted access to the jobs that they have been approved to run. Rundeck logs all execution so that auditors can see who ran what, when, where, and what the outcome was. This includes traceability of job definitions to ensure the correct job (and underlying scripts) are run.
“We think next year Rundeck is probably going to save us 28 people-years worth of work, at fairly conservaitive estimates, and we haven’t really rolled it out particularly widely yet.” -Shaun Norris