How to manage serial software offenders
Let’s examine some typical “pre-software crime scenarios”, what we can do to prevent them, and what, if any, punishment should be applied.
I’m a sucker for good Sci-fi reads, especially those of Phillip K Dick. I really got a kick out of the Minority Report, which describes a future where a specialized police squad can apprehend would be criminals based on the forewarnings of three psychics – the “precogs.”
Wouldn’t it be fascinating if we had the same capability in IT? Where a specialized team could be forewarned of technology related crimes that persist across the software development lifecycle?
It sounds fanciful, but we have to accept that everyone at some stage is guilty of making some software misdemeanor – like introducing buggy code, skipping testing, or holding back changes.
Every IT pro is not a crook, of course. Usually software crimes happen for many reasons beyond personal gain, which are well within our ability to control (with or without psychics). But in the spirit of classic sci-fi and using some fictional personas, let’s examine some typical “pre-crime scenarios”, what we can do to prevent them, and what, if any, punishment should be applied — unlike the movie, cryogenically imprisoning the offender is not an option.
Let’s take Sarah for example. Working for Cruise Inc., Sarah is a gun developer who supports a critical web application. Part of her charter is to work closely with marketing to assess the impact of updates in terms of sales conversions. Ideally, she wants to do this using cohort analysis or A/B testing to selected customer groups – but there’s a problem and she’s about to commit a crime.
Sarah is always frustrated by how long it takes operations to make infrastructure available for testing. Too often it takes days to make systems available, and because of the constraints she can’t immediately assess the effectiveness of her coding efforts in context of generating new business. No surprise then that Sarah conspires with marketing to procure dedicated lab infrastructure and starts manually coding test stubs.
This of course is a crime Sarah shouldn’t need to commit. Now, advanced techniques such as Service Virtualization would allow developers like Sarah to create simulations of needed infrastructure, applications and test data and make these available throughout the software development lifecycle. As such, developers, testers and performance teams could work in parallel for faster delivery, lower costs and higher quality software applications.
Justin is a junior developer at Cruise and being fairly new to the team he’s eager to impress. Sometimes too eager, but since his group is measured and incentivized on functions delivered he just wants to crank out code as quickly as possible. But along with other members on the team, Justin has become a serial offender.
Because of persistent release delays he knows he can sometimes produce rushed, sloppy code so as to quickly move onto other tasks. Sure, he always intends to clean it up before the actual release, but given that delays often run into weeks it’s more likely it’ll never happen.
This crime could have been avoided using two common sense DevOps practices. Firstly, quantitative based incentives are recipes for disaster (more doesn’t mean better), so if Justin and his crew are to be incentivized this should really be business outcome focused. Secondly, by establishing continuous integration/delivery, automated release processes and integrated testing, bad code should have nowhere to hide.
Over in change management, Frank has seen it all. A 30- year veteran, Frank has put his ITIL training to good effect; designing a process whereby all update requests are routed to his team. He’s especially proud of what he sees as the many procedural improvements, like the establishment of a change advisory board and a test readiness review committee.
Even though he’s always under pressure from development, Frank sees no reason to relax his hold. After all, his system has worked fine for the large ERP system he helped implement 15 years ago. Indeed he feels so strongly, he’s about to send an email to the operations manager (who’s an old personal friend) titled – “If it isn’t broken, don’t fix it” to canvass support for his risk avoidance crusade.
Frank’s position reminds us that no matter how good our development or operations functions, any one person or dumb process can potentially derail a DevOps style initiative. In this case the crime is rigidity and low trust, resulting in longer lead times and delays. While Frank is the bottleneck, the bigger villain is organizational management for allowing entrenched and misaligned behaviors to dominate.
You don’t need psychic abilities in the software game to work out when crimes are going to be committed. They happen every day – from an idea’s inception right through to production.
Whether you’re in Dev, testing or operations, there’ll always be some crazy process, intransigent behavior or lousy tool that’ll motivate you to do the wrong thing. Any DevOps initiative should take this into account; examining the people factor at all stages across the software lifecycle and by tailoring processes and tools to better support them.