Open Source Software Adoption – Size vs Speed
How is it that smaller companies move fast and innovate quicker than large firms? In this blog we form a balanced view from each perspective and draw out some key comparison points with respect to using open-source software.
Comparison
On one side we have our large investment bank, let’s say they have 76k employees with 38k employees in technology aligned roles. On the other side, we have a growing media company with 60 employees and 50 of those in technology aligned roles.
Over the last three to five years large investment banks have been changing mindsets:
“Cloud First”
“Think like a startup”
“Break down the barriers of bureaucracy that hold us back”
Large firms are observing the technology innovation and the rate of adaptation to the market demonstrated by smaller more nimble companies. To bring products to market or release internal applications with the same speed that a start-up can is near impossible. However, there are patterns that form in startups that make sense in a larger context. Vice versa is also true, with large technology budgets and armies of highly talented people large firms are able to solve problems just not possible in a more focussed smaller outfit.
At M2A our high profile media partners have a very low tolerance for error, our software delivery and infrastructure orchestration has to be automated and repeatable. We put in significant thought and effort into optimising our software delivery pipelines. Today we are fast, tomorrow we want to go even faster. We want multiple zero-touch releases per day. We want a fully automated unit and BDD test framework. We want vulnerability scanning and extensive quality assurance on every release. We need quicker and more scalable environments-on-demand to boost developer productivity.
In our size vs speed comparison let us take upgrading an open-source software product as an example. Recently at M2A Media we found a minor bug in the version of Terraform being used. We read through release notes (also with some pointers from StackOverflow) and found the exact bug had been corrected in a later release. The path to upgrade was almost trivial. A two-character code change and then let the CI/CD pipeline do its thing. Integrate code, check dependencies, unit test, feature level test, end to end test, auto screen code quality and deploy all automatically. Job Done. Manual checking and inspection that the ephemeral cloud infrastructure stood up by the new version of IaC platform are equivalent to that created using the previous release was also performed. As M2A Media has confidence in Hashicorp not to introduce malicious or vulnerable code in later releases of the product we can simply upgrade, integrate-test-build and move on. Given that we can also go and inspect the open source code and rely on a huge developer community to point out any concerns – any further investigation might be considered valuable time wasted.
Now if we cross-compare how the same scenario would play out at a large tier one bank. The steps might break down as follows:
- Bug identified.
- Research into the solution to the smaller company – search the error message and come to understand it’s a known issue – upgrade required.
- Look in the internal list of approved outside software and local Terraform available versions.
- Find that the latest version is not available (inevitably this has lagged behind the latest release).
- Enter a ticket to the Application Infra team to make the new Terraform version available.
- Wait for someone to pick up the ticket.
- Once the ticket has been prioritized the app infra team will download and test the new version in a quarantined lab environment.
- A full set of very thorough but somewhat manual testing will occur – largely outside the context of the intended use case.
- A change delta analysis on the difference between the current Terraform version in use and the addition of the proposed version would be drafted.
- A spot check on the licensing model will occur to ensure there no alteration to terms and that it still fits with company policy.
- All the findings will be presented to the Security Architecture team to review – via a ticket.
- Once picked up by the Security Architecture team they either approve or reject the proposed version upgrade.
- Once the new version is approved by the Security Architecture team the software is made available to the developer community in the firm.
- Developers can now perform the required two-character code change and use the new version of Terraform and hopefully observe the new release fixes the bug identified.
- If dev testing proves the bug is fixed, a similar process to the small firm can now commence – push it through the CI/CD pipeline and verify everything is good.
Reflection on differences
Given that the small company can move quickly upgrading when onboarding third party open source software, they will clearly have a huge advantage over the larger company. The small company knows that multiple large firms use the same product and will inevitably go through a rigorous set of checks to ensure nothing untoward is introduced. It is likely that any negative findings will be fed back to the community. Furthermore trusting firms like Hashicorp to properly gatekeep their products is not really considered a massive leap of faith given the reputation and standing in the tech arena.
On the flip side, the large company has too much at risk to take the leap of faith and just take the version upgrade. How would it look in a severity 2 post-incident review that determines the root cause of a mass PII info leak due to compromised code introduced in an upgrade to open-source based product? The checks and balances implemented may seem overly cautious to some but if that software product is used across the firm by hundreds of applications, there is little or no appetite to take that upgrade risk. The focus for the larger firms to improve their situation is not to avoid the due process, but rather to accelerate and streamline the outside software onboarding process.
Finally, it would be interesting to cross-compare the open-source or outside software onboarding approach at a range of companies. For example, Netflix often regarded as a leader in the DevOps space, what do they do? With around 10k employees they have a relatively low proportion of technology aligned staff – with 1.6k in engineering and 1k in operations (source: LinkedIn). It seems unlikely they allocate the same amount of tech resources in this space as firms with tens of thousands of IT staff.
About the author
Leon Box has spent decades working for tier 1 investment banks and other companies large in size. Now situated as a Software Development Manager at M2A Media he is ideally placed to draw comparisons between firms at extreme end of the size scale.
