Dealing with legacy systems
In an ideal DevSecOps world all infrastructure servers and auxiliary applications are running an up-to-date LTS versions and are subject to an actual lifecycle maintenance policy. In the real world, we have to deal with end-of-life applications installed on unpatched end-of-life operating systems… and still make this secure.
This should be said categorically from the beginning: running unpatched and/or end-of-life applications and systems is a very bad practice in general.
Why not? In the first place, using EOL software is like using old machinery. You save on the cost of change now, but since it’s not maintained, it will eventually break.
It will break either as result of an exploit, or just stop working as result of random interactions with other parts of the system. Any “hey, we saved a day per year by not upgrading” will come back as sleepless weekend while desperately trying to revive an end-of-life system. Effectively bringing it back to its “end-of-life but working” state.
And obviously, over time unpatched applications accumulate a significant amount of vulnerabilities discovered – and patched – in newer versions, and you can nothing but watch them being published as exploits and then turned into mass-scanning campaigns.
Unfortunately, powers that be tend to have different priorities. The whole system may be already end-of-life but new and shiny replacement is delayed. The legacy application may be planned to be decomissioned in two years and priority is on the new one. Obviously, these same powers expect that the system is not hacked and you will find yourself in such position sometimes. At this stage what you do before that exploit was published is of critical importance. What needs to be done?
-
Watch what goes EOL when. Even if you were living under a stone, you would eventually
learn about Ubuntu 14 going end-of-life
in April 2019 simply because every single Linux news website was writing about it
for weeks. But your software inventory is much larger than that, and
you certainly do have software that does not come with distribution
packages, such as Jenkins or ElasticSearch or whatever. These too go EOL
and you need to know in advance. Have an inventory, even in a text file,
and make note of their EOL dates. Each of these products has an
announcemailing list, go subscribe. And subscribe your tech leads too, let them feel the pressure ;) - Create tickets for upgrade of EOL components. Yes, it may take months for them to be prioritised for implementation, but even if it takes, the job is done sooner than “never”, the risk is reduced. And you have CYA. One extremely effective solution to prioritise tickets is to just go and implement them yourself. Learn Ansible, Puppet, Salt, whatever is used and just submit a merge request.
- Isolate. Comparmentalisation of application components is one of the security architecture principles, but you also need to isolate legacy, unpatchable systems even further. If you cannot upgrade some of your systems, be it old Linux, Solaris servers or Windows XP embedded devices, you cannot keep them in the same subnet as general usage devices like laptops, smartphones, workstations used to access Internet and insert USB sticks. NHS learned that the hard way and it now recommends that “obsolete platforms are segregated from the rest of the network.”
- Mitigate. Does your business just have to run an EOL version of ImageMagick with all the juicy 2016 exploits? Fine, just run it through firejail (need to modify the app that calls it) or create an enforcing AppArmor profile (app doesn’t even know) that blocks access to anything except for working directory, especially network. Ugly, but buys you some hope.
- Negotiate partial upgrades. The business is reluctant to upgrade complex application servers with EOL everything inside. But you also have servers there that basically run the OS and one daemon from distro package, that work as routers, proxies, firewalls or monitoring. These can be upgraded always, so do it. In general, try to keep your DevSecOps infrastruture up-to-date even if the rest of the network is EOL, simply because this may be the last line of defence that will at best prevent and at least detect a breach.