Bench Talk

Applying security updates is one of the most fundamental security controls that you must regularly perform to keep your devices and network safe from attack. Security updates protect your device from attackers and indiscriminate malware that exploit vulnerabilities in your installed software to gain access to or otherwise harm your system or network. While simple in concept, evidence shows regularly patching computer systems remains difficult. Vulnerability management practices were challenged in early 2000 by the Code Red and NIMDA malware worms; yet, nearly two decades later these processes still failed in some organizations as demonstrated by the WannaCry malware outbreak in 2017. All three of these malicious worms that trashed networks of unpatched servers had security updates available months before they spread. In the time between these high-profile attacks, there have been thousands of pieces of malware in the wild that damage and disrupt unpatched servers and devices every day.

In concept, applying a security update is very easy and usually takes just a few minutes. However, this is a software change and should follow a standard and well-documented change control process that includes:

• Testing
• Scheduling of an outage window
• Planning rollbacks

These activities take time to develop and practice. Additionally, this subjectively simple process can be complicated if you do not fully understand the devices that need to be updated. For example, you may be afraid of what might happen if the update fails or the system must reboot. Plus, with the influx of new devices running all sorts of different firmware and operating systems it can be difficult to stay on top of all the required security updates. These complications can overwhelm immature vulnerability management processes to the point that some systems go for long times—months or even years—without updates.

Not applying security updates results in very real security problems and risks to these unpatched systems. In 2017 the WannaCry malware exploited a vulnerability in the Windows operating system server message block (SMB) protocol. WannaCry included a replication mechanism that allowed it to scan the network for vulnerable systems, attack those vulnerabilities, and upon a successful attack install malware (in this case, ransomware) and in turn, use that machine to infect others. WannaCry affected hundreds of thousands of systems worldwide. Microsoft knew about this specific vulnerability and released a software update over a month prior, but many organizations did not have an adequate vulnerability management program to ensure the patch was applied in time.

Developing a disciplined vulnerability management program will protect your devices and systems from malware like these. A proactive program may also lower risks to your systems by avoiding the inevitable frenzy of patching emergency activities when malware like these do hit. Put quite simply, a solid vulnerability management program will provide a critical foundation of good security hygiene. And if you develop software, it is especially important to put into place best practices to lower the chance of vulnerabilities in your own code and lower the overall count of bugs that would need future patches.

Software manufacturers release security updates to correct discovered vulnerabilities in their code. These vulnerabilities may be simple software bugs that an attacker can exploit to access the entire device or system. Many software manufacturers follow a formal security development lifecycle (SDL) process to reduce the introduction of vulnerabilities into their code. The SDL may call out specific tools and procedures for reviewing code to find vulnerabilities including threat modeling, fuzzing for malformed inputs, and processing all source code through specific tools that scan for problems that could result in a vulnerability. In addition, outside security researchers often partner with software manufacturers to find these vulnerabilities before the bad guys do. The open source and crowd sourcing community enroll larger audiences and organize bug hunts to find vulnerabilities in software. Once the bug is found, it can be analyzed by the software manufacturer for the best corrective fix, which often comes in the form of a software patch. A software patch is typically downloaded from the manufacturer and applied to the affected system in the form of a security update. Additionally, the manufacturer may recommend mitigation controls that users can put into place until they apply the update. A mitigation control could include blocking a network port at a firewall or disabling a service on the vulnerable device.

Taking the time to understand and document what software and hardware you have as well as the steps for how to update them are important first steps to developing a vulnerability management program. It’s analogous to practicing changing your tire before getting a flat. Knowing where the tools are and generally how the tools work is a lot easier in your garage than the side of the road! 

Taking Inventory of All the Hardware and Software

Begin with taking inventory of all the hardware and software running on your network or under your control. Large hardware and software companies like Microsoft and Apple have developed very sophisticated security update notification and installation processes, and they have made inventory collection easier with some of their enterprise management tools. But these tools have limitations and might only recognize certain types of devices and servers. The influx of embedded devices means that there is likely a much higher diverse set of devices on your network and not all of these may be discoverable by your enterprise tools. Because of this, it is not always easy to know whether your device is up-to-date or even where to go to get security updates for your specific components. Therefore, it is important to take a good inventory (manually if necessary or for smaller networks), as a basis for collecting subsequent support information. For each of your devices record their make, model, and their function. Visit the manufacturer website and look in their technical support or downloads section for evidence of software updates. For embedded devices, these might be in the form of new firmware. Remember also that security updates must be applied in both the operating system as well as any applications that run on the device—so don’t leave those out. Include the operating system, firmware, and vendor URL for software updates in your inventory database as well.

Understanding How to Install and Verify Security Updates

Once you have a solid inventory, be sure to understand how to install and verify the actual security updates. This process might vary by device or software manufacturer. Windows has specific system settings to manage security updates and the operating system takes care of most of the internal dependency checking, download, and install process leaving the user to simply click to install. In most software applications, the developer will include a configuration item to check for and apply new updates. Modern applications often take care of this behind the scenes and will prompt for the installation of new updates on launch or exit of the application. For more simple devices, you might be required to download an update to another device, transfer the update to the vulnerable device via TFTP, HTTP or USB memory stick, and apply with very specific instructions. Depending on the nature of the software patch, the system might need to be restarted which requires an understanding of how people use that system and what other dependencies other systems have on this system. I believe many organizations fear the reboot more than applying the update itself. This fear comes from not fully understanding dependencies and stability of their critical systems. Exercising these processes regularly and applying security updates helps assuage these fears.

Subscribe to distribution lists for when new security updates are available from manufacturers. Sometimes these lists are not public facing, and you might need to ask your manufacturer to be added to the security update announcements for their product. Receiving notification directly from the manufacturer might give you additional time to plan for an update rollout days or weeks ahead of broader announcements.

Triaging New Security Updates and Assigning Deadlines

Develop a process for triaging new security updates and assign deadlines. Many companies will often incorporate these requirements into their security policy. For example, all severity-1 security updates must be installed within 24 hours, and lower severity updates must be installed within 30 days with appropriately accelerated procedures for testing and staged rollout. Conditions that could elevate a severity-1 update include recommendations from the manufacturer, whether an exploit is already in the wild, or simply what an attacker could do with this vulnerability. Document these policies and procedures and make sure your operators and developers understand where to find them and what is expected of them.

Keeping Everyone Accountable

Lastly, keep everyone—or at least yourself—accountable. Set aside time monthly to review your security update posture and measure how many updates have been applied. If you are behind, plan how to catch up. Review metrics that show the percentage of vulnerable systems month over month.

Applying security updates regularly to all your computing systems is an essential first step in protecting those systems. In addition to fundamentally helping keep the attackers at bay, a comprehensive vulnerability management program pays dividends in improvements to other IT process areas as well—from inventory to dependency management.