Having an established and well-organized patch management program is absolutely vital in helping an organization maintain good system hygiene and a strong cybersecurity posture. Well-developed procedures, robust deployment and monitoring tools, and good analysis methodologies, and precise documentation will go a long way towards enabling an organization to achieve and maintain a healthy information security posture.
Centrally managed patching systems, such as Lumension Security, IBM BigFix, and Microsoft System Center, are a big part of this process, but those systems are but one of the types of tools that will facilitate effective patch management. Unfortunately, this tool is not a “set it and forget it” type of deal. The organization also needs to have a good change management program and a good program for performing meaningful risk assessments. Processes and tools for regular status reporting, patch status analysis, and a vulnerability scanning program to evaluate and confirm that security patches and updates have been applied will be key to providing verification and validation for the patch management program as well.
A patching and vulnerability management program will serve as a vehicle for tool and process organization and will help to maintain consistency and standards-based compliance methodologies. In my organization (we are a large enterprise), we have what is known as a Patching and Vulnerability Working Group (PVWG) that meets regularly to discuss patching and vulnerability issues, updates about new patches being released, prioritizing patches and deployment timing, and discuss lessons learned from previous patch deployment cycles. We also conduct training in report analysis and tool usage. This PVWG is made up of IT specialists who represent the different business groups in the organization as well as members of the patching team, change advisory board, vulnerability scanning team, the information assurance team, and other customer support people. We meet at least monthly, and more frequently if there are any emergent patching issues that need to be discussed.
As far as the patching process itself, having a defined schedule of events will help to ensure that all aspects of the patching program are always followed the same way. In my organization, we always follow some general timelines for our typical enterprise patching process from the time that patches are released to actual deployment, and including some activities that are continuous throughout the patching cycle.
The outline below lists the time-lines for patch receipt and deployment for normal monthly patches received from the vendors. Out-of-Band, or emergency patches, typically undergo an expedited testing and deployment process.
- Ongoing: Continually monitor US-CERT, the patchmanagement.org listserv, and other vendor notices for information about new patches.
- Ongoing: Perform vulnerability scans on a regular basis.
- New Microsoft and other vendors’ patches are usually released to the general public on the second Tuesday (2T) of each month.
- 2T + 1 day: New patches are typically received by the subscription services on the organization’s centralized patching system servers. (Note: In a large environment, it normally takes a few days for all computers to check in and be evaluated for patch applicability).
- 2T + 3 days: Testing on non-production desktop computers and test servers begins.
- 2T + 6 days: Pilot group production patch testing begins on selected pilot computers within the organization.
- 2T + 9 days: Risk assessments are performed, testing issues are discussed and resolved, and patch deployment prioritization and timing recommendations are made to the Patch Management Team.
- 2T + 9 days: Patch Management Team submits Request for Change (RFC) using the organization’s established change management procedures. (Note: In my organization, this is considered a “standard” change that has already been preapproved by our change advisory board).
- 2T + 10 days: Patch Management Team sends broadcast messages to customers to announce patch deployments and any system outages that will take place.
- 2T + 13 days: Begin deploying patches to the production environment.
- 2T + 16 days: Deploy “Therapeutic Reboot” to all computers. This helps clear what is known as a “dirty state” that computers are in when they receive patches but are not yet rebooted. This also helps to put computers into a fresh state to help with better performance.
- 2T + 28 +/- days: Analyze patch status and vulnerability scan reports to determine patching effectiveness and identify gaps.
- 2T + 30 days: Patch Management Team adds patches from previous month to a mandatory baseline in the centralized patching system to ensure that computers that infrequently check-in can receive the required patches and updates.
Smaller businesses are not necessarily going to have the resources or the need to have large change management programs, separate patching and vulnerability teams, or even an established working group. But the need to be aware of new security patches, testing the patches to ensure that business processes are not broken by patch deployments, troubleshooting and resolving issues, and then regularly applying the patches are still crucial for any sized business to maintain a secure computing environment. The smaller IT staffs will need to wear many hats in addition to their normal support functions, and patching/vulnerability management will need to be a high priority on their list.
A word about documentation: If you don’t document it, it didn’t happen. Also, if you don’t document it, it is more difficult to figure out what changes took place in the event that troubleshooting business disruption issues need to be performed. If you don’t document lessons learned, it is much more difficult to make meaningful improvements to the information security program. In my PVWG group that I mentioned above, for example, we always keep meeting minutes that contain lists of patches deployed, current patch statuses, and any break/fix issues that our members discuss. We also record lessons learned and new ideas for improvements. Patching status trend analysis is performed to help illustrate patching program performance. Not only do our patch status reports show statuses for individual patches, but also information about which patches are causing the largest gaps in our patching program. Documentation is also a key component in helping the organization conduct cybersecurity self-assessments, and/or will help with the evidence gathering activities with third-party audits.
Having an established patching program with consistent timelines and processes will help the organization to meet critical patching timelines. This will be crucial in ensuring that business processes are disrupted as little as possible while maximizing the organization’s security posture. An established working group that meets regularly to discuss security patching issues will help everyone stay informed about patch releases, risk analysis, testing results, and patch deployment effectiveness. And finally, documentation will ensure that the organization has accurate records of patching program activities which will help illustrate and overall patching program, and will also assist in any security self-assessments or audits later on.
NIST SP 800-40 Rev 3 “Guide to Enterprise Patch Management Technologies”
https://csrc.nist.gov/publications/detail/sp/800-40/rev-3/final
Patch Management Listserv:
http://www.patchmanagement.org/
United States Computer Emergency Readiness Team (US-CERT):
https://www.us-cert.gov/
ITL Bulletin: “Creating a Program to Manage Security Patches and Vulnerabilities: NIST Recommendations for Improving System Security”
https://csrc.nist.gov/publications/detail/itl-bulletin/2006/02/creating-a-program-to-manage-security-patches-and-vulnerabilitie/final
