How to Establish a Process for Patch Management

Patch management is a crucial element of any organization's security initiative. Identifying hot fixes, and testing and applying patches to client and server operating systems can pose significant challenges.

The following are some tips to ease the process and minimize the risks involved in updating mission-critical systems.

Keeping Windows Up to Date

Using an automated tool is the easiest way to ensure that your servers and PCs have all the latest patches installed. Built-in functionality in Windows can be used for this purpose; also, some free tools are available from Microsoft. Windows Update is a consumer technology that’s integrated into Vista and Windows 7, and it can be set to automatically scan and deploy missing security patches for Windows and other Microsoft software. But it doesn’t provide the control and reporting features needed by most school districts.

The Microsoft Baseline Security Analyzer (MBSA) is a free download that scans PCs and servers on a network for configuration anomalies and missing security patches.

Windows Server Update Services (WSUS) builds on the technology used in Windows Update to provide a local server where system administrators can centralize patch management and download, approve and report on the status of systems across the network. WSUS can be downloaded for Windows Server 2003 or installed as a server role on Windows Server 2008 R2, and either installed on a single server or as part of a tiered infrastructure. WSUS allows sysadmins to target groups of machines, which is useful for applying updates to a limited set of devices for testing purposes.

System Center Configuration Manager (SCCM) is designed for large organizations that need more control and, again, builds on the technology WSUS provides for better reporting, targeting and deploying software, among a host of other useful features.

Figure 1 – Microsoft Baseline Security Analyzer in Action

Risk Assessment

Microsoft releases patches monthly and categorizes them as either “critical” or “important.” You can sign up for the Microsoft Security Bulletin, which is published a week before patch day and provides information about the nature of the vulnerabilities to be patched, the affected software and whether it’s necessary to reboot after installation, which is especially relevant for servers.

Figure 2 – Microsoft Advance Security Bulletin April 2012

The risk associated with each vulnerability must be established and a decision made on whether to patch systems; if so, the level of urgency also must be determined. It’s worth bearing in mind that the window for patching is becoming shorter as exploits are sometimes developed within hours of vulnerability information being made public. It’s generally considered that critical systems should be patched within 72 hours, but don’t discount the importance of keeping desktop PCs up to date.

Test, Test and Test Again

It’s vital to test patches in a lab or pre-production environment to determine if they might cause an outage on production servers. A test lab should mirror your production servers as closely as possible, and should be used to check that functionality isn’t affected after a patch is installed. (At the most basic level, make sure that servers reboot after patch installation.) Testing is easier if you’ve already established a baseline configuration for your servers; for example, documenting how they should be configured.

IT staff can test patches on their own PCs. While this is adequate for basic testing, don’t forget to test patches on a variety of devices, including notebooks. Conduct this testing in different departments because operating systems and software will vary, and the impact of a patch can only be fully assessed if distributed to a wide sample of users.

A comprehensive testing procedure will involve more than just installing a patch and making sure a system still boots. Scripts can be run on servers to perform a series of tests that confirm vital functionality hasn’t been affected by the update. PowerShell or VBscripts can run through the tests automatically to accelerate the process and generate reports. For instance, which Windows services should run in your baseline configuration? After a patch is installed and the server rebooted, are all of the required services still running? Do you have basic network connectivity?

Use PowerShell to get a list of running services and compare the output to your baseline:

GET-SERVICE | WHERE-OBJECT {$_.STATUS -EQ "RUNNING"} | OUT-GRIDVIEW

The OUT-GRIDVIEW commandlet is optional, but it can be useful as it pipes the results of GET-SERVICE to a separate window where additional filtering and display options are easily accessible via the graphical interface.

Figure 3 – Running Services Enumerated by PowerShell

Check the TCP/IP stack using the TEST-CONNECTION commandlet:

TEST-CONNECTION –COMPUTERNAME 127.0.0.1

You can create a PowerShell script to perform more sophisticated network checks, such as pinging known server names and DNS name servers specified in the network adapter settings. Microsoft Exchange comes with a useful set of PowerShell commandlets for testing basic functionality, including:

TEST-MAPICONNECTIVITY
TEST-EXCHANGESEARCH
TEST-OWACONNECTIVITY
TEST-WEBSERVICESCONNECTIVITY
TEST-POPCONNECTIVITY
TEST-IMAPCONNECTIVITY

Checks can also be run against Microsoft SQL server using PowerShell; for example, pinging the host and testing a SQL connection.

A preproduction or lab environment is a luxury for some districts, so a tiered rollout also can be part of the testing phase, in which a patch is deployed to a limited number of users (or servers) before being installed across an entire organization. Remember that in the testing phase, you are not only attempting to determine any detrimental effects that a patch might have on your production environment, but also testing your chosen patch distribution technology.

Deploying Updates

To deploy updates, back up all your servers before patches are installed and ensure that a restore operation has been successfully tested. Install updates on least critical servers first, and check thoroughly that there are no problems by observing event logs and help desk calls and by running the scripts used in the testing phase. Once you’re sure the patches aren’t causing any problems on these devices, continue to deploy patches to critical servers. Continue monitoring the environment for problems after all patches have been successfully deployed.