So you want to make your Windows network more secure by “hardening” it against attack. Or maybe you want to tweak Windows to make your servers perform better. Or maybe you want to develop a custom application that takes advantage of some cool Windows “hack” you’ve read about. Should you proceed?

What many administrators don’t know is that some steps they take to harden their systems, eke out performance gains or take advantage of undocumented registry hacks might have a downside. They can place your systems and even your network into an unsupported state. What does this mean? It means that when things start going wrong and you call Microsoft Customer Support Services (CSS) for assistance and tell the support engineer what you’ve done, the engineer replies, “Sorry, you’re out of luck — what you’ve done has placed your server into an unsupported state. You need to flatten and rebuild.” Yikes!

Let’s look at five ways you can break your Windows network and place it into an unsupported state. And by “unsupported state” I don’t mean only that CSS may have difficulty supporting you when you have trouble. I also mean you as an administrator might find it difficult to support the network you’ve built with your own hands.

Hardening your network so much that it becomes brittle and cracks start to appear

Everyone is concerned about security these days, especially network administrators. But it’s easy to become too concerned about network security, and an obsession with hardening your systems can quickly lead to an unsupportable condition. Here’s an example: You’re scratching your head wondering why Outlook Express is installed by default on Windows servers. “Who needs a mail client on a server?” you ask. So you nuke OE in an effort to reduce your attack surface and sit back, happy that you’ve just improved the security of your servers. Then things start going wrong and you discover that Microsoft Operations Manager is no longer sending you e-mail alerts from your servers. Could there be a connection?

Here’s another example of how overzealous hardening can cause undesirable things. You notice one day that print queues are no longer being scavenged by Active Directory. What’s going on? After a long talk with support engineers, you discover you shouldn’t have disabled the Print Spooler service on your domain controllers. “But my domain controllers aren’t running the Print Server role,” you say. “Therefore, I thought the Spooler service wasn’t needed on them, so I disabled it.”

Once again, your zeal for security has gotten you into trouble. This time the support engineer helped you determine the problem — next time you might not be so lucky. For example, the quickest way to thoroughly mess up your Windows servers by hardening them is to modify the default Access Control lists on your system direc­tories. You wouldn’t believe how often people try this. Don’t do it.

Developing applications that break when the next version of Windows comes out

You’re proud that you developed a custom application for your organization that uses an undocumented registry setting to streamline the work flow between Active Directory and some line-of-business application. Then Service Pack X for Windows comes out, and when you apply it, users complain the application no longer works.

What’s going on? Some digging reveals that the problem is caused by a change in the way the registry setting stores data. Your initial reaction is to blame Microsoft: “How dare they change the registry setting without telling me!” But calm reflection tells you that you shot yourself in the foot.

Undocumented registry settings are undocumented for a reason. They’re hidden implementation details that are subject to change without notice. And that goes not just for the registry but also for Win32 and .NET application programming interfaces (APIs), too. Just because you’ve figured out how to make an API do something it’s not documented to do doesn’t mean you should use your discovery in an application your organization depends on. If you want to write apps that last, use only documented APIs and registry settings because these are the only ones that Microsoft officially supports.

Making your deployment so complex it becomes almost unmanageable

Group Policy is a Windows administrator’s friend, but not if you make things too complicated. Sure, Group Policy features such as Block Inheritance, Enforced, Loopback, Security Filtering, WMI Filters and so on give you a lot of flexibility in how you ensure or prevent specific policies from applying to particular users or groups in different situations, but did you know all this flexibility can make your deployment so flexible that it breaks?

“Hmm, why isn’t this policy applying to that user?” you ask as you stare at dozens of Group Policy Objects displayed in a report generated by the Group Policy Results Wizard.

Domains are another example of complexity that can become unmanageable. Does each unit really need its own domain? Are multilevel domain trees really needed in an institution your size? You’d be surprised how many large (even very large) organizations get by with deploying a single domain in their Active Directory forest. The moral here: Build a network you’ll be able to support.

Using products and tools that are grossly inappropriate for the size and scope of your organization

Complexity is also a factor in the tools you choose to get the job done. For instance, deploying Microsoft Systems Center Configuration Manager would probably be overkill if your network consisted of only 75 machines. On the other hand, using the Microsoft Solution Accelerator for Business Desktop Deployment (BDD) for deploying Vista to 5,000 desktops across an organization would also probably be inappropriate. BDD is more suited for midsize networks than large institutions.

You’ll quickly tie yourself in knots if you don’t choose the right tool for the job, and your support costs can escalate quickly if you’re not on the ball. And just in case you weren’t sure, so-called unsupported tools such as the Windows Server 2003 Resource Kit Tools really are unsupported. Use them at your own risk, and don’t expect CSS to provide you with an easy out should you foul up your network using them.

Tweaking performance to eke out more gains until all hell breaks loose

I’ve seen this kind of thing again and again, especially lately with Windows Vista. For example, you’re not happy with how the indexing service works in Windows Vista so you start messing around with its configuration to improve things, and soon it’s not working at all. Or you’re unhappy with slow file transfers for large files and you start modifying TCP/IP registry settings to tune network performance, and in no time your machine can’t see the network at all.

Let the techies at Microsoft have a go at fixing bugs instead of trying to work around them using tweaks and hacks.

Mitch Tulloch is a Microsoft Most Valuable Professional and was the lead author for Windows Vista Resource Kit from Microsoft Press. You can reach Mitch at mktulloch@mtit.com.