February 2, 2009

Tuning Basics

When you run into performance problems, it's easy to add network equipment, servers, or bandwidth to cope with the load. However, you will often need to fix the issues without spending additional capital. This means tuning. Unfortunately, many websites on tuning simply give specific values without any explanation, such as a set of sysctl values for Linux or a bunch of registry keys for Windows. These values are often tuned to the poster's hardware and environment, and may even introduce new and obscure problems if you apply them without knowing what they do and how they apply to your environment.

So why am I writing a blog on tuning, then? Well, I would like to present techniques for identifying bottlenecks and tuning your systems correctly. I will also provide specific tools for monitoring and testing performance where applicable. Finally, I will list specific tuning options at times (mostly so I can find them again), but I will always preface these with an explanation, including the details of my environment and system specs. 

Let's start with the basic rules of system tuning:

  • Identify the problem first
  • Only tune if it will actually help (For example, tuning can't fix a bad switch port. Troubleshoot and eliminate obvious problems first.) 
  • Don't ever tune a live server (If you have an emergency, clone the box to a VM or spare box instead)
  • Back up the current system state and all data before you touch anything
  • Make a performance baseline
  • Make a performance goal
  • Document everything
  • Back up the current system state and all data before you touch anything
  • Write down what you change when you change it
  • Classify results in multiple categories - keep track of what hurts, what helps, and what doesn't affect performance
  • Make regular backups (or better, snapshots) as you progress so you can revert
  • Test each change separately
  • Test in a pre-production environment before going live - don't assume it will work in your datacenter because it works in your lab.

Check back often (I will try to post at least once a week), and if there's a specific question you have or benchmarks you'd like to see, let me know and I'll do what I can to help.

"Tuning Basics" continued here...