System Analysis and Tuning Guide

Publication Date 19 Dec 2011

Copyright © 2006– 2011 Novell, Inc. and contributors. All rights reserved.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled GNU Free Documentation License.

For Novell trademarks, see the Novell Trademark and Service Mark list Linux* is a registered trademark of Linus Torvalds. All other third party trademarks are the property of their respective owners. A trademark symbol (®, ™ etc.) denotes a Novell trademark; an asterisk (*) denotes a third party trademark.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither Novell, Inc., SUSE LINUX Products GmbH, the authors, nor the translators shall be held liable for possible errors or the consequences thereof.


About This Guide
1. Available Documentation
2. Feedback
3. Documentation Conventions
4. About the Making of This Manual
5. Source Code
6. Acknowledgments
I. Basics
1. General Notes on System Tuning
1.1. Be Sure What Problem to Solve
1.2. Rule Out Common Problems
1.3. Finding the Bottleneck
1.4. Step-by-step Tuning
II. System Monitoring
2. System Monitoring Utilities
2.1. Multi-Purpose Tools
2.2. System Information
2.3. Processes
2.4. Memory
2.5. Networking
2.6. The /proc File System
2.7. Hardware Information
2.8. Files and File Systems
2.9. User Information
2.10. Time and Date
2.11. Graph Your Data: RRDtool
3. Monitoring with Nagios
3.1. Features of Nagios
3.2. Installing Nagios
3.3. Nagios Configuration Files
3.4. Configuring Nagios
3.5. Troubleshooting
3.6. For More Information
4. Analyzing and Managing System Log Files
4.1. System Log Files in /var/log/
4.2. Viewing and Parsing Log Files
4.3. Managing Log Files with logrotate
4.4. Monitoring Log Files with logwatch
III. Kernel Monitoring
5. SystemTap—Filtering and Analyzing System Data
5.1. Conceptual Overview
5.2. Installation and Setup
5.3. Script Syntax
5.4. Example Script
5.5. For More Information
6. Kernel Probes
6.1. Supported Architectures
6.2. Types of Kernel Probes
6.3. Kernel probes API
6.4. Debugfs Interface
6.5. For More Information
7. Perfmon2—Hardware-Based Performance Monitoring
7.1. Conceptual Overview
7.2. Installation
7.3. Using Perfmon
7.4. Retrieving Metrics From DebugFS
7.5. For More Information
8. OProfile—System-Wide Profiler
8.1. Conceptual Overview
8.2. Installation and Requirements
8.3. Available OProfile Utilities
8.4. Using OProfile
8.5. Using OProfile's GUI
8.6. Generating Reports
8.7. For More Information
IV. Resource Management
9. General System Resource Management
9.1. Planning the Installation
9.2. Disabling Unnecessary Services
9.3. File Systems and Disk Access
10. Kernel Control Groups
10.1. Technical Overview and Definitions
10.2. Scenario
10.3. Control Group Subsystems
10.4. Using Controller Groups
10.5. For More Information
11. Power Management
11.1. Power Management at CPU Level
11.2. The Linux Kernel CPUfreq Infrastructure
11.3. Tuning Options for P-states
11.4. Tuning Options for C-states
11.5. Creating and Using Power Management Profiles
11.6. Monitoring Power Consumption with powerTOP
11.7. Troubleshooting
11.8. For More Information
V. Kernel Tuning
12. Installing Multiple Kernel Versions
12.1. Enabling Multiversion Support
12.2. Installing/Removing Multiple Kernel Versions with YaST
12.3. Installing/Removing Multiple Kernel Versions with zypper
13. Tuning Per-Device I/O Performance
13.1. I/O Scheduler -- /sys/block/<device>/queue/scheduler
13.2. I/O Barrier Tuning
14. Tuning the Task Scheduler
14.1. Introduction
14.2. Process Classification
14.3. O(1) Scheduler
14.4. Completely Fair Scheduler
14.5. For More Information
15. Tuning the Memory Management Subsystem
15.1. Memory Usage
15.2. Reducing Memory Usage
15.3. Virtual Memory Manager (VM) Tunable Parameters
15.4. Non-Uniform Memory Access (NUMA)
15.5. Monitoring VM Behavior
16. Tuning the Network
16.1. Configurable Kernel Socket Buffers
16.2. Detecting Network Bottlenecks and Analyzing Network Traffic
16.3. Netfilter
16.4. For More Information
VI. Handling System Dumps
17. Tracing Tools
17.1. Tracing System Calls with strace
17.2. Tracing Library Calls with ltrace
17.3. Debugging and Profiling with Valgrind
17.4. For More Information
18. Kexec and Kdump
18.1. Introduction
18.2. Required Packages
18.3. Kexec Internals
18.4. Basic Kexec Usage
18.5. How to Configure Kexec for Routine Reboots
18.6. Basic Kdump Configuration
18.7. Analyzing the Crash Dump
18.8. Advanced Kdump Configuration
18.9. For More Information
A. GNU Licenses
A.1. GNU General Public License
A.2. GNU Free Documentation License

List of Figures

2.1. iptraf Running in Interactive Mode
2.2. Example Graph Created with RRDtool
7.1. Architecture of perfmon2
8.1. GUI for OProfile
10.1. Resource Planning
12.1. The YaST Software Manager - Multiversion View
18.1. YaST2 Kdump Module - Start-Up Page

List of Tables

7.1. Supported Processors
7.2. Needed Packages
7.3. Read-Only Files in /sys/kernel/debug/perfmon/cpu*/
11.1. C-States
18.1. Recommended Values for Additional Kernel Command Line Parameters

List of Examples

2.1. vmstat Output on a Lightly Used Machine
2.2. vmstat Output on a Heavily Used Machine (CPU bound)
3.1. A Host Object Definition
3.2. A Service Object Definition
3.3. A Contact and Contactgroup Definition
4.1. Example for /etc/logrotate.conf
5.1. Simple SystemTap Script
5.2. Probe with Timer Event
5.3. printf Function with Format Specifiers
5.4. Using Global Variables
5.5. Monitoring Incoming TCP Connections with tcp_connections.stp
11.1. Example powerTOP Output