First things first …

A few common things to possibly turn on, install, or enable:

• APC (Another PHP Cache)
• Memcached

MySQL Optimization

The easiest way (and the quickest) to determine what can be done to optimize MySQL is to use a script called mysqltuner.pl. The really interesting thing about this script is that to get it you simply run the following:

wget mysqltuner.pl

After you’ve gotten this script run it with your perl interpreter:

perl mysqltuner.pl

It will prompt you for your username and password for mysql and then print out a nice report outlining how your mysql daemon has been running. Here’s an example of this output:

>> MySQLTuner 1.0.1 – Major Hayden
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with ‘–help’ for additional options and output filtering
Please enter your MySQL administrative login: root
Please enter your MySQL administrative password:

——– General Statistics ————————————————–
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.90-log
[OK] Operating on 32-bit architecture with less than 2GB RAM

——– Storage Engine Statistics ——————————————-
[--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 28M (Tables: 96)
[--] Data in InnoDB tables: 1M (Tables: 55)
[!!] Total fragmented tables: 2

——– Performance Metrics ————————————————-
[--] Up for: 1h 39m 49s (24K q [4.163 qps], 354 conn, TX: 9M, RX: 34M)
[--] Reads / Writes: 60% / 40%
[--] Total buffers: 148.6M global + 960.0K per thread (100 max threads)
[OK] Maximum possible memory usage: 242.3M (11% of installed RAM)
[OK] Slow queries: 0% (0/24K)
[OK] Highest usage of available connections: 28% (28/100)
[OK] Key buffer size / total MyISAM indexes: 64.0K/6.3M
[OK] Key buffer hit rate: 99.6% (164K cached / 682 reads)
[OK] Query cache efficiency: 81.3% (14K cached / 17K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 463 sorts)
[!!] Joins performed without indexes: 42
[OK] Temporary tables created on disk: 12% (54 on disk / 417 total)
[OK] Thread cache hit rate: 70% (106 created / 354 connections)
[!!] Table cache hit rate: 2% (48 open / 2K opened)
[OK] Open file limit used: 9% (95/1K)
[OK] Table locks acquired immediately: 98% (7K immediate / 7K locks)
[!!] Connections aborted: 7%
[OK] InnoDB data size / buffer pool: 1.0M/4.0M

——– Recommendations —————————————————–
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours – recommendations may be inaccurate
Adjust your join queries to always utilize indexes
Increase table_cache gradually to avoid file descriptor limits
Your applications are not closing MySQL connections properly
Variables to adjust:
join_buffer_size (> 128.0K, or always use indexes with joins)
table_cache (> 48)

The first thing to notice is that mysql should be running for 24 to 48 hours before you trust the results of this script. The second thing to acknowledge is that it isn’t omniscient. Understanding how these parameters affect your memory footprint and your runtime efficiency are what we’re most interested in though.

Glossary of Parameters

The best source of my.cnf parameters is of course the MySQL documentation, but I’ll clarify a few of the more pertinent items from the above output.

• Maximum Possible Memory – This is not the actual hard limit on your memory usage but a calculated theoretical maximum based on the parameters in your my.cnf and if this is over ~90% you may use your swap more than you want to. After making any changes I suggest you check this to make sure you haven’t over-allocated yourself.
• Highest Usage of Available Connections – This is the historical max connections used by mysql and if it’s at 100% this isn’t a guarantee that you need to increase max_connections but may just indicate that your connections spiked once. That being said I recommend following its advice if you have the memory for it.
• Query Cache Efficiency – The hit rate for the cache on your queries (I hope you have enabled). As with any hash table if you’re not hitting what’s in the top bucket you need to increase the number of buckets.
• Temporary Tables Created on Disk – This can be a finicky parameter to get right and is the first place I would recommend cutting if you don’t have the memory to run mysql like you would want, but if you do have the memory then up this as far as you can within reason.
• Thread Cache Hit Rate – Just like any other cache.
• Table Cache Hit Rate – Just like any other cache.

Fragmented Tables

If you noticed the fragmented tables along with the recommendation to run OPTIMIZE TABLE that’s not something that will typically slow mysql down noticeably with today’s disks, but if you want to defragment them I recommend reading this article on the topic.

Apache Tweaking

The following parameters are usually in httpd.conf unless you run a distribution that organizes apache in an easier to work with manner (my preference of course is Gentoo which stores these parameters in: 00_mpm.conf).

The parameters for tweaking apache’s processes depend on which worker module you have in use. The following is the pertinent sections from my configuration:

#prefork MPM
# This is the default MPM if USE=-threads
#
# MinSpareServers: Minimum number of idle child server processes
# MaxSpareServers: Maximum number of idle child server processes

StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 50
MaxRequestsPerChild 500

# worker MPM
# This is the default MPM if USE=threads
#
# MinSpareThreads: Minimum number of idle threads available to handle request spikes
# MaxSpareThreads: Maximum number of idle threads
# ThreadsPerChild: Number of threads created by each child process

StartServers 2
MinSpareThreads 15
MaxSpareThreads 50
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 1000

These parameters are pretty self explanatory but just for completeness’ sake I’ll do a small recap on these here:

* StartServers – The number of servers to have running and handling connections at first launch.
* MinSpareServers – The minimum number of servers to have running not currently handling connections.
* MaxSpareServers – The maximum number of servers to have running not currently handling connections.
* MaxClients – The maximum number of clients that can simultaneously connect to Apache.
* MaxRequestsPerChild – The maximum number of requests that a child will respond to before terminating.

Using these parameters we can control the amount of memory that Apache will consume at the cost of other things we may want to have like lots of connections or many sequential requests.

1. query_cache_size:
   • MySQL 4 provides one feature that can prove very handy – a query cache. In a situation where the database has to repeatedly run the same queries on the same data set, returning the same results each time, MySQL can cache the result set, avoiding the overhead of running through the data over and over and is extremely helpful on busy servers.
2. key_buffer_size:
   • The value of key_buffer_size is the size of the buffer used with indexes. The larger the buffer, the faster the SQL command will finish and a result will be returned. The rule-of-thumb is to set the key_buffer_size to at least a quarter, but no more than half, of the total amount of memory on the server. Ideally, it will be large enough to contain all the indexes (the total size of all .MYI files on the server).
   • A simple way to check the actual performance of the buffer is to examine four additional variables: key_read_requests, key_reads, key_write_requests, and key_writes.
   • If you divide the value of key_read by the value of key_reads_requests, the result should be less than 0.01. Also, if you divide the value of key_write by the value of key_writes_requests, the result should be less than 1.
3. table_cache:
   • The default is 64. Each time MySQL accesses a table, it places it in the cache. If the system accesses many tables, it is faster to have these in the cache. MySQL, being multi-threaded, may be running many queries on the table at one time, and each of these will open a table. Examine the value of open_tables at peak times. If you find it stays at the same value as your table_cache value, and then the number of opened_tables starts rapidly increasing, you should increase the table_cache if you have enough memory.
4. sort_buffer:
   • The sort_buffer is very useful for speeding up myisamchk operations (which is why it is set much higher for that purpose in the default configuration files), but it can also be useful everyday when performing large numbers of sorts.
5. read_rnd_buffer_size:
   • The read_rnd_buffer_size is used after a sort, when reading rows in sorted order. If you use many queries with ORDER BY, upping this can improve performance. Remember that, unlike key_buffer_size and table_cache, this buffer is allocated for each thread. This variable was renamed from record_rnd_buffer in MySQL 4.0.3. It defaults to the same size as the read_buffer_size. A rule-of-thumb is to allocate 1KB for each 1MB of memory on the server, for example 1MB on a machine with 1GB memory.
6. thread_cache:
   • If you have a busy server that’s getting a lot of quick connections, set your thread cache high enough that the Threads_created value in SHOW STATUS stops increasing. This should take some of the load off of the CPU.
7. tmp_table_size:
   • “Created_tmp_disk_tables” are the number of implicit temporary tables on disk created while executing statements and “created_tmp_tables” are memory-based. Obviously it is bad if you have to go to disk instead of memory all the time.

The following tutorial describes the steps to optimize the performance of a MySQL database with the mysqltuner script.

Login to your server on the shell, then execute the following commands:

Download the mysqltuner script:

cd /usr/local/bin
wget http://mysqltuner.pl/mysqltuner.pl
chmod +x mysqltuner.pl

Run mysqltuner

/usr/local/bin/mysqltuner.pl

Then enter root as username and the mysql root password.

You will get a output similar to this:

root@v221:/usr/local/bin# /usr/local/bin/mysqltuner.pl

>>  MySQLTuner 1.0.1 – Major Hayden <major@mhtx.net>
>>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
>>  Run with ‘–help’ for additional options and output filtering
Please enter your MySQL administrative login: root
Please enter your MySQL administrative password:

——– General Statistics ————————————————–
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.51a-24+lenny2
[!!] Switch to 64-bit OS – MySQL cannot currently use all of your RAM

——– Storage Engine Statistics ——————————————-
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 26M (Tables: 215)
[!!] InnoDB is enabled but isn’t being used
[!!] Total fragmented tables: 33

——– Performance Metrics ————————————————-
[--] Up for: 96d 23h 3m 41s (10M q [1.239 qps], 686K conn, TX: 701M, RX: 1B)
[--] Reads / Writes: 44% / 56%
[--] Total buffers: 58.0M global + 2.6M per thread (100 max threads)
[OK] Maximum possible memory usage: 320.5M (12% of installed RAM)
[OK] Slow queries: 0% (20/10M)
[OK] Highest usage of available connections: 33% (33/100)
[OK] Key buffer size / total MyISAM indexes: 16.0M/8.5M
[OK] Key buffer hit rate: 99.9% (57M cached / 30K reads)
[OK] Query cache efficiency: 78.6% (5M cached / 6M selects)
[!!] Query cache prunes per day: 483
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 408K sorts)
[!!] Temporary tables created on disk: 36% (269K on disk / 745K total)
[OK] Thread cache hit rate: 99% (427 created / 686K connections)
[!!] Table cache hit rate: 2% (64 open / 3K opened)
[OK] Open file limit used: 11% (120/1K)
[OK] Table locks acquired immediately: 99% (3M immediate / 3M locks)

——– Recommendations —————————————————–
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
Run OPTIMIZE TABLE to defragment tables for better performance
Enable the slow query log to troubleshoot bad queries
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
query_cache_size (> 16M)
tmp_table_size (> 32M)
max_heap_table_size (> 16M)
table_cache (> 64)

The script recommends to adjust or add the following variables in the mysql my.cnf file. The location of my.cnf is normally /etc/my.cnf or /etc/mysql/my.cnf depending on the Linux distribution that is installed on your server.

Open the my.cnf file:

vi /etc/mysql/my.cnf

and increase or set the variables in the [mysqld] section of the file. Mine looks now like this:

[mysqld]
#
# * Basic Settings
#
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
language        = /usr/share/mysql/english
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            = 127.0.0.1
#
# * Fine Tuning
#
key_buffer              = 16M
max_allowed_packet      = 16M
thread_stack            = 128K
thread_cache_size       = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover          = BACKUP
#max_connections        = 100
table_cache            = 128
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit       = 1M

query_cache_size        = 32M
tmp_table_sizee        = 64M
max_heap_table_sizee        = 32M

Then save the file and restart mysql. After a few hours, rerun mysqltuner and check again if the values are fine now or if the have to be increased to a higher value.

MySQL settings, many concurrent users.
I run a site for a client that has over 3000 users that log in for about 5-7 hours
per day each. So, at peak times, we have to handle about 2000 concurrent users.
When configured correctly, PHP and MySQL can handle this load wonderfully on fairly
cheap Intel architecture. First off, hardware.

1) It is better to have 2 separate servers for Apache/PHP and MySQL with the Linux of your choice.
2) Try not to run too much else on either box; leave the resources for Apache/PHP and MySQL.

Here are the specs on each box in my config:
1) Apache/PHP: Pentium 3, 600 MHZ, 512 megs ram.
2) MySQL: Dual Pentium 3, 750 MHZ (1500 MHZ total), 2 gigs ram.

The reason for this configuration is that it is very database heavy;
it is a members only web site with username and password required for login, fully personalized.
It is an online school, so each student has their suite of tools for attending school,
their courses, report cards, time logging, and much more. Teachers have web based tools to create
their courses, including lessons, text to speech audio, and more.

1) PHP coding: be sure to use persistent connections!
Opening and closing a connection from your Apache/PHP box to your MySQL box is a very heavy load.
By using persistent connections, a high capacity site will open connections and share them to
exchange data rather than opening a connection on each pagerequest, sending the data, then closing,
and repeating that process at least once for every user click!
Be sure to use “mysql_pconnect” instead of “mysql_connect” and also that appropriate changes are made
in “php.ini” or overridden by using the command ”ini_set”.
You can find more documentation on doing this at the php web site.

2) Apache set up (“httpd.conf”): I’ve changed these various settings, and played with them until they
seem to keep the most “idle %” reported in “top”.
MinSpareServers 10
MaxSpareServers 20
StartServers 70
MaxClients 255

3) Mysql set up (“my.cnf”). The MySQL config file, my.cnf.
Here is what to add under the [mysqld] heading.
The two lines, “max_connections” and “max_user_connections” are where the magic happens.
Since your Apache/PHP box is connecting to MySQL, it appears as a single user.
MySQL defaults to 1 max connection, with 1 max connection per user. The following lines make
it so your Apache/PHP box can connect to your MySQL box up to the number you have set “MaxClients”
to in the Apache config above. By using persistent connections, you can pretty much get Apache up,
have it connect to MySQL upon start up, and just use the persistent connections to pass data between
the two boxes rather than opening connections. Its much more efficient that way.
set-variable = max_connections = 300
(this must be higher than “MaxClients” set in Apache, or you won’t fully maximize use)
set-variable = max_user_connections = 300
set-variable = table_cache=1200
(max number of tables in join multiplied by max_user_connections)

A few other MySQL tunings:
set-variable = max_allowed_packet=1M (sanity check to stop runaway queries)
set-variable = max_connect_errors=999999
(stop mysqld from shutting down if there are connect errors – this defaults to 1 error and mysqld stops!)

What one can and should optimize

• Hardware
• OS / libraries
• SQL server (setup and queries)
• API
• Application

Optimizing hardware for MySQL

• If you need big tables ( > 2G), you should consider using 64 bit hardware like Alpha, Sparc or the upcoming IA64. As MySQL uses a lot of 64 bit integers internally, 64 bit CPUs will give much better performance.
• For large databases, the optimization order is normally RAM, Fast disks, CPU power.
• More RAM can speed up key updates by keeping most of the used key pages in RAM.
• If you are not using transaction-safe tables or have big disks and want to avoid long file checks, a UPS is good idea to be able to take the system down nicely in case of a power failure.
• For systems where the database is on a dedicated server, one should look at 1G Ethernet. Latency is as important as throughput.

Optimizing disks

• Have one dedicated disk for the system, programs and for temporary files. If you do very many changes, put the update logs and transactions logs on dedicated disks.
• Low seek time is important for the database disk; For big tables you can estimate that you will need: log(row_count) / log(index_block_length/3*2/(key_length + data_ptr_length))+1 seeks to find a row. For a table with 500,000 rows indexing a medium int: log(500,000)/log(1024/3*2/(3+4)) +1 = 4 seeks The above index would require:500,000 * 7 * 3/2 = 5.2M. In real life, most of the blocks will be buffered, so probably only 1-2 seeks are needed.
• For writes you will need (as above) 4 seek requests, however, to find where to place the new key, and normally 2 seeks to update the index and write the row.
• For REALLY big databases, your application will be bound by the speed of your disk seeks, which increase by N log N as you get more data.
• Split databases and tables over different disks. In MySQL you can use symbolic links for this.
• Striping disks (RAID 0) will increase both read and write throughput.
• Striping with mirroring (RAID 0+1) will give you safety and increase the read speed. Write speed will be slightly lower.
• Don’t use mirroring or RAID (except RAID 0) on the disk for temporary files or for data that can be easily re-generated..
• On Linux use hdparm -m16 -d1 on the disks on boot to enable reading/writing of multiple sectors at a time, and DMA. This may increase the response time by 5-50 %.
• On Linux, mount the disks with async (default) and noatime.
• For some specific application, one may want to have a ram disk for some very specific tables, but normally this is not needed.

Optimizing OS

• No swap; If you have memory problems, add more RAM instead or configure your system to use less memory.
• Don’t use NFS disks for data (you will have problems with NFS locking).
• Increase number of open files for system and for the SQL server. (add ulimit -n # in the safe_mysqld script).
• Increase the number of processes and threads for the system.
• If you have relatively few big tables, tell your file system to not break up the file on different cylinders (Solaris).
• Use file systems that support big files (Solaris).
• Choose which file system to use; Reiserfs on Linux is very fast for open, read and write. File checks take just a couple of seconds.

Choosing API

• PERL
  • Portable programs between OS and databases
  • Good for quick prototyping
  • One should use the DBI/DBD interface
• PHP
  • Simpler to learn than PERL.
  • Uses less resources than PERL, which makes it good for embedding in Web servers.
  • One can get more speed by upgrading to PHP4.
• C
  • The native interface to MySQL.
  • Faster and gives more control
  • Lower level, so you have to work more.
• C++
  • Higher level gives you more time to code your application.
  • Is still in development.
• ODBC
  • Works on Windows and Unix
  • Almost portable between different SQL servers.
  • Slow; MyODBC, which is a simple pass-through driver is 19 % slower than using a native interface.
  • Many ways to do the same thing; Hard to get things to work as many ODBC drivers have different bugs in different areas.
  • Problematic; Microsoft changes the interface once in a while.
  • Insecure future (Microsoft pushes more for OLE than for ODBC).
• JDBC
  • In theory portable between OS and databases.
  • Can be run in the web client.
• Python + others
  • May be fine, but we don’t use them.

Optimizing the application

• One should concentrate on solving the problem.
• When writing the application one should decide what is most important:
  • Speed
  • Portability between OS
  • Portability between SQL servers
• Use persistent connections.
• Cache things in your application to lessen the load of the SQL server.
• Don’t query columns that you don’t need in your application.
• Don’t use SELECT * FROM table_name...
• Benchmark all parts of your application, but put the most effort into benchmarking the whole application under the worst possible ‘reasonable’ load. By doing this in a modular fashion you should be able to replace the found bottleneck with a fast ‘dummy module’, you can then easily identify the next bottleneck (and so on).
• Use LOCK TABLES if you do a lot of changes in a batch; For example group multiple UPDATES or DELETES together.

Portable applications should use

• Perl DBI/DBD
• ODBC
• JDBC
• Python (or any other language that has a generalized SQL interface)
• You should only use SQL constructs which exist in all the target SQL servers or can easily be emulated with other constructs. The crash-me pages on dev.mysql.com can help you with this.
• Write wrappers to provide missing functionality for other OSes / SQL servers.

If you need more speed, you should:

• Find the bottleneck (CPU, disk, memory, SQL server, OS, API, or application) and concentrate on solving this.
• Use extensions that give you more speed / flexibility.
• Get to know your SQL server so that you can use the fastest possible SQL constructs for your problem and avoid bottlenecks.
• Optimize your table layouts and queries.
• Use replication to get more select speed.
• If you have a slow net connection to the database, use the compressed client/server protocol.

Don’t be afraid to make the first version of your application not perfectly portable; when you have solved your problem, you can always optimize it later.

Optimizing MySQL

• Choose compiler and compiler options.
• Find the best MySQL startup options for your system.
• Scan the the MySQL manual and read Paul DuBois’ MySQL book.
• Use EXPLAIN SELECT, SHOW VARIABLES, SHOW STATUS and SHOW PROCESSLIST.
• Learn how the query optimizer works.
• Optimize your table formats.
• Maintain your tables (myisamchk, CHECK TABLE, OPTIMIZE TABLE).
• Use MySQL extensions to get things done faster.
• Write a MySQL UDF function if you notice that you would need some function in many places.
• Don’t use GRANT on table level or column level if you don’t really need it.
• Pay for MySQL support and get help to solve your problem

Compiling and installing MySQL

• By choosing the best possible compiler for your system, you can usually get 10-30 % better performance.
• On Linux/Intel, compile MySQL with pgcc. (The Pentium optimized version of gcc). The binary will only work with Intel Pentium CPUs, however.
• Use the optimize options that are recommended in the MySQL manual for a particular platform.
• Normally a native compiler for a specific CPU (like Sun Workshop for Sparc) should give better performance than gcc, but this is not always the case.
• Compile MySQL with only the character sets you are going to use.
• Compile the mysqld executable statically (with --with-mysqld-ldflags=-all-static) and strip the final executable with strip sql/mysqld.
• Note that as MySQL doesn’t use C++ exceptions, compiling MySQL without exceptions support will give a big performance win!
• Use native threads (instead of the included mit-pthreads) if your OS supports native threads.
• Test the resulting binary with the MySQL benchmark test.

Maintenance

• If possible, run OPTIMIZE table once in a while. This is especially important on variable size rows that are updated a lot.
• Update the key distribution statistics in your tables once in a while with myisamchk -a; Remember to take down MySQL before doing this!
• If you get fragmented files, it may be worth it to copy all files to another disk, clear the old disk and copy the files back.
• If you have problems, check your tables with myisamchk or CHECK table.
• Monitor MySQL status with: mysqladmin -i10 processlist extended-status
• With the MySQL GUI client you can monitor the process list and the status in different windows.
• Use mysqladmin debug to get information about locks and performance.

Optimizing SQL

Use SQL for the things it’s good at, and do other things in your application. Use the SQL server to:

• Find rows based on WHERE clause.
• JOIN tables
• GROUP BY
• ORDER BY
• DISTINCT

Don’t use an SQL server:

• To validate data (like date)
• As a calculator

Tips

• Use keys wisely.
• Keys are good for searches, but bad for inserts / updates of key columns.
• Keep by data in the 3rd normal database form, but don’t be afraid of duplicating information or creating summary tables if you need more speed.
• Instead of doing a lot of GROUP BYs on a big table, create summary tables of the big table and query this instead.
• UPDATE table set count=count+1 where key_column=constant is very fast!
• For log tables, it’s probably better to generate summary tables from them once in a while than try to keep the summary tables live.
• Take advantage of default values on INSERT.

Speed difference between different SQL servers (times in seconds)

In the above test, MySQL was run with a 8M cache; the other databases were run with installations defaults.

Important MySQL startup options

Optimizing tables

• MySQL has a rich set of different types. You should try to use the most efficient type for each column.
• The ANALYSE procedure can help you find the optimal types for a table: SELECT * FROM table_name PROCEDURE ANALYSE()
• Use NOT NULL for columns which will not store null values. This is particularly important for columns which you index.
• Change your ISAM tables to MyISAM.
• If possible, create your tables with a fixed table format.
• Don’t create indexes you are not going to use.
• Use the fact that MySQL can search on a prefix of an index; If you have and INDEX (a,b), you don’t need an index on (a).
• Instead of creating an index on long CHAR/VARCHAR column, index just a prefix of the column to save space. CREATE TABLE table_name (hostname CHAR(255) not null, index(hostname(10)))
• Use the most efficient table type for each table.
• Columns with identical information in different tables should be declared identically and have identical names.

How MySQL stores data

• Databases are stored as directories.
• Tables are stored as files.
• Columns are stored in the files in dynamic length or fixed size format. In BDB tables the data is stored in pages.
• Memory-based tables are supported.
• Databases and tables can be symbolically linked from different disks.
• On Windows MySQL supports internal symbolic links to databases with .sym files.

MySQL table types

• HEAP tables; Fixed row size tables that are only stored in memory and indexed with a HASH index.
• ISAM tables; The old B-tree table format in MySQL 3.22.
• MyISAM tables; New version of the ISAM tables with a lot of extensions:
  • Binary portability.
  • Index on NULL columns.
  • Less fragmentation for dynamic-size rows than ISAM tables.
  • Support for big files.
  • Better index compression.
  • Better key statistics.
  • Better and faster auto_increment handling.
• Berkeley DB (BDB) tables from Sleepycat: Transaction-safe (with BEGIN WORK / COMMIT | ROLLBACK).

MySQL row types (only relevant for ISAM/MyISAM tables)

• MySQL will create the table in fixed size table format if all columns are of fixed length format (no VARCHAR, BLOB or TEXT columns). If not, the table is created in dynamic-size format.
• Fixed-size format is much faster and more secure than the dynamic format.
• The dynamic-size row format normally takes up less space but may be fragmented over time if the table is updated a lot.
• In some cases it’s worth it to move all VARCHAR, BLOB and TEXT columns to another table just to get more speed on the main table.
• With myisampack (pack_isam for ISAM) one can create a read-only, packed table. This minimizes disk usage which is very nice when using slow disks. The packed tables are perfect to use on log tables which one will not update anymore.

MySQL caches (shared between all threads, allocated once)

• Key cache ; key_buffer_size, default 8M
• Table cache ; table_cache, default 64
• Thread cache ; thread_cache_size, default 0.
• Hostname cache ; Changeable at compile time, default 128.
• Memory mapped tables ; Currently only used for compressed tables.

Note that MySQL doesn’t have a row cache, but lets the OS handle this!

MySQL buffer variables (not shared, allocated on demand)

• sort_buffer ; ORDER BY / GROUP BY
• record_buffer ; Scanning tables
• join_buffer_size ; Joining without keys
• myisam_sort_buffer_size ; REPAIR TABLE
• net_buffer_length ; For reading the SQL statement and buffering the result.
• tmp_table_size ; HEAP-table-size for temporary results.

How the MySQL table cache works

• Each open instance of a MyISAM table uses an index file and a data file. If a table is used by two threads or used twice in the same query, MyISAM will share the index file but will open another instance of the data file.
• The cache will temporarily grow larger than the table cache size if all tables in the cache are in use. If this happens, the next table that is released will be closed.
• You can check if your table cache is too small by checking the mysqld variable Opened_tables. If this value is high you should increase your table cache!

MySQL extensions / optimization that gives you speed

• Use the optimal table type (HEAP, MyISAM, or BDB tables).
• Use optimal columns for your data.
• Use fixed row size if possible.
• Use the different lock types (SELECT HIGH_PRIORITY, INSERT LOW_PRIORITY)
• Auto_increment
• REPLACE (REPLACE INTO table_name VALUES (...))
• INSERT DELAYED
• LOAD DATA INFILE / LOAD_FILE()
• Use multi-row INSERT to insert many rows at a time.
• SELECT INTO OUTFILE
• LEFT JOIN, STRAIGHT JOIN
• LEFT JOIN combined with IS NULL
• ORDER BY can use keys in some cases.
• If you only query columns that are in one index, only the index tree will be used to resolve the query.
• Joins are normally faster than subselects (this is true for most SQL servers).
• LIMIT
  • SELECT * from table1 WHERE a > 10 LIMIT 10,20
  • DELETE * from table1 WHERE a > 10 LIMIT 10
• foo IN (list of constants) is very optimized.
• GET_LOCK()/RELEASE_LOCK()
• LOCK TABLES
• INSERT and SELECT can run concurrently.
• UDF functions that can be loaded into a running server.
• Compressed read-only tables.
• CREATE TEMPORARY TABLE
• CREATE TABLE .. SELECT
• MyISAM tables with RAID option to split a file over many files to get over the 2G limit on some file system.
• Delayed_keys
• Replication

When MySQL uses indexes

• Using >, >=, =, <, <=, IF NULL and BETWEEN on a key.

  • SELECT * FROM table_name WHERE key_part1=1 and key_part2 > 5;
  • SELECT * FROM table_name WHERE key_part1 IS NULL;
• When you use a LIKE that doesn't start with a wildcard.

  • SELECT * FROM table_name WHERE key_part1 LIKE 'jani%'
• Retrieving rows from other tables when performing joins.

  • SELECT * from t1,t2 where t1.col=t2.key_part
• Find the MAX() or MIN() value for a specific index.

  • SELECT MIN(key_part2),MAX(key_part2) FROM table_name where key_part1=10
• ORDER BY or GROUP BY on a prefix of a key.

  • SELECT * FROM foo ORDER BY key_part1,key_part2,key_part3
• When all columns used in the query are part of one key.

  • SELECT key_part3 FROM table_name WHERE key_part1=1

When MySQL doesn’t use an index

• Indexes are NOT used if MySQL can calculate that it will probably be faster to scan the whole table. For example if key_part1 is evenly distributed between 1 and 100, it’s not good to use an index in the following query:
  • SELECT * FROM table_name where key_part1 > 1 and key_part1 < 90
• If you are using HEAP tables and you don’t search on all key parts with =
• When you use ORDER BY on a HEAP table
• If you are not using the first key part
  • SELECT * FROM table_name WHERE key_part2=1
• If you are using LIKE that starts with a wildcard
  • SELECT * FROM table_name WHERE key_part1 LIKE '%jani%'
• When you search on one index and do an ORDER BY on another
  • SELECT * from table_name WHERE key_part1 = # ORDER BY key2

Learn to use EXPLAIN

Use EXPLAIN on every query that you think is too slow!

mysql> explain select t3.DateOfAction, t1.TransactionID
    -> from t1 join t2 join t3
    -> where t2.ID = t1.TransactionID and t3.ID = t2.GroupID
    -> order by t3.DateOfAction, t1.TransactionID;
+-------+--------+---------------+---------+---------+------------------+------+---------------------------------+
| table | type   | possible_keys | key     | key_len | ref              | rows | Extra                           |
+-------+--------+---------------+---------+---------+------------------+------+---------------------------------+
| t1    | ALL    | NULL          | NULL    |    NULL | NULL             |   11 | Using temporary; Using filesort |
| t2    | ref    | ID            | ID      |       4 | t1.TransactionID |   13 |                                 |
| t3    | eq_ref | PRIMARY       | PRIMARY |       4 | t2.GroupID       |    1 |                                 |
+-------+--------+---------------+---------+---------+------------------+------+---------------------------------+

Types ALL and range signal a potential problem.

Learn to use SHOW PROCESSLIST

Use SHOW processlist to find out what is going on:

+----+-------+-----------+----+---------+------+--------------+-------------------------------------+
| Id | User  | Host      | db | Command | Time | State        | Info                                |
+----+-------+-----------+----+---------+------+--------------+-------------------------------------+
| 6  | monty | localhost | bp | Query   | 15   | Sending data | select * from station,station as s1 |
| 8  | monty | localhost |    | Query   | 0    |              | show processlist                    |
+----+-------+-----------+----+---------+------+--------------+-------------------------------------+

Use KILL in mysql or mysqladmin to kill off runaway threads.

How to find out how MySQL solves a query

Run the following commands and try to understand the output:

• SHOW VARIABLES;
• SHOW COLUMNS FROM ...\G
• EXPLAIN SELECT ...\G
• FLUSH STATUS;
• SELECT ...;
• SHOW STATUS;

MySQL is extremely good

• For logging.
• When you do many connects; connect is very fast.
• Where you use SELECT and INSERT at the same time.
• When you don’t combine updates with selects that take a long time.
• When most selects/updates are using unique keys.
• When you use many tables without long conflicting locks.
• When you have big tables (MySQL uses a very compact table format).

Things to avoid with MySQL

• Updates to a table or INSERT on a table with deleted rows, combined with SELECTS that take a long time.
• HAVING on things you can have in a WHERE clause.
• JOINS without using keys or keys which are not unique enough.
• JOINS on columns that have different column types.
• Using HEAP tables when not using a full key match with =
• Forgetting a WHERE clause with UPDATE or DELETE in the MySQL monitor. If you tend to do this, use the --i-am-a-dummy option to the mysq client.

Different locks in MySQL

• Internal table locks.
• LOCK TABLES (Works on all table types)
• GET_LOCK()/RELEASE_LOCK()
• Page locks (for BDB tables)
• ALTER TABLE also does a table lock on BDB tables.
• LOCK TABLES gives you multiple readers on a table or one writer.
• Normally a WRITE lock has higher priority than a READ lock to avoid starving the writers. For writers that are not important one can use theLOW_PRIORITY keyword to let the lock handler prefer readers.

    UPDATE LOW_PRIORITY SET value=10 WHERE id=10;

Tricks to give MySQL more information to solve things better

Note that you can always comment out a MySQL feature to make the query portable:

SELECT /*! SQL_BUFFER_RESULTS */ ...

• SELECT SQL_BUFFER_RESULTS ...
  Will force MySQL to make a temporary result set. As soon as the temporary set is done, all locks on the tables are released. This can help when you get a problem with table locks or when it takes a long time to transfer the result to the client.
• SELECT SQL_SMALL_RESULT ... GROUP BY ...
  To tell the optimizer that the result set will only contain a few rows.
• SELECT SQL_BIG_RESULT ... GROUP BY ...
  To tell the optimizer that the result set will contain many rows.
• SELECT STRAIGHT_JOIN ...
  Forces the optimizer to join the tables in the order in which they are listed in the FROM clause.
• SELECT ... FROM table_name [USE INDEX (index_list) | IGNORE INDEX (index_list)] table_name2
  Forces MySQL to use/ignore the listed indexes.

Example of doing transactions

• How to do a transaction with MyISAM tables:

  mysql> LOCK TABLES trans READ, customer WRITE;
  mysql> select sum(value) from trans where customer_id=some_id;
  mysql> update customer set total_value=sum_from_previous_statement
             where customer_id=some_id;
  mysql> UNLOCK TABLES;

• How to do a transaction with Berkeley DB tables:

  mysql> BEGIN WORK; 
  mysql> select sum(value) from trans where customer_id=some_id;
  mysql> update customer set total_value=sum_from_previous_statement
             where customer_id=some_id;
  mysql> COMMIT;

• Note that you can often avoid transactions altogether by doing:

  UPDATE customer SET value=value+new_value WHERE customer_id=some_id;

Example of using REPLACE

REPLACE works exactly like INSERT, except that if an old record in the table has the same value as a new record on a unique index, the old record is deleted before the new record is inserted. Instead of using

  SELECT 1 FROM t1 WHERE key=#
  IF found-row
    LOCK TABLES t1
    DELETE FROM t1 WHERE key1=#
    INSERT INTO t1 VALUES (...)
    UNLOCK TABLES t1;
  ENDIF

Do

  REPLACE INTO t1 VALUES (...)

General tips

• Use short primary keys. Use numbers, not strings, when joining tables.
• When using multi-part keys, the first part should be the most-used key.
• When in doubt, use columns with more duplicates first to get better key compression.
• If you run the client and MySQL server on the same machine, use sockets instead of TCP/IP when connecting to MySQL (this can give you up to a 7.5 % improvement). You can do this by specifying no hostname or localhost when connecting to the MySQL server.
• Use --skip-locking (default on some OSes) if possible. This will turn off external locking and will give better performance.
• Use application-level hashed values instead of using long keys:

    SELECT * FROM table_name WHERE hash=MD5(concat(col1,col2)) AND
    col_1='constant' AND col_2='constant'

• Store BLOB’s that you need to access as files in files. Store only the file name in the database.
• It is faster to remove all rows than to remove a large part of the rows.
• If SQL is not fast enough, take a look at the lower level interfaces to access the data.

Benefits of using MySQL 3.23

• MyISAM ; Portable BIG table format
• HEAP ; In memory tables
• Berkeley DB ; Transactional tables from Sleepycat.
• A lot of raised limits
• Dynamic character sets
• More STATUS variables.
• CHECK and REPAIR table.
• Faster GROUP BY and DISTINCT
• LEFT JOIN ... IF NULL optimization.
• CREATE TABLE ... SELECT
• CREATE TEMPORARY table_name (...)
• Automatic conversion of temporary HEAP to MyISAM tables
• Replication
• mysqlhotcopy script.

Important features that we are actively working on

• Improving transactions
• Fail safe replication
• Text searching
• Delete with many tables (Updates with many tables will be done after this.)
• Better key cache
• Atomic RENAME (RENAME TABLE foo as foo_old, foo_new as foo)
• A query cache
• MERGE TABLES
• A better GUI client

24 iostat, vmstat and mpstat Examples for Linux Performance Monitoring

$ iostat
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.72      1096.66      1598.70 2719068704 3963827344
sda1            178.20       773.45      1329.09 1917686794 3295354888
sda2             16.51       323.19       269.61  801326686  668472456
sdb             371.31       945.97      1073.33 2345452365 2661206408
sdb1            371.31       945.95      1073.33 2345396901 2661206408
sdc             408.03       207.05       972.42  513364213 2411023092
sdc1            408.03       207.03       972.42  513308749 2411023092

$ iostat -c
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

$ iostat -d
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.71      1096.61      1598.63 2719068720 3963827704
sda1            178.20       773.41      1329.03 1917686810 3295355248
sda2             16.51       323.18       269.60  801326686  668472456
sdb             371.29       945.93      1073.28 2345452365 2661209192
sdb1            371.29       945.91      1073.28 2345396901 2661209192
sdc             408.01       207.04       972.38  513364213 2411024484
sdc1            408.01       207.02       972.38  513308749 2411024484

$ iostat -n
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)        07/09/2011

avg-cpu:  %user   %nice    %sys %iowait   %idle
           4.33    0.01    1.16    0.31   94.19

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.83         0.35         5.39   29817402  457360056
sda1              3.32        50.18         4.57 4259963994  387641400
sda2              0.20         0.76         0.82   64685128   69718576
sdb               6.59        15.53        42.98 1318931178 3649084113
sdb1             11.80        15.53        42.98 1318713382 3649012985

Device:                  rBlk_nor/s   wBlk_nor/s   rBlk_dir/s   wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
192.168.1.4:/home/data      90.67        0.00         0.00         0.00         5.33         0.00
192.168.1.4:/backup         8.74         0.00         0.00         0.00         8.74         0.00
192.168.1.8:/media          0.02         0.00         0.00         0.00         0.01         0.00

$ iostat -m
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda             194.70         0.54         0.78    1327670    1935463
sda1            178.19         0.38         0.65     936370    1609060
sda2             16.51         0.16         0.13     391272     326402
sdb             371.27         0.46         0.52    1145240    1299425
sdb1            371.27         0.46         0.52    1145213    1299425
sdc             407.99         0.10         0.47     250666    1177259
sdc1            407.99         0.10         0.47     250639    1177259

$ iostat -p sda
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.69      1096.51      1598.48 2719069928 3963829584
sda2            336.38        27.17        54.00   67365064  133905080
sda1            821.89         0.69       243.53    1720833  603892838

$ iostat -t
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

Time: 08:57:52 AM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.69      1096.49      1598.45 2719070384 3963829704
sda1            178.18       773.32      1328.88 1917688474 3295357248
sda2             16.51       323.14       269.57  801326686  668472456
sdb             371.25       945.82      1073.16 2345452741 2661228872
sdb1            371.25       945.80      1073.16 2345397277 2661228872
sdc             407.97       207.02       972.27  513364233 2411030200
sdc1            407.97       207.00       972.27  513308769 2411030200

$ iostat -x
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda              27.86    63.53 61.77 132.91  1096.46  1598.40    13.84     0.21    1.06   2.28  44.45
sda1              0.69    33.22 48.54 129.63   773.30  1328.84    11.80     1.39    7.82   2.28  40.57
sda2             27.16    30.32 13.23  3.28   323.13   269.56    35.90     0.55   32.96   3.44   5.68
sdb              39.15   215.16 202.20 169.04   945.80  1073.13     5.44     1.05    2.78   1.64  60.91
sdb1             39.15   215.16 202.20 169.04   945.77  1073.13     5.44     1.05    2.78   1.64  60.91
sdc               8.90     3.63 356.56 51.40   207.01   972.24     2.89     1.04    2.56   1.55  63.30
sdc1              8.90     3.63 356.55 51.40   206.99   972.24     2.89     1.04    2.56   1.55  63.30

$ iostat -x sda1
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda1              0.69    33.21 48.54 129.62   773.23  1328.76    11.80     1.39    7.82   2.28  40.56

$ iostat 2
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.67      1096.39      1598.33 2719070584 3963891256
sda1            178.16       773.26      1328.79 1917688482 3295418672
sda2             16.51       323.11       269.54  801326878  668472584
sdb             371.22       945.74      1073.08 2345454041 2661251200
sdb1            371.22       945.72      1073.08 2345398577 2661251200
sdc             407.93       207.00       972.19  513366813 2411036564
sdc1            407.93       206.98       972.19  513311349 2411036564
..

$ iostat 2 3

$ iostat -N

$ iostat -V
sysstat version 7.0.2
(C) Sebastien Godard

$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 305416 260688  29160 2356920    2    2     4     1    0    0  6  1 92  2  0

$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
 0  0 305416 253820 1052680 2688928    2    2     4     1    0    0  6  1 92  2  0

$ vmstat -f
     81651975 forks

$ vmstat 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 537144 182736 6789320    0    0     0     0    1    1  0  0 100  0  0
 0  0      0 537004 182736 6789320    0    0     0     0   50   32  0  0 100  0  0
..

$ vmstat 2 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 537144 182736 6789320    0    0     0     0    1    1  0  0 100  0  0
 0  0      0 537004 182736 6789320    0    0     0     0   50   32  0  0 100  0  0
..

$ vmstat -t 1 100
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp---
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 3608728 148368 3898200    0    0     0     0    1    1  0  0 100  0  0     2011-07-09 21:16:28 PDT
 0  0      0 3608728 148368 3898200    0    0     0     0   60   15  0  0 100  0  0     2011-07-09 21:16:29 PDT
 0  0      0 3608712 148368 3898200    0    0     0     0   32   28  0  0 100  0  0     2011-07-09 21:16:30 PDT

$ vmstat -V
procps version 3.2.8

$ vmstat -m
Cache                       Num  Total   Size  Pages
fib6_nodes                    5    113     32    113
ip6_dst_cache                 4     15    256     15
ndisc_cache                   1     15    256     15
RAWv6                         7     10    768      5
UDPv6                         0      0    640      6
tw_sock_TCPv6                 0      0    128     30
...

$ vmstat -s
      4149928  total memory
      3864824  used memory
      2606664  active memory
      1098180  inactive memory
       285104  free memory
        19264  buffer memory
      2326692  swap cache
      4192956  total swap
       274872  used swap
      3918084  free swap
   1032454000 non-nice user cpu ticks
        14568 nice user cpu ticks
     89482270 system cpu ticks
  16674327143 idle cpu ticks
    368965706 IO-wait cpu ticks
      1180468 IRQ cpu ticks
..

$ vmstat -d
disk- ------------reads------------ ------------writes----------- -----IO------
       total merged sectors      ms  total merged sectors      ms    cur    sec
sda   153189971 69093708 2719150864 737822879 329617713 157559204 3965687592 4068577985      0 1102243
sdb   501426305 97099356 2345472425 731613156 419220973 533565961 2661869460 1825174087      0 1510434
sdc   884213459 22078974 513390701 452540172 127474901 8993357 2411187300 2133226954      0 1569758

$ vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 3608688 148368 3898204    0    0     0     0    1    1  0  0 100  0  0
 0  0      0 3608804 148368 3898204    0    0     0     0   72   30  0  0 100  0  0
 0  0      0 3608804 148368 3898204    0    0     0     0   60   27  0  0 100  0  0

$ vmstat -w 1 3
procs -------------------memory------------------ ---swap-- -----io---- --system-- -----cpu-------
 r  b       swpd       free       buff      cache   si   so    bi    bo   in   cs  us sy  id wa st
 0  0          0    3608712     148368    3898204    0    0     0     0    1    1   0  0 100  0  0
 0  0          0    3608712     148368    3898204    0    0     0     0   93   23   0  0 100  0  0
 0  0          0    3608696     148368    3898204    0    0     0     0   35   34   0  0 100  0  0

$ vmstat -p sdb1
sdb1          reads   read sectors  writes    requested writes
           501423248 2345417917  419221612 2661885948

$ vmstat -S m
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0    281    288     19   2386    0    0     4     1    0    0  6  1 92  2  0

$ mpstat
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

10:25:32 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
10:25:32 PM  all    5.68    0.00    0.49    2.03    0.01    0.02    0.00   91.77    146.55

$ mpstat -A
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (4 CPU)

10:26:34 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:26:34 PM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.99
10:26:34 PM    0    0.01    0.00    0.01    0.01    0.00    0.00    0.00    0.00   99.98
10:26:34 PM    1    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00   99.98
10:26:34 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:26:34 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

10:26:34 PM  CPU    intr/s
10:26:34 PM  all     36.51
10:26:34 PM    0      0.00
10:26:34 PM    1      0.00
10:26:34 PM    2      0.04
10:26:34 PM    3      0.00

10:26:34 PM  CPU     0/s     1/s     8/s     9/s    12/s    14/s    15/s    16/s    19/s    20/s    21/s    33/s   NMI/s   LOC/s   SPU/s   PMI/s   PND/s   RES/s   CAL/s   TLB/s   TRM/s   THR/s   MCE/s   MCP/s   ERR/s   MIS/s
10:26:34 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    7.47    0.00    0.00    0.00    0.00    0.02    0.00    0.00    0.00    0.00    0.00    0.00    0.00
10:26:34 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    4.90    0.00    0.00    0.00    0.00    0.03    0.00    0.00    0.00    0.00    0.00    0.00    0.00
10:26:34 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.04    0.00    0.00    0.00    0.00    0.00    3.32    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
10:26:34 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    4.17    0.00    0.00    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00    0.00    0.00

$ mpstat -P ALL
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (4 CPU)

10:28:04 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:28:04 PM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.99
10:28:04 PM    0    0.01    0.00    0.01    0.01    0.00    0.00    0.00    0.00   99.98
10:28:04 PM    1    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00   99.98
10:28:04 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:28:04 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

$ mpstat -P 0
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (8 CPU)

10:28:53 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:28:53 PM    0    0.01    0.00    0.01    0.01    0.00    0.00    0.00    0.00   99.98

$ mpstat -P 1
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (8 CPU)

10:28:55 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:28:55 PM    1    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00   99.98

$ mpstat -V
sysstat version 9.0.4
(C) Sebastien Godard (sysstat  orange.fr)

Note that this benchmark is single-threaded, so it measures the minimum time for the operations performed. We plan to add multi-threaded tests to the benchmark suite in the future.

To use the benchmark suite, the following requirements must be satisfied:

After you obtain a MySQL source distribution, you can find the benchmark suite located in its sql-benchdirectory. To run the benchmark tests, build MySQL, and then change location into the sql-bench directory and execute the run-all-tests script:

shell> cd sql-bench
shell> perl run-all-tests --server=server_name

server_name should be the name of one of the supported servers. To get a list of all options and supported servers, invoke this command:

shell> perl run-all-tests --help

The crash-me script also is located in the sql-bench directory. crash-me tries to determine what features a database system supports and what its capabilities and limitations are by actually running queries. For example, it determines:

In this tutorial we will see some tools to execute one command on multiple remote servers using ssh.First you need to make sure you have ssh installed in your machine or you can install using the following command

sudo aptitude install ssh

Now we are going to see the following tools which does the required job we are looking for

1) Pssh

2) cluster ssh

3) Multixterm

1) Pssh

pssh provides a number of commands for executing against a group of computers, using SSH. It’s most useful for operating on clusters of homogenously-configured hosts.

The package contains:

- Parallel ssh (parallel-ssh, upstream calls it pssh), executes commands on multiple hosts in parallel

- Parallel scp (parallel-scp, upstream calls it pscp), copies files to multiple remote hosts in parallel

- Parallel rsync (parallel-rsync, upstream calls it prsync), efficiently copies files to multiple hosts in parallel

- Parallel nuke (parallel-nuke, upstream calls it pnuke), kills processes on multiple remote hosts in parallel

- Parallel slurp (parallel-slurp, upstream calls it pslurp), copies files from multiple remote hosts to a central host in parallel

These tools are good for controlling large collections of nodes, where faster alternatives

Install pssh in Ubuntu

sudo aptitude install pssh

pssh Syntax

pssh command -h hosts-file options

-h –hosts   hosts file (each line “host[:port] [user]“)
-l –user    username (OPTIONAL)
-p –par     max number of parallel threads (OPTIONAL)
-o –outdir  output directory for stdout files (OPTIONAL)
-t –timeout timeout in seconds to do ssh to a host (OPTIONAL)
-v –verbose turn on warning and diagnostic messages (OPTIONAL)
-O –options SSH options (OPTIONAL)

where the hosts-file contains a list of all the hosts that you want to have the command executed on.

pssh Examples

The following example runs hostname on three machines (IPs or hostnames) specified in the file ips.txt using login irb2 and saves the output in /tmp/foo.

sudo cat ips.txt
128.112.152.122
18.31.0.190
128.232.103.201

sudo pssh -h ips.txt -l irb2 -o /tmp/foo hostname

Success on 128.112.152.122:22
Success on 18.31.0.190:22
Success on 128.232.103.201:22

sudo ls /tmp/foo

128.112.152.122  128.232.103.201  18.31.0.190

sudo cat /tmp/foo/*

planetlab-1.cs.princeton.edu
planetlab1.xeno.cl.cam.ac.uk
planetlab1.lcs.mit.edu

By default, pssh uses at most 32 ssh processes in parallel to ssh to the various nodes. (This is somewhat important if you’re controlling hundreds or thousands of machines.) By default, it also uses a timeout of one minute to ssh to a node and obtain a result. For ssh commands that take longer than this (e.g., sleep 61), the -t option can be used. Note that pssh and pnuke have a default timeout of one minute. pscp and prsync have no default timeout, but one can be specified using the -t option.

pscp

Here’s an example of using pscp to copy files in parallel to a set of machines.

sudo  pscp -h ips.txt -l irb2 /etc/hosts /tmp/hosts

Success on 128.112.152.122:22
Success on 18.31.0.190:22
Success on 128.232.103.201:22

Using the -r option will perform a recursive copy for copying entire directories.

2) cluster ssh

administer multiple ssh or rsh shells simultaneouslyClusterSSH allows you to control multiple ssh or rsh sessions from a single input window. You can also configure clusters of machines for easy invocation and interact with individual terminal windows during a session.

Install Cluster ssh in Ubuntu

sudo aptitude install clusterssh

Configuration Files

This /etc/clusters file is cluster tag database. Contains a list of tags and hostnames, in the form

<tag> [<username>@]hostname [...]

Newlines and comments (delimited with a #) are ignored.

Cluster definitions can also be added to the $HOME/.csshrc file

This /etc/csshrc file is Global configuration file, sourced by all users. Can be generated by:

sudo cssh -u > /etc/csshrc

$HOME/.csshrc

Per user configuration file. Can be generated by:

cssh -u > /etc/csshrc </tt>

To add a cluster definition to this file, use the following format

cluster_tag1 = server1 server2 user@server3
cluster_tag2 = server4 server5
clusters = cluster_tag1 cluster_tag2

Be aware of using “reserved names” and no check is currently performed for them.

Using Cluster SSH

If you want to perform the same command on the three servers one, two, & three use the following command

sudo cssh one two three

This will open three consoles, one for each server, over an ssh connection, and one little console to type your command.

3) Multixterm

Expect is a program that talks to other interactive programs according to a script. Expect is useful for running any program which requires interaction between the program and the user.In Ubuntu multixterm included in expect package.

Install expect in Ubuntu

sudo aptitude install expect

Multixterm Syntax

multixterm -[ args ] “command arg1 arg2″ server1 server2…

Arguments

-xa – The optional -xa argument indicates arguments to pass to xterm.

-xc – The optional -xc argument indicates a command to be run in each named xterm (see -xn). With no -xc argument, the command is the current shell.

-xd – The optional -xd argument indicates a directory to search for files that will appear in the Files menu. By default, the directory is: ~/lib/multixterm

-xf – The optional -xf argument indicates a file to be read at startup. See FILES below for more info.

-xn – The optional -xn argument indicates a name for each xterm. This name will also be substituted for any %n in the command argument (see -xc).

-xv – The optional -xv flag puts multixterm into a verbose mode where it will describe some of the things it is doing internally. The verbose output is not intended to be understandable to anyone but the author.

Multixterm Examples

The following command line starts up two xterms using ssh to the hosts 192.168.2.100 and 192.168.2.101

sudo multixterm -xc “ssh root@%n” 192.168.2.100 192.168.2.101

-xc – The optional -xc argument indicates a command to be run in each named xterm (see -xn). With no -xc argument, the command is the current shell.

ssh root@%n : ssh is a command to connect remote server with root user. This name will also be substituted for any %n in the command argument.

ClusterSSH is a Tk/Perl wrapper around standard Linux tools like XTerm and SSH. As such, it’ll run on just about any POSIX-compliant OS where the libraries exist — I’ve run it on Linux, Solaris, and Mac OS X. It requires the Perl libraries Tk (perl-tk on Debian or Ubuntu) and X11::Protocol (libx11-protocol-perl on Debian or Ubuntu), in addition to xterm and OpenSSH.

Installation

Installing ClusterSSH on a Debian or Ubuntu system is trivial — a simple sudo apt-get install clusterssh will install it and its dependencies. It is also packaged for use with Fedora, and it is installable via the ports system on FreeBSD. There’s also a MacPorts version for use with Mac OS X, if you use an Apple machine. Of course, it can also be compiled from source.

Configuration

ClusterSSH can be configured either via its global configuration file — /etc/clusters, or via a file in the user’s home directory called .csshrc. I tend to favor the user-level configuration as that lets multiple people on the same system to setup their ClusterSSH client as they choose. Configuration is straightforward in either case, as the file format is the same. ClusterSSH defines a “cluster” as a group of machines that you’d like to control via one interface. With that in mind, you enumerate your clusters at the top of the file in a “clusters” block, and then you describe each cluster in a separate section below.

For example, let’s say I’ve got two clusters, each consisting of two machines. “Cluster1″ has the machines “Test1″ and “Test2″ in it, and “Cluster2″ has the machines “Test3″ and “Test4″ in it. The ~.csshrc (or/etc/clusters) control file would look like this:

clusters = cluster1 cluster2

cluster1 = test1 test2
cluster2 = test3 test4

You can also make meta-clusters — clusters that refer to clusters. If you wanted to make a cluster called “all” that encompassed all the machines, you could define it two ways. First, you could simply create a cluster that held all the machines, like the following:

clusters = cluster1 cluster2 all

cluster1 = test1 test2
cluster2 = test3 test4
all = test1 test2 test3 test4

However, my preferred method is to use a meta-cluster that encompasses the other clusters:

clusters = cluster1 cluster2 all

cluster1 = test1 test2
cluster2 = test3 test4
all = cluster1 cluster2

Figure 1: Lauching ClusterSSH

By calling out the “all” cluster as containing cluster1 and cluster2, if either of those clusters ever change, the change is automatically captured so you don’t have to update the “all” definition. This will save you time and headache if your .csshrc file ever grows in size.

Using ClusterSSH

Using ClusterSSH is similar to launching SSH by itself. Simply running cssh -l <username> <clustername>will launch ClusterSSH and log you in as the desired user on that cluster. In the figure below, you can see I’ve logged into “cluster1″ as myself. The small window labeled “CSSH [2]” is the Cluster SSH console window. Anything I type into that small window gets echoed to all the machines in the cluster — in this case, machines “test1″ and “test2″. In a pinch, you can also login to machines that aren’t in your .csshrc file, simply by running cssh -l <username> <machinename1> <machinename2> <machinename3>.

If I want to send something to one of the terminals, I can simply switch focus by clicking in the desired XTerm, and just type in that window like I usually would. ClusterSSH has a few menu items that really help when dealing with a mix of machines. As per the figure below, in the “Hosts” menu of the ClusterSSH console there’s are several options that come in handy.

“Retile Windows” does just that if you’ve manually resized or moved something. “Add host(s) or Cluster(s)” is great if you want to add another set of machines or another cluster to the running ClusterSSH session. Finally, you’ll see each host listed at the bottom of the “Hosts” menu. By checking or unchecking the boxes next to each hostname, you can select which hosts the ClusterSSH console will echo commands to. This is handy if you want to exclude a host or two for a one-off or particular reason. The final menu option that’s nice to have is under the “Send” menu, called “Hostname”. This simply echoes each machine’s hostname to the command line, which can be handy if you’re constructing something host-specific across your cluster.

Figure 2: Resize Windows

Caveats with ClusterSSH

Like many UNIX tools, ClusterSSH has the potential to go horribly awry if you aren’t very careful with its use. I’ve seen ClusterSSH mistakes take out an entire tier of Web servers simply by propagating a typo in an Apache configuration. Having access to multiple machines at once, possibly as a privileged user, means mistakes come at a great cost. Take care, and double-check what you’re doing before you punch that Enter key.

Conclusion

ClusterSSH isn’t a replacement for having a configuration management system or any of the other best practices when managing a number of machines. However, if you need to do something in a pinch outside of your usual toolset or process, or if you’re doing prototype work, ClusterSSH is indispensable. It can save a lot of time when doing tasks that need to be done on more than one machine, but like any power tool, it can cause a lot of damage if used haphazardly.