Bonnie++ is a short test, running about 30 minutes on a single client. The information it yields can be used to compare differences between different mount options, but its real value is in deriving CPU usage during the various tests it performs. A test that returns a high CPU utilization indicates that your bottleneck is the CPU itself and that a faster processor will improve performance. A low CPU utilization indicates that the problem is elsewhere.
There are two parts to the bonnie++ 'benchmark.' The first part works with large files and the second part tests with small files. As the bonnie++man page states, the first part simulates what would happen to a filesystem when being used as a database server, while the second part is good for things like a mail cache or for a cluster node hosting a large number of small output files that need processing.
Firstly, you can install Bonnie++ from the EPEL repository. A standard run on a 2 GB machine is something like this, where we wish to write files to
/home/bryan, as the user
[[email protected]] bonnie++ -u bryan -r 2048 -d /home/bryan/bonnie/ -m server-baseline
Each time you run bonnie++, change the -m option to reflect what options you've enabled or changed between each test. For example, a subsequent run that enables noatime could use "-m server-noatime" and another test with 32 K block sizes and noatime might be "-m server-noatime-32k". The naming convention is up to you.
The primary output of Bonnie++ is a human-readable, plain-text in 80 columns, which is designed to fit well when pasted into email and which will work well in most terminals. The second type of output is CSV (Comma Seperated Values). This can easily be imported into any spreadsheet, database or graphing tool.
Once all of your bonnie++ tests are complete, you can start analyzing the results. What you're looking for is the highest numbers regardless of CPU utilization. In the case of two numbers being the same, take the lower CPU utilization as the better of the two. Now compare this to what you'll be using the server for. If you have larger files, put more importance on the results of the first half of the test. For lots of smaller files, pay attention to the second half of the tests.