There can be a number of reasons for this. Below is a list of the more common issues:
1. DNS based rules are activated, but there is no local DNS server on the system
Check to see if you have any DNS based rules loaded. Please see the mod_security page for a list of rules and which ones are DNS based.
These rules use DNS lookups to check for various characteristics of the connecting system, for example is it on a blacklist, is the IP address known to be used by a trusted source (Google for example). This means that for every connection to your system, a DNS lookup is done against the IP to see if the source IP is listed in some zone or blacklist. If you do not have a local DNS server, this can take a very long time depending on how busy the DNS server is that you are communicating with, and because the request is not cached, your system is asking the DNS server the same question several times in a row. This is because of the way HTTP works. A typical user request may need to open 100 or more connections to your site, and each time a DNS request is made to the RBL.
The simple fast solution is to make sure you are running a local DNS server on your system and that it caches requests. That way your RBL lookups are only to your local system, and if it already has the answer it will not have to ask the RBLs DNS server across the Internet. The difference in performance is night and day with a local DNS server, so we highly recommend you use on if you are going to use any RBL with any product.
2. The system is low on resources
As with any piece of software, the WAF will consume some resources on the system. If the system is short on resources the WAF has to compete with other parts of the system for those resources. Check to make sure you do not have any process that is using up RAM or cycles on the system.
3. Custom rules may be inefficient
If you are using custom rules, try disabling them. If a rule is written in an inefficient manner it can kill performance on the system.
4. Rules are loaded twice
If you have setup modsecurity yourself, check to make sure you arent loading the rules more than once. One several cpanel installations we've seen cases where users inadvertently setup modsecurity to load the same rules twice, and in one case a user set them up to load three times. Apache is a literal animal, if you load the same rule twice it will process that same rule twice, doubling the work on the system! And because our rules use advanced branching logic, if you load them multiple times that can have a very adverse effect on performance as the branching tree logic won't work correctly and you lose our on all the performance benefits a modern ruleset provides.
5. Suboptimal custom configuration
If you setup modsecurity yourself, your configuration could be suboptimal. You may have serial logging enabled, debugging enabled, certain inspection options selected, or any number of things could be wrong. We recommend you use our ASL or aum products to setup modsecurity, its automatic and it will always configure the optimal settings for your system.
If you prefer to do it yourself, then please make sure you have read the modsecurity documentation and understand the options for modsecurity before you enable them. modsecurity is a powerful piece of software that can easily be misconfigured. Care should be taken when setting up this software yourself.
For do it yourselfers, we have put together a guide to help you on the Atomic ModSecurity Rules page.