The major OS vendors, Redhat, Suse, Ubuntu and others do an excellent job of building Apache correctly. However, all software has bugs, and you should make sure that you are both running the latest updates from your vendor for a currently supported version of operating system from that vendor, and that you have ruled out an actual bug in apache. modsecurity is a complex piece of software, that relies of an even more complex piece of software, Apache, to work correctly. To improve performance, modsecurity will use apaches built in memory caching capabilities, and if there is a bug in these Apache supplied capabilities memory leaks can occur. We always recommend you discuss Apache memory leaks with your OS vendor first.
Please be especially diligent with source builds and Apache builds form non OS vendors. These builds are generally not as well tested, and so far are the only ones we have seen with memory management problems. If you use an Apache build from a non-OS vendor, you should discuss any memory management issues with them first if you have rules out the previous causes, as they are most likely a byproduct of a build error, or a bug in Apache.