Examples:
Jun 8 09:14:16 system kernel: php[5963]: segfault at a801cecc ip a800809b sp bff43d70 error 7 in ld-2.5.so[a8001000+1b000]
23:00:43 system 12 30104 [notice] child pid 26148 exit signal Segmentation fault (11)
exit signal bus error
ASL gives you more visibility into your systems errors, so in the case of segfaults ASL is not actually causing them its just reporting them (which the system didnt actually do very well before ASL was installed).
Segfaults can be indications of a minor or serious problem with a web application, for example, if the segfault was with apache or they can mean that an application died because of a buggy module, or a bug in apache itself. In other cases, a segfault may be caused by a systems configuration. And in still others, it could be an actual hardware problem.
Its not possible to explain here what may be causing your application to segfault. segfaults are a very generic error, however to help you we have provided this courtesy section to explain these non-ASL errors and to provide some cases where we have a good idea of the cause and a solution. In those cases where the issue is generic we provide a guide for tracking the source of a segfault.
If you are using the secure ASL kernel:
Step 1)
Determine what application is causing the segfault. If you are using our secure kernel, it will tell you the exact application that did this, for example if PHP segfaulted you will an error like this:
Jun 8 09:14:16 system kernel: php[5963]: segfault at a801cecc ip a800809b sp bff43d70 error 7 in ld-2.5.so[a8001000+1b000]
The kernel is indicating that "php" process 5963 segfaulted, and the library and position in memory where this occurred.
Step 2) Look at the subsections below for solutions to commonly known segfaults.
If you are not using our kernel
Your problem is a little more complex, as your system will not log what caused the segfault. Unfortunately, you may have to do some detective work to find out what application is causing the segfault. Please review the final subsection, "apache segfaults" which links to an article that provides generic guidance for tracking down the source of segfaults through core dump analysis.
php segfaults
If you are not using the ASL kernel, then ASL has nothing to do with the segfault in PHP.
Example error messages:
Mar 6 19:25:39 host kernel: grsec: From 1.2.3.4: denied marking stack executable as requested by PT_GNU_STACK marking in /usr/local/cpanel/3rdparty/php/54/zendopt/ZendGuardLoader.so by /usr/local/cpanel/3rdparty/php/54/bin/php-cgi[php-cgi:13019] uid/euid:614/614 gid/egid:615/615, parent /usr/local/cpanel/cpsrvd-ssl[cpsrvd-ssl:13018] uid/euid:614/614 gid/egid:615/615
Jun 8 09:14:16 system kernel: php[5963]: segfault at a801cecc ip a800809b sp bff43d70 error 7 in ld-2.5.so[a8001000+1b000]
Jun 8 09:14:11 system kernel: grsec: From 207.46.13.89: Segmentation fault occurred at a622eecc in /usr/bin/php[php:5952] uid/euid:590/590 gid/egid:590/590, parent /usr/local/apache/bin/httpd[httpd:5684] uid/euid:99/99 gid/egid:99/99 $ /usr/bin/php -v Segmentation fault
Discussion:
Some third party PHP modules, such as ioncube and Zend Optimizer are sometimes built in a very insecure manner, that introduces a vulnerability into the system that can lead to full compromise of the server. When they are loaded, they will attempt to open a hole in the system that could lead to a full or partial server compromise. The ASL kernel can detect when these applications attempt to open this hole, and will prevent them from doing so. This will cause php to segfault, because these insecurely configured libraries are correctly prevented from opening a hole in your system.
The solution (described below) is to just set the libraries to not attempt to open this hole, and to report this as a bug to your PHP vendor. These libraries do not need to be configured in this insecure manner to work correctly, it is both unnecessary and introduces a very large hole in your system that attackers can use to partially or fully compromise the entire server.
ASL will try to detect these libraries when they are installed and fix them on installation. With package managed systems this happens automatically.
However, if you have a customized php install or have installed php from source (as happens with CPanel), then your operating system does not manage this software dynamically and ASL can not automatically fix these broken and insecure libraries. You will need to find the insecure third party library that PHP is loading. In most cases this will be the ioncube and zend optimizer libraries, and in a package managed system (Such as Centos and Redhat) they are normally stored in locations such as (this is not a complete list):
/usr/lib64/php/ioncube/ /usr/lib/php/ioncube/
On non-package managed systems, such as cpanel systems, where the the package managed versions of PHP have been replaced with locally compiled source builds that location may and often is different. For example, cpanel usually locally compiles PHP from source and places it in locations similar to the list below (these may change, please contact cpanel or the party that build PHP from source for more information):
/usr/local/IonCube/ /usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ /usr/local/Zend/ /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ (This is not a complete list, you will need to check what external libraries your source built PHP has installed).
One trick to find these libraries is to use your systems "locate" command. For example, you can use this command to find the ioncube libraries:
locate -i ioncube | egrep ".so$" | grep lib
And this one to find the zend libraries:
locate -i zend | egrep ".so$" | grep lib
Solutions
Step 1) Determine if PHP is vulnerable
There are two conditions in which ASL can be associated with a segfault of PHP, and both of these involve conditions when PHP is doing something very dangerous and unnecessary, and ASL is preventing PHP from compromising your system. In short, if you are using the secure ASL kernel, and you are using an improperly configured version of PHP that is dangerously misconfigured and has a serious vulnerability in it, ASL will stop PHP from opening this hole in your system and PHP may generate a segfault. ASL will also log the actual vulnerabilities, which is described in more detail below.
If both of these are true (you are using the ASL kernel and you have a dangerously misconfigured PHP) then the ASL kernel will detect this dangerous vulnerability in PHP when PHP attempts to open this hole, and the ASL kernel will prevent PHP from compromising your system. Run this command to see if PHP is trying to compromise your system with this command, run as root:
egrep -i "denied rwx|mmap|mprotect" /var/log/messages
If you see any error messages that say either:
mprotect(): 13 (Permission denied)
denied RWX mmap of
Then your system has this dangerous vulnerability, and ASL is protecting your from it. We highly recommend you fix this vulnerability.
Please see the articles below for solutions to these vulnerabilities, and follow them for next steps:
https://www.atomicorp.com/wiki/index.php/ASL_error_messages#denied_RWX_mmap_of
If you do not find either the mprotect or mmap error messages occurring on your system check and you have not disabled the option at the link below, then ASL is not associated with the cause of your segfaults, and ASL is not causing them.
https://www.atomicorp.com/wiki/index.php/ASL_Configuration#RWXMAP_LOGGING
If you do see these events logged, then your system has this dangerous PHP vulnerability. Move on to step 2 below.
Step 2) Fix the vulnerability in PHP
Option 1) Use a package managed PHP
This is the easiest and best option. If your control panel company or system administrators are using source built versions of PHP, insist that they do not put execstack on their libraries. Its not needed and its very insecure. Doing so will open your system up to compromise, as as said before there is no need to dangerously misconfigure these libraries in this manner. They work just fine without it.
Option 2) Run ASL in scan and fix mode
ASL will look for php.ini files in standard locations and will analyze them to attempt to find any libraries that are included via your php.ini. If the insecure libraries are included via php.ini (which is usually the case, but not always especially with third party and source built PHP installs) ASL will then remove the unnecessary and highly insecure executable bit flags on those libraries. Just run this command as root to do this:
asl -s -f
Option 3) Clear the "executable stack" switch on these libraries manually
If ASL can not find your php.ini file, or if your libraries are loaded via some non-standard method you will need to manually remove these insecure flags from the libraries.
Simply run this command as root:
execstack -c /path/to/library
for example, if you wanted to fix the library "/usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so" you would run this command as root:
execstack -c /usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so
Or, if you want to fix the library "/usr/local/IonCube/ioncube_loader_lin_5.2.so", you would run this command as root:
execstack -c /usr/local/IonCube/ioncube_loader_lin_5.2.so
To find these libraries, assuming its ioncube and Zend that are causing this and not some other library, you can use these commands that are typically included with your operating system to typically find them:
locate -i ioncube | egrep ".so$" | egrep "lib|zendopt"
locate -i zend | egrep ".so$" | egrep "lib|zendopt"
locate -i zend | grep -i optimizer | egrep ".so$"
If you are using a third party PHP install, as with cpanel, these libraries may be stored in non-standard locations such as:
/usr/local/cpanel/3rdparty/php/53/zendopt/
Therefore you may need to use a broader search pattern, and you will need to contact CPanel to see where they may have installed these libraries.
Note: Check to make sure you have the "locate" command installed on your system.
Option 3) Quick method
This may work, if the vulnerable libraries are ioncube and zend. If you have other vulnerable libraries, you will need to find those libraries and follow the entire process in Option 3. You should also report this as a vulnerability to your PHP vendor.
execstack -c `locate -i ioncube | egrep ".so$" | grep lib`
execstack -c `locate -i zend | egrep ".so$" | grep lib`
execstack -c `locate -i zend | grep -i optimizer | egrep ".so$"`
If the vulnerable libraries are not ioncube and zend, then this will not work.
Option 4) Use an insecure kernel
This is the least secure option, is not necessary, and is recommended. This will leave your system open to root level compromise due to the vulnerability in PHP.
If you want to do this, you would simply use your the stock insecure kernel that came with your OS.
A secure kernel, such as the ASL kernel, will not allow libraries to do very unsafe things. This protects you from compromise by proactively protecting you from vulnerabilities in applications that do very dangerous things. An insecure kernel will not protect you from these things, and also will allow applications to do dangerous things on your system, and will make it possible an attacker to compromise your system
Tomcat segfaults
Solution: Tomcat uses Java, and Java performs certain actions that violate stack protection security models. To allow Tomcat to run in this manner, you simply need to run chpax to configure tomcat and Java to be allowed to do anything they want on your system, and to turn off all security protections for Java. This vulnerability exists on all platforms, with or without ASL, if you want to run Java.
paxctl -mps /path/to/java/bin/java
Where /path/to/java/bin/java is the path to your java vinary, for example:
paxctl -mps /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java
And do the same for tomcat, for example:
paxctl -mps /path/to/tomcat
You may also need to configure certain tomcat support tools to run without protection as well, gcj-dbtool is one such case:
paxctl -emrspx /usr/bin/gcj-dbtool
gcj-dbtool segfaults
Please see the above FAQ question.
execheap segfaults
See this article:
execstack segfaults
execbss segfaults
execdata segfaults
shlibbss segfaults
shlibdata segfaults
anonmap segfaults
apache segfaults
Examples:
[Fri Feb 1 01:00:00 2011] [notice] child pid 12345 exit signal Segmentation fault (11)
Or
kernel: grsec: From 1.2.3.4: Segmentation fault occurred at 0000003000003dbc in /usr/sbin/httpd[httpd:15804]
These are usually caused by a buggy web application, APR, or an apache module such as an accelerator, or other module. They are not caused by ASL. We have put together a courtesy page at the link below to assist you with these generic errors:
Other examples:
Sep 23 01:43:27 srv1 kernel: grsec: From 1.2.3.4: Segmentation fault occurred at 00000063000079bf in /usr/local/apache/bin/httpd[httpd:31167] uid/euid:99/99 gid/egid:99/99, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0
This generally occurs with systems that do not use a package managed version of apache, or apache compiled from source.
Some third party apache modules, such as ioncube and Zend Optimizer set their libraries to allow the stack to be executable (a very huge security vulnerability the kernel will prevent, and a setting that is not necessary for this libraries to work). These libraries do not need to be configured in this very insecure state to work. When apache tries to load these libraries the kernel will prevent these libraries from calling a special function, mprotect, in an insecure manner. This will cause apache to segfault.
ASL will try to detect these libraries when it is installed and fix them on installation. With package managed systems this happens automatically.
However, if you have an apache install compiled from source (as happens with cpanel), then ASL can not manage this software dynamically and fix these broken libraries. You will need to find the insecure third party library that apache is loading. In most cases this will be the ioncube and zend optimizer libraries, and in a package managed system (Such as Centos and Redhat) they are normally stored in locations such as (this is not a complete list):
/usr/lib64/php/ioncube/ /usr/lib/php/ioncube/
On non-package managed systems, such as cpanel systems, where the the package managed versions of PHP have been replaced with source compiled builds that location may be different. For example, cpanel usually compiles and installs these libraries and places them in locations similar to this (these may change, please contact cpanel or the party that build apache from source for more information):
/usr/local/IonCube/ /usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ /usr/local/Zend/ /usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/ (This is not a complete list, you will need to check what external libraries your source built apache has installed).
Solutions:
Option 1) Use a package managed apache
This is the easiest and best option. If your control panel company or system administrators are using source built versions of apache, insist that they do not put execstack on their libraries. Its not needed and its very insecure.
Option 2) Clear the "executable stack" switch on these libraries.
Simply run this command as root:
execstack -c /path/to/library
for example, if you wanted to fix the library "/usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so" you would run this command as root:
execstack -c /usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so
Or, if you want to fix the library "/usr/local/IonCube/ioncube_loader_lin_5.2.so", you would run this command as root:
execstack -c /usr/local/IonCube/ioncube_loader_lin_5.2.so
Comments
0 comments
Article is closed for comments.