From: rhcelinuxguide.wordpress.com and other sources.<\/p>\n
Overview:
\nThere are two main kinds of kernel panics:
\n1) Hard Panic (also known as Aieee! )
\n2) Soft Panic (also known as Oops )<\/p>\n
Most panics are the result of unhandled processor exceptions in kernel code, such as references to invalid memory addresses and can also be caused by one or more of the following issues:<\/p>\n
# Defective or incompatible RAM are the most frequent causes of kernel panics.<\/p>\n
# Incompatible, obsolete, or corrupted kernel extensions. If a kernel
\nextension or one of its dependencies is corrupted, such as the result of
\nhard disk corruption, kernel panics are likely to occur when the kernel
\nattempts to load or execute the extension.<\/p>\n
# Incompatible, obsolete, or corrupted drivers. Similar to kernel extensions,
\ndrivers for third-party hardware which are incompatible with the OS version
\nyou are using, or which have become corrupted, will cause in kernel panics.<\/p>\n
# Hard disk corruption, including bad sectors, directory corruption, and other
\nhard-disk ills.<\/p>\n
# Incorrect permissions on System-related files or folders.<\/p>\n
# Insufficient RAM and available hard disk space.<\/p>\n
# Improperly installed hardware or software.<\/p>\n
# Defective hardware or software. Hardware failures, including a defective
\nCPU, or programming errors can result in kernel panics.<\/p>\n
# Incompatible hardware.<\/p>\n
Problem Description:
\nKernel panics on Linux are hard to identify and troubleshoot. Troubleshooting kernel panics often requires reproducing a situation that occurs rarely and collecting data that is difficult to gather.<\/p>\n
Solution Summary:
\nThis document outlines several techniques that will help reduce the amount of time necessary to troubleshoot a kernel panic.<\/p>\n
Technical Discussion:
\nWhat is a kernel panic?
\nAs the name implies, the Linux kernel gets into a situation where it doesn\u2019t know what to do next. When this happens, the kernel gives as much information as it can about what caused the problem, depending on what caused the panic.<\/p>\n
There are two main kinds of kernel panics:<\/p>\n
1. Hard Panic \u2013 also known as Aieee!
\n 2. Soft Panic \u2013 also known as Oops<\/p>\n
What can cause a kernel panic?
\nOnly modules that are located within kernel space can directly cause the kernel to panic. To see what modules are dynamically loaded, do lsmod \u2013 this shows all dynamically loaded modules (Dialogic drivers, LiS, SCSI driver, filesystem, etc.). In addition to these dynamically loaded modules, components that are built into the kernel (memory map, etc.) can cause a panic.<\/p>\n
Since hard panics and soft panics are different in nature, we will discuss how to deal with each separately.<\/p>\n
How to Troubleshoot a Hard Kernel Panic
\nHard Panics \u2013 Symptoms:<\/p>\n
1. Machine is completely locked up and unusable.
\n 2. Num Lock \/ Caps Lock \/ Scroll Lock keys usually blink.
\n 3. If in console mode, dump is displayed on monitor (including the phrase \u201cAieee!\u201d).
\n 4. Similar to Windows Blue Screen.<\/p>\n
Hard panics \u2013 causes:
\nThe most common cause of a hard kernel panic is when a driver crashes within an interrupt handler, usually because it tried to access a null pointer within the interrupt handler. When this happens, that driver cannot handle any new interrupts and eventually the system crashes. This is not exclusive to Dialogic drivers.<\/p>\n
Hard panics \u2013 information to collect:
\nDepending on the nature of the panic, the kernel will log all information it can prior to locking up. Since a kernel panic is a drastic failure, it is uncertain how much information will be logged. Below are key pieces of information to collect. It is important to collect as many of these as possible, but there is no guarantee that all of them will be available, especially the first time a panic is seen.<\/p>\n
1. \/var\/log\/messages \u2014 sometimes the entire kernel panic stack trace will be logged there
\n 2. Application \/ Library logs (RTF, cheetah, etc.) \u2013 may show what was happening before the panic
\n 3. Other information about what happened just prior to the panic, or how to reproduce
\n 4. Screen dump from console. Since the OS is locked, you cannot cut and paste from the screen. There are two common ways to get this info:
\n * Digital Picture of screen (preferred, since it\u2019s quicker and easier)
\n * Copying screen with pen and paper or typing to another computer<\/p>\n
If the dump is not available either in \/var\/log\/message or on the screen, follow these tips to get a dump:<\/p>\n
1. If in GUI mode, switch to full console mode \u2013 no dump info is passed to the GUI (not even to GUI shell).
\n 2. Make sure screen stays on during full test run \u2013 if a screen saver kicks in, the screen won\u2019t return after a kernel panic. Use these settings to ensure the screen stays on.
\n * setterm -blank 0
\n * setterm -powerdown 0
\n * setvesablank off
\n 3. From console, copy dump from screen (see above).<\/p>\n
Hard panics \u2013 Troubleshooting when a full trace is available
\nThe stack trace is the most important piece of information to use in troubleshooting a kernel panic. It is often crucial to have a full stack trace, something that may not be available if only a screen dump is provided \u2013 the top of the stack may scroll off the screen, leaving only a partial stack trace. If a full trace is available, it is usually sufficient to isolate root cause. To identify whether or not you have a large enough stack trace, look for a line with EIP, which will show what function call and module caused the panic. In the example below, this is shown in the following line:
\nEIP is at _dlgn_setevmask [streams-dlgnDriver] 0xe<\/p>\n
If the culprit is a Dialogic driver you will see a module name with:
\nstreams-xxxxDriver (xxxx = dlgn, dvbm, mercd, etc.)<\/p>\n
Hard panic \u2013 full trace example:
\nUnable to handle kernel NULL pointer dereference at virtual address 0000000c
\nprinting eip:
\nf89e568a
\n*pde = 32859001
\n*pte = 00000000
\nOops: 0000
\nKernel 2.4.9-31enterprise
\nCPU: 1
\nEIP: 0010:[] Tainted: PF
\nEFLAGS: 00010096
\nEIP is at _dlgn_setevmask [streams-dlgnDriver] 0xe
\neax: 00000000 ebx: f65f5410 ecx: f5e16710 edx: f65f5410
\nesi: 00001ea0 edi: f5e23c30 ebp: f65f5410 esp: f1cf7e78
\nds: 0018 es: 0018 ss: 0018
\nProcess pwcallmgr (pid: 10334, stackpage=f1cf7000)
\nStack: 00000000 c01067fa 00000086 f1cf7ec0 00001ea0 f5e23c30 f65f5410 f89e53ec
\nf89fcd60 f5e16710 f65f5410 f65f5410 f8a54420 f1cf7ec0 f8a4d73a 0000139e
\nf5e16710 f89fcd60 00000086 f5e16710 f5e16754 f65f5410 0000034a f894e648
\nCall Trace: [setup_sigcontext+218\/288] setup_sigcontext [kernel] 0xda
\nCall Trace: [] setup_sigcontext [kernel] 0xda
\n[] dlgnwput [streams-dlgnDriver] 0xe8
\n[] Sm_Handle [streams-dlgnDriver] 0\u00d71ea0
\n[] intdrv_lock [streams-dlgnDriver] 0\u00d70
\n[] Gn_Maxpm [streams-dlgnDriver] 0\u00d78ba
\n[] Sm_Handle [streams-dlgnDriver] 0\u00d71ea0
\n[] lis_safe_putnext [streams] 0\u00d7168
\n[] __insmod_streams-dvbmDriver_S.bss_L117376 [streams-dvbmDriver] 0xab8 [] dvbmwput [streams-dvbmDriver] 0\u00d76f5
\n[] dvwinit [streams-dvbmDriver] 0\u00d72c0
\n[] lis_safe_putnext [streams] 0\u00d7168
\n[] lis_strputpmsg [streams] 0\u00d754c
\n[] __insmod_streams_S.rodata_L35552 [streams] 0\u00d7182e
\n[] sys_putpmsg [streams] 0\u00d76f
\n[system_call+51\/56] system_call [kernel] 0\u00d733
\n[] system_call [kernel] 0\u00d733
\nNov 28 12:17:58 talus kernel:
\nNov 28 12:17:58 talus kernel:
\nCode: 8b 70 0c 8b 06 83 f8 20 8b 54 24 20 8b 6c 24 24 76 1c 89 5c<\/p>\n
Hard panics \u2013 Troubleshooting when a full trace is not available
\nIf only a partial stack trace is available, it can be tricky to isolate the root cause, since there is no explicit information about what module of function call caused the panic. Instead, only commands leading up to the final command will be seen in a partial stack trace. In this case, it is very important to collect as much information as possible about what happened leading up to the kernel panic (application logs, library traces, steps to reproduce, etc).<\/p>\n
Hard panic \u2013 partial trace example (note there is no line with EIP information)
\n[] ip_rcv [kernel] 0\u00d7357
\n[] sramintr [streams_dlgnDriver] 0\u00d732d
\n[] lis_spin_lock_irqsave_fcn [streams] 0\u00d77d
\n[] inthw_lock [streams_dlgnDriver] 0\u00d71c
\n[] pwswtbl [streams_dlgnDriver] 0\u00d70
\n[] dlgnintr [streams_dlgnDriver] 0\u00d74b
\n[] Gn_Maxpm [streams_dlgnDriver] 0\u00d77ae
\n[] __run_timers [kernel] 0xd1
\n[] handle_IRQ_event [kernel] 0\u00d75e
\n[] do_IRQ [kernel] 0xa4
\n[] default_idle [kernel] 0\u00d70
\n[] default_idle [kernel] 0\u00d70
\n[] call_do_IRQ [kernel] 0\u00d75
\n[] default_idle [kernel] 0\u00d70
\n[] default_idle [kernel] 0\u00d70
\n[] default_idle [kernel] 0\u00d72d
\n[] cpu_idle [kernel] 0\u00d72d
\n[] __call_console_drivers [kernel] 0\u00d74b
\n[] call_console_drivers [kernel] 0xeb
\nCode: 8b 50 0c 85 d2 74 31 f6 42 0a 02 74 04 89 44 24 08 31 f6 0f
\n<0> Kernel panic: Aiee, killing interrupt handler!
\nIn interrupt handler \u2013 not syncing<\/p>\n
Hard panics \u2013 using kernel debugger (KDB)
\nIf only a partial trace is available and the supporting information is not sufficient to isolate root cause, it may be useful to use KDB. KDB is a tool that is compiled into the kernel that causes the kernel to break into a shell rather than lock up when a panic occurs. This enables you to collect additional information about the panic, which is often useful in determining root cause.<\/p>\n
Some important things to note about using KDB:<\/p>\n
1. If this is a potential Dialogic issue, technical support should be contacted prior to the to use of KDB
\n 2. Must use base kernel \u2013 i.e. 2.4.18 kernel instead of 2.4.18-5 from RedHat. This is because KDB is only available for the base kernels, and not the builds created by RedHat. While this does create a slight deviation from the original configuration, it usually does not interfere with root cause analysis.
\n 3. Need different Dialogic drivers compiled to handle the specific kernel.<\/p>\n
How to Troubleshoot a Soft Kernel Panic
\nSoft panics \u2013 symptoms:<\/p>\n
1. Much less severe than hard panic.
\n 2. Usually results in a segmentation fault.
\n 3. Can see an oops message \u2013 search \/var\/log\/messages for string \u2018Oops\u2019.
\n 4. Machine still somewhat usable (but should be rebooted after information is collected).<\/p>\n
Soft panics \u2013 causes:
\nAlmost anything that causes a module to crash when it is not within an interrupt handler can cause a soft panic. In this case, the driver itself will crash, but will not cause catastrophic system failure since it was not locked in the interrupt handler. The same possible causes exist for soft panics as do for hard panics (i.e. accessing a null pointer during runtime).<\/p>\n
Soft panics \u2013 information to collect:
\nWhen a soft panic occurs, the kernel will generate a dump that contains kernel symbols \u2013 this information is logged in \/var\/log\/messages. To begin troubleshooting, use the ksymoops utility to turn kernel symbols into meaningful data.<\/p>\n
To generate a ksymoops file:<\/p>\n
1. Create new file from text of stack trace found in \/var\/log\/messages. Make sure to strip off timestamps, otherwise ksymoops will fail.
\n 2. Run ksymoops on new stack trace file:
\n Generic: ksymoops -o [location of Dialogic drivers] filename
\n Example: ksymoops -o \/lib\/modules\/2.4.18-5\/misc ksymoops.log
\n All other defaults should work fine<\/p>\n
For a man page on ksymoops, see the following webpage:
\nhttp:\/\/gd.tuwien.ac.at\/linuxcommand.org\/man_pages\/ksymoops8.html<\/p>\n
####################################################################<\/p>\n
So you\u2019re trying to start Linux for the first time and \u2026 wham! You get messages like:<\/p>\n
* Unable to mount root device.
\n * Kernel panic \u2013 not syncing.<\/p>\n
What do I do now? Oh, how I love Windows \u2026<\/p>\n
Here\u2019s the scoop \u2026<\/p>\n
(1) The first part of the system that starts running is the \u201cboot loader,\u201d usually grub. This is the program that loads Linux, and\/or Windows if you so desire. (The \u201cmaster boot record,\u201d or MBR, enables the computer to load grub.)<\/p>\n
(2) The first thing that Grub needs to know is \u2026 \u201cwhere is the kernel?\u201d It gets this from the \/boot\/grub\/grub.conf file. The way that you specify the correct drive and partition in Grub is a little different from, like \u201c(hd0,0)\u201d what you use in ordinary Linux. The kernel will be in some file named \u201cvmlinuz-\u2026\u201d<\/p>\n
(3) Once Grub has loaded the kernel into memory, the first thing that the kernel needs to know is, \u201cwhere is the root filesystem?\u201d The root= parameter is passed to the kernel to provide this information. Notice that now you are talking \u201cto Linux,\u201d and you identify devices \u201cin Linux\u2019s terms,\u201d like \u201c\/dev\/hda2?.<\/p>\n
(4) Given this information, Linux is going to try to mount the root filesystem \u2026 prepare it for use. The most common mistake at this point is that you\u2019ve specified the wrong device in step #3. Unfortunately, the message that results is rather nasty looking\u2026<\/p>\n
When Linux doesn\u2019t know how to proceed, as in this case, it says \u201ckernel panic\u201d and it stops. But, even then, it tries to go down gracefully. It tries to write anything to disk that hasn\u2019t been written out (an operation called \u201csyncing\u201d, for some darn-fool reason), and if it succeeds in doing so it will say \u201cnot syncing.\u201d What\u2019s totally misleading about this message combination is that it implies, incorrectly, that the reason for the panic is \u201cnot syncing,\u201d when actually the reason for the panic will be found in the preceding few lines.<\/p>\n
You might see the message, \u201ctried to kill \u2018init\u2019.\u201d That really means that a program called init died\u2026 which it is not allowed to ever do. init is a very special program in Linux\u2026 the first program created when the machine starts.<\/p>\n
So, basically, when you get these messages on startup \u2026 the situation is really a lot more dreadful looking than it actually is. You have probably just made a \u201ctpyo\u201d when entering the information in grub.conf.<\/p>\n
(Another common place to make a typo is in \/etc\/fstab, which tells Linux where all the other drives are.)<\/p>\n
So what do you do? If you\u2019re doing a first-time install you can just start over. Otherwise, you need to boot a separate CD-ROM, which will give you a stand-alone Linux installation from which you can edit the offending files.<\/p>\n
Explained: \u201ckernel panic \u2013 not syncing \u2013 attempted to kill init\u201d<\/p>\n
When the kernel gets into a situation where it does not know how to proceed (most often during booting, but at other times), it issues a kernel panic by calling the panic(msg) routine defined in kernel\/panic.c. (Good name, huh?) This is a call from which No One Ever Returns.<\/p>\n
The panic() routine adds text to the front of the message, telling you more about what the system was actually doing when the panic occurred \u2026 basically how big and bad the trail of debris in the filesystem is likely to be. This is where the \u201cnot syncing\u201d part comes from, and when you see that, it\u2019s good. (panic() does try to issue a sinc() system-call to push all buffered data out to the hard-disks before it goes down.)<\/p>\n
The second part of the message is what was provided by the original call to panic(). For example, we find panic(\u201cTried to kill init!\u201d) in kernel\/exit.c.<\/p>\n
So, what does this actually mean? Well, in this case it really doesn\u2019t mean that someone tried to kill the magical init process (process #1\u2026), but simply that it tried to die. This process is not allowed to die or to be killed.<\/p>\n
When you see this message, it\u2019s almost always at boot-time, and the real messages \u2026 the cause of the actual failure \u2026 will be found in the startup messages immediately preceding this one. This is often the case with kernel-panics. init encountered something \u201creally bad,\u201d and it didn\u2019t know what to do, so it died, so the kernel died too.<\/p>\n
BTW, the kernel-panic code is rather cute. It can blink lights and beep the system-speaker in Morse code. It can reboot the system automagically. Obviously the people who wrote this stuff encountered it a lot\u2026<\/p>\n
In diagnosing, or at least understanding, kernel-panics, I find it extremely helpful to have on-hand a copy of the Linux source-code, which is usually stored someplace like \/usr\/src\/linux-2.x. You can use the grep utility to locate the actual code which caused the panic to occur.<\/p>\n","protected":false},"excerpt":{"rendered":"
From: rhcelinuxguide.wordpress.com and other sources. Overview: There are two main kinds of kernel panics: 1) Hard Panic (also known as Aieee! ) 2) Soft Panic (also known as Oops ) Most panics are the result of unhandled processor exceptions in kernel code, such as references to invalid memory addresses and can also be caused by… <\/p>\n