Introduction
If you are using a laptop then powertop is a useful tool to help reduce power consumption. powertop examines system activity such as the reasons for processor wakeups and reports this periodically.
On some laptops users have seen alarming amounts of "Recheduling Interrupts" (Kernel IPI events) on Hardy systems, which can be caused by several reasons, from incorrect BIOS settings to misbehaving applications. This page will describe how to track down and reduce the amount of processor wakeups, and hopefully increase battery life on a laptop.
What are Rescheduling Interrupts?
With multi core machines, such as Intel Centrino Duos, the scheduler tries to spread processor activity across as many cores as possible. The general rule of thumb is that it is preferable to have as many processes running on all the cores in lower power (lower clock frequencies) rather than have one core really busy running at full speed while other cores are sleeping.
When the scheduler decides to offload work from one core to another sleeping core, a kernel IPI "message" is sent to a sleeping core to wake it up from a low power sleep to make it start running a process. These IPI events are reported by powertop as "Rescheduling Interrupts". The information is available by reading /proc/interrupts - the IPI events are reported as an interrupt.
Generally speaking wakeup events are caused by timer wakeups or interrupts from hardware and these causes processes to be woken up and get scheduled to run. The scheduler will generate IPIs when it deems it necessary to put these woken up processed onto a sleeping core, hence a system with lots of wakeups or interrupts may manifest many "Rescheduling Interrupts".
If powertop is reporting many hundreds or possibly thousands of "Rescheduling Interrupts" then there probably is a problem with a BIOS setting (misconfigured interrupt setup) or a misbehaving application (generating lots of wakeup from timer in threads).
Troubleshooting ACPI and APIC problems
The first thing to try is to see if the ACPI is not configured correctly causing the kernel to configure hardware sub-optimally.
1. Try booting with the "acpi=noirq" kernel parameter
- This will disable IRQ configuration derived from ACPI. If there is no difference with noirq enabled or disabled, this may not be an ACPI IRQ configuration issue.
2. Try booting with the "noapic" kernel parameter
- This will disable the IO-APIC configuration. Again, if there is no difference with it enabled or disabled, this is not an APIC problem.
3. Try booting with "nolapic" kernel parameter
- This disables local APIC. Again, of there is no difference with it enabled or disabled, this is not a local APIC problem.
4. Try booting with the "acpi=off" kernel parameter
- This disables ACPI totally. Again, of there is no difference with it enabled or disabled, this is not a local ACPI problem.
If any of the above solves the problem, then it is advisable to:
1. Check if there is a BIOS upgrade for your machine to see if this fixes the problem. 2. Failing this, one may need to fix one's ACPI configuration, see the ACPIBattery wiki page for more help on how to do this.
Troubleshooting misconfigured hardware
Of course, hardware maybe misconfigured, causing a device to cause many hundreds of interrupts a second which cause (spurious) wakeups.
Interrupts can be views by cat'ing /proc/interrupts and seeing which interrupt(s) are overly busy. It is worth taking two captures of /proc/interrupts with a 10 second interval between captures and then comparing the difference to find out the busy interrupt line.
Some may be very legitimate, for example, busy ethernet, audio and video hardware, so some judgment is called for to figure out if an interrupt is being correctly generated or not.
If some hardware is misbehaving and producing spurious interrupts it is worth seeing if a kernel module driver that handles the specific interrupt can be removed to see if this stops the problem or not. Failing this, it is worth filing a bug report.
Troubleshooting applications
The final thing to check is to see if the problem is caused by an application that is doing (too many) frequent wakeups. Some applications such as media players have threads that feed the audio and video hardware with data frequently at precise periodic times and can cause many hundreds of wakeup events per second, for example, Amarok when playing back mp3s.
Other applications may have many threads doing animation, for example, the Opera web browser may have several pages open simultaneously, and on each page there may be flash based animations which all can contribute to many tens or hundreds of wakeups per second.
There are several tools available to examine application behavior to pin point rogue programs:
1. Use vmstat 1 to see how busy the overall system is.
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 15868 27716 58212 379880 0 0 38 106 143 529 7 1 90 2 0 0 15868 27528 58212 379904 0 0 0 0 167 354 1 0 99 0 0 0 15868 27596 58220 379904 0 0 0 52 398 1126 11 4 86 0 ^C
Look at the "in" column to see the number of interrupts a second. If this is very high, then go back and check the interrupts on your system as described in the previous steps above.
Look at the "cs" column to see the number of context switches a second - if this is high there may be (many) applications sleeping and waking causing the scheduler to generate IPI events which show up as "Rescheduling Interrupts".
You may see some tight coupling between 2 processes or threads, e.g. a producer/consumer arrangement where a process does some work and wakes up another to process something and then once this is done the consumer sleeps again waking for the producer. If this occurs very frequently you may see a high amount of context switches a high amount of legitimate "Rescheduling Interrupts" - which is a good sign as it means the scheduler is working to get a good spread of load across the processor cores.
2. Use top to see the which are the busiest processes on your system and see if stopping these applications reduces the wakeups. Sometimes the busiest processes are causing the problem with frequent short sleeps and many hundreds of wakeups.
top -b -n 1
top - 09:53:07 up 2:15, 4 users, load average: 1.01, 0.53, 0.29
Tasks: 153 total, 1 running, 152 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.1%us, 1.3%sy, 0.1%ni, 89.7%id, 1.7%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 1026056k total, 1010256k used, 15800k free, 49492k buffers
Swap: 3164764k total, 38048k used, 3126716k free, 251244k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5889 root 20 0 447m 122m 12m S 14 12.2 6:20.88 Xorg
22541 joeuser 20 0 98196 77m 7580 S 12 7.7 0:20.34 frozen-bubble
6290 joeuser 20 0 28920 4584 3800 S 8 0.4 0:09.16 pulseaudio
22266 joeuser 20 0 127m 30m 20m S 6 3.0 0:10.75 amarokapp
6384 joeuser 20 0 25832 18m 7420 S 4 1.8 0:38.22 compiz.real
22588 root 20 0 84460 18m 10m S 4 1.8 0:01.46 etherape
22703 joeuser 20 0 94952 30m 19m S 2 3.1 0:02.51 ekiga
1 root 20 0 2844 1664 520 S 0 0.2 0:01.50 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/0 If you find an application generating far too many wakeups per second, it may be worth filing a bug report against it or trying to fix the problem yourself and supply a patch. Alternatively, finding an alternative application that is aggressive in wakeups may be worth pursuing.