Normally when I see a problem to do with CPU, I/O, network or memory resource hogging I turn to my trusty tools such as vmstat, iostate or ifstat to check system behaviour.
I stumbled upon dstat today, which boasts to be "a versatile replacement for vmstat, iostat and ifstat.". So let's see what it can do. First, install with:
sudo apt-get install dstat
Running dstat with no arguments displays CPU stats (user,system,idle,wait,hardware interrupts, software interrupts), Disk I/O stats (read/write), Network stats (receive,send), Paging stats (in/out) and System stats (interrupts, context switches). Not bad at all.
Dstat even highlights in colour the active values, so you don't miss relevant deltas in the statistics being churned out by the tool. Colours can be disabled with the --nocolor option.
But there is more. There are a plethora of options to show various system activities, such as:
memory stats (used, buffers, cache, free)
process stats (runnable, uninterruptible, new)
aio stats (asynchronous I/O)
filesystem stats (open files, inodes)
ipc stats (message queue, semaphores, shared memory)
socket stats (total, tcp, udp, raw, ip-fragments)
TCP stats (listen, established, syn, time_wait, close)
UDP stats (listen, active)
virtual memory stats (hard pagefaults, soft pagefaults, allocated, free)
..to name but a few. As it is, this already is more functional that vmstat, iostat and ifstat. But that's not all - dstat is extensible by the use of dstat plugins and the packaged version of dstat comes shipped with a few already, for example:
top block I/O activity by process
most expensive CPU process
most expensive I/O process
process with highest latency
..and many more besides - consult the manual page for dstat to see more.
So, dstat really is a Swiss army knife - a useful tool to get to know and use whenever you need to quickly spot misbehaving processes or devices.