nixCraft

Hi,

I had a strange problem with my RHEL 5.
RHEL installed at Blade 460C with Quad Core.
with kernel-PAE-2.6.18-92.1.18.el5.i686.rpm installed.

This server attached with a SAN storage (MSA1000) with 2 fiber cable.
SAN storage divided into 2 partition (part A and part B)

Then, I try to simulate workload process.
I copied some files (around 10 files, each has 150 MB size). I run this process with 5 console instantly ( 5 consoles with copying process)

capture from top

--------------------------------------------------------------------
Tasks: 458 total, 1 running, 457 sleeping, 0 stopped, 0 zombie
Cpu0 : 1.3%us, 26.9%sy, 0.0%ni, 0.0%id, 71.4%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 1.3%us, 4.0%sy, 0.0%ni, 94.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8309944k total, 8130192k used, 179752k free, 274168k buffers
Swap: 16777208k total, 100k used, 16777108k free, 7362092k cached
--------------------------------------------------------------------

CPU0 has 0% idle, then this condition jumps to CPU6 then CPU 2, and so on..

My questions,
How to bind workload to all processor (distributed workload across the 8 CPUs), so idle 0% at one of the CPU is not happening again.

Thanks for you idea/comment...

BR,
b4ndit

Linux kernel will pickup CPU randomly or ideal CPU. You can override the kernel's process scheduling and bind a certain process to a specific CPU or all cpu using a command called taskset. See:

• Linux Setting processor affinity for a certain task or process

__________________
Vivek Gite
Linux Evangelist
Be proud RHEL user, and let the world know about your enterprise choices! Join RedHat user group.
Always use CODE tags for posting system output and commands!
Do you run a Linux? Let's face it, you need help

Thanks, great info.
Since this machine will mostly used its resource to accessing the SAN storage, could I dedicate
some CPU or CPUss) for this task?

Another question comes to my mind, Can we set a lower threshold for %idle at each processor?
My aim is to transfer processes at every CPU if its reach a certain %idle value.
Example: if CPU0 reach %idle value of 20, then (CPU1) or (CPU1 and CPU2) will help CPU0.

Many thanks...

Usually, CPU only dedicated to server such as XEN, VMWARE, Oracle database and so on. In case SAN, disk I/O handled by SAN and not by your Linux server. So I do not see any specific issue here..


No. Kernel does that for you. You do not have to adjust these values by hand. Either assign a specific CPU or let the kernel do the job for you.

__________________
Vivek Gite
Linux Evangelist
Be proud RHEL user, and let the world know about your enterprise choices! Join RedHat user group.
Always use CODE tags for posting system output and commands!
Do you run a Linux? Let's face it, you need help

Many thanks, I hope this will solve the issue

They (my supervisors) don't really satisfied with this one...

What They expect is the workload of every processor are nearly similar.
In linux affinity, we can set some process to some CPU.
They want to bind processes to all CPUs (8 CPUs we had acting as one group of a processor).
So, all of CPU %user,%system and other value nearly the same. (if we compare it to M$ Task Manager, the green lines of every cpu shows the same workload)

comments are very apprecited...

Thanks.

Hi,

The Kernel (Linux or others) on SMP architecture will automatically distribute the load across processors if there is a load to distribute...
- if multiple process are running and runnable (not waiting for I/O) then they will be dispatched across all available processor queues (if processor sets are not implemented)
- if a single process is running and runnable, it will be dispatched to single processor queue unless the process is multi-threaded.
- if the process is multi-threaded, it will be able to use entire cpu power (dispatched on all cpu queues) only if there are more runnable threads than cpu queues

Lastly, you do want process/cpu affinity: it avoid transferring cpu registers data from one processing queue to the other and improve performance...

Thus what your supervisor is asking for does not make sense !

Good day!

I see, that's why processes always jumped to another CPU..

I wonder, are there any possibilities that we could make those 8 CPU acting as one unity? so they could share same load ?
Like this one, I've read this...but I don't really understand to use it..
links:
Scheduling domains [LWN.net]

Again, comments are very...very...appreciated..

Thanks.

For Scheduling domains you need a kernel patch. Currently, only SGI (and may few other HPS provider got such patch). However, RHEL 6 and Suse 11 will support Scheduling domains. You need to wait some time or get patches from SGI or kernel.org.

HTH

__________________
Vivek Gite
Linux Evangelist
Be proud RHEL user, and let the world know about your enterprise choices! Join RedHat user group.
Always use CODE tags for posting system output and commands!
Do you run a Linux? Let's face it, you need help

I just did a little more research. The default is to run a process on all CPUs, giving a mask of
So command should be
Also, it depends upon your architectures like NUMA or SMP or uniprocessor and so on.

HTH

__________________
Vivek Gite
Linux Evangelist
Be proud RHEL user, and let the world know about your enterprise choices! Join RedHat user group.
Always use CODE tags for posting system output and commands!
Do you run a Linux? Let's face it, you need help