“A context switch is the process of storing and restoring the state (context) of a process or thread”, it is also “an essential feature of a multitasking operating system” according to Wikipedia.
In this post we go a little deeper into detail on why context switching might be expensive and how you can spot these hidden costs. First I read the fantastic article on Tsuna’s blog titled: “How long does it take to make a context switch?” In the end Benoit Sigoure writes:
Context switching is expensive. My rule of thumb is that it’ll cost you about 30µs of CPU overhead.
Okay, that is quite clear 
Now how do we measure the context switches of an application? I wrote “time_ctxt_switches.sh” in bash:
#!/bin/bash T="$(date +%s)" $@ > /dev/null & pid=$! watch -n1 grep ctxt /proc/$pid/status V="$(cat /proc/$pid/status | grep ^voluntary_ctxt_switches | cut -f2)" N="$(cat /proc/$pid/status | grep ^nonvoluntary_ctxt_switches | cut -f2)" T="$(($(date +%s)-T))" echo "voluntary_ctxt_switches: $((V/T))/sec" echo "nonvoluntary_ctxt_switches: $((N/T))/sec" kill -9 $pid
This scripts gets the time in seconds. Then it starts whatever you put as arguments and pipes the output to /dev/null. It backgrounds the process and stores the PID. We watch the context switches in real time until you press Ctrl-C. After that we get the amount of context switches and we get the difference in amount of seconds. We calculate some average and then we finally (force) kill the application.
Inotify in Python
After the last post we are particularly interested in the context switching overhead of inotify in Python. I tested a simple Python inotify script using:
wget https://github.com/seb-m/pyinotify/blob/master/python2/pyinotify.py bash time_ctxt_switches.sh python2 pyinotify.py /var/log/nginx/access.log
And in a separate terminal I ran:
siege -c 100 http://localhost/
I’ve found that the inotify script does about:
voluntary_ctxt_switches: 125/sec nonvoluntary_ctxt_switches: 23/sec
148 x 30 microseconds = 4.44 ms CPU just for the context switching each second. 4 ms out of 1000 ms = 0.4% CPU load. And note that this is expected to linearly increase with a higher load on the Nginx server, because this is now doing “only” 200 requests per second.
Nginx buffered access logging
My colleague Carlos pointed out that I was wrong in my last post and that inotify is great choice for looking at Nginx access logs (and he is right). The only thing you need to change is that you need to reduce the amount I/O operations that Nginx does for the logging. By default the access log is not buffered, but you can easily change that in /etc/nginx/nginx.conf:
access_log /var/log/nginx/access.log;
To enable buffered logging change this into:
access_log /var/log/nginx/access.log combined buffer=1024k flush=5s;
Now Nginx will only write every 1024k to disk or every 5 seconds, whatever comes first. You can easily verify the improved behavior using the “time_ctxt_switches.sh” bash script.
The post Linux context switching costs of inotify in Python appeared first on LeaseWeb Labs.