Optimizing bash scripts for multi-core processors
Swapping Out to Other Machines
If you aren't satisfied with the efficiencies of parallel processing on a local machine, you can even apply this principle to the network. In that case, a first-level dispatcher could use TCP/IP to talk to multiple second-level dispatchers on various machines. The second-level dispatchers then talk to their local worker processes. This approach is only useful if you have a secure network, of course.
Conclusions
With just a couple of lines of code, you can use the techniques described in this article to parallelize existing shell scripts. Other scripting languages can use this approach; however, some languages offer superior alternatives. For example, Python uses explicit forking (os.fork()) in addition to pipes (os.pipe()), which allows for a low-level solution that is very close to the efficiency of the C programming language.
Infos
- Dynamic dispatcher source code: http://www.linux-magazine.com/resources/article_code
« Previous 1 2
Buy this article as PDF
(incl. VAT)