The Python Language: Multi-Processing

Python has built-in multiprocessing module in its standard library.

One simple example of using the Pool class is the following.

def myfunc(myfuncargs):
    'some thing here'

with Pool(10) as p:
    records = p.map(myfunc, myfuncargs)

However, there are limitations on this, especially on pickles. Another approach.

from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool

with ThreadPool(1) as p:
    records = p.map(myfunc, myfuncargs)

Beware that map function will feed in a list of args to the function. So I have to use p.map(myfunc, [arg]) for one arg.