Introduction to Multiprocessing in Python

The multiprocessing package supports spawning processes using an API similar to the threading module. It also offers both local and remote concurrency. This tutorial will discuss multiprocessing in Python and how to use multiprocessing to communicate between processes and perform synchronization between processes, as well as logging.

Introduction to Multiprocessing

Multiprocessing works by creating a Process object and then calling its start() method as shown below.

from multiprocessing import Process


def greeting():
    print 'hello world'

if __name__ == '__main__':
    p = Process(target=greeting)
    p.start()
    p.join()

In the example code above, we first import the Process class and then instantiate the Process object with the greeting function which we want to run.

We then tell the process to begin using the start() method, and we finally complete the process with the join() method. 

Additionally, you can also pass arguments to the function by providing the args keyword argument like so:

from multiprocessing import Process


def greeting(name):
    print 'hello' + " " + name

if __name__ == '__main__':
    p = Process(target=greeting, args=('world',))
    p.start()
    p.join()

Example

Let's look at a more detailed example that covers all the concepts we have discussed above.

In this example, we are going to create a process that calculates the square of numbers and prints the results to the console.

from multiprocessing import Process


def square(x):

    for x in numbers:
        print('%s squared  is  %s' % (x, x**2))

if __name__ == '__main__':
    numbers = [43, 50, 5, 98, 34, 35]

    p = Process(target=square, args=('x',))
    p.start()
    p.join
    print "Done"
    
    
#result
Done
43 squared  is  1849
50 squared  is  2500
5 squared  is  25
98 squared  is  9604
34 squared  is  1156
35 squared  is  1225

You can also create more than one process at the same time, as shown in the example below, in which process p1 gets the results of numbers squared, while the second process p2 checks if the given numbers are even.

from multiprocessing import Process


def square(x):

    for x in numbers:
        print('%s squared  is  %s' % (x, x**2))


def is_even(x):

    for x in numbers:
        if x % 2 == 0:
            print('%s is an even number ' % (x))


if __name__ == '__main__':
    numbers = [43, 50, 5, 98, 34, 35]

    p1 = Process(target=square, args=('x',))
    p2 = Process(target=is_even, args=('x',))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    print "Done"
    
#result

43 squared  is  1849
50 squared  is  2500
5 squared  is  25
98 squared  is  9604
34 squared  is  1156
35 squared  is  1225
50 is an even number 
98 is an even number 
34 is an even number 
Done

Communication Between Processes

Multiprocessing supports two types of communication channels between processes:

  • Pipes
  • Queues

Queues

Queue objects are used to pass data between processes. They can store any pickle-able Python object, and you can use them as shown in the example below:

import multiprocessing


def is_even(numbers, q):
    for n in numbers:
        if n % 2 == 0:
            q.put(n)

if __name__ == "__main__":

    q = multiprocessing.Queue()
    p = multiprocessing.Process(target=is_even, args=(range(20), q))

    p.start()
    p.join()

    while q:
        print(q.get())

In the above example, we first create a function that checks if a number is even and then put the result at the end of the queue. We then instantiate a queue object and a process object and begin the process.

Finally, we check if the queue is empty, and if not, we get the values from the front of the queue and print them to the console.

We have shown how to share data between two processes using a queue, and the result is as shown below.

# result

0
2
4
6
8
10
12
14
16
18

It's also important to note that Python has a Queue module which lives in the process module and is used to share data between threads, unlike the multiprocessing queue which lives in shared memory and is used to share data between processes.

Pipes

Pipes in multiprocessing are primarily used for communication between processes. Usage is as simple as:

from multiprocessing import Process, Pipe

def f(conn):
    conn.send(['hello world'])
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()
    print parent_conn.recv()   
    p.join()

Pipe() returns two connection objects which represent the two ends of the pipe. Each connection object has send() and recv() methods. Here we create a process that prints the string hello world and then shares the data across.

Result

# result

['hello world']

Locks

Locks work by ensuring that only one process is executed at a time, hence blocking other processes from executing similar code. This allows the process to be completed, and only then can the lock be released.

The example below shows a pretty straightforward usage of the Lock method.

from multiprocessing import Process, Lock


def greeting(l, i):
    l.acquire()
    print 'hello', i
    l.release()

if __name__ == '__main__':
    lock = Lock()
    names = ['Alex', 'sam', 'Bernard', 'Patrick', 'Jude', 'Williams']

    for name in names:
        Process(target=greeting, args=(lock, name)).start()


#result
hello Alex
hello sam
hello Bernard
hello Patrick
hello Jude
hello Williams

In this code, we first import the Lock method, acquire it, execute the print function, and then release it.

Logging

The multiprocessing module also provides support for logging, although the logging package doesn't use locks so messages between processes might end up being mixed up during execution.

Usage of logging is as simple as:

import multiprocessing, logging
logger = multiprocessing.log_to_stderr()
logger.setLevel(logging.INFO)
logger.warning('Error has occurred')

Here we first import the logging and multiprocessing modules, and we then define the multiprocessing.log_to_stderr() method, which performs a call to get_logger() as well as adding a handler which sends output to sys.stderr. Finally, we set the logger level and the message we want to convey.

Conclusion

This tutorial has covered what is necessary to get started with multiprocessing in Python. Multiprocessing overcomes the problem of GIL (Global Interpreter Lock) since it leverages the use of subprocesses instead of threads.

There is much more in the Python documentation that isn’t covered in this tutorial, so feel free to visit the Python multiprocessing docs and utilize the full power of this module.



from Envato Tuts+ Tutorials

Comments

Post a Comment

Popular posts from this blog

How to Build Self-Confidence: 10+ Steps to Improve Your Self-Esteem

Image
Do you have friends who can walk up to someone at a party and they’d instantly hit it off? These are the people who often get free drinks, invites to cool events, and make new friends whenever they go out. You can gain confidence in yourself and improve different aspects of your life. (Image source:  Envato Elements ) Meanwhile, there are people who feel anxious in almost every social encounter. Their first day on a job, a big company meeting, and even a housewarming party all make them feel vulnerable and unsure of themselves. Questions like the following often spring to mind during these events:  “ What will I say? ”  “Who do I talk to?” “ What will I do if there’s an awkward silence? ”  While social awkwardness is common, it’s not a lifetime thing. You can learn how to build self-confidence that doesn’t feel forced or unnatural. You can still be yourself and have the self-confidence to try new things, meet new people, and all the benefits that come with...
Read more

Best of 2020: Great Landing Pages for Design Inspiration (Showcase Examples)

Image
A great landing page design is crucial when it comes to announcing new products or services as well as the launch of your company. Great landing pages help visitors focus on a singular action. That makes them great for driving conversions and signups. Zimed is one of our best landing page design examples on Envato Elements. Nowadays, you don’t have to be a web design expert to create a landing page. There's no shortage of  quality landing page samples on various marketplaces so all you need to do is find the one you like, buy it, and set it up. That's why we've set up this landing page showcase. In today's post, we'll share our best landing page examples from Envato Elements and ThemeForest. Find Great Landing Page Design Examples on Envato Elements for 2020 Envato Elements is a great starting point when it comes to the best  landing page design . Envato Elements is a subscription-based marketplace with dozens of landing page templates. When you start bro...
Read more

18+ Best Free PowerPoint Puzzle Pieces Templates (Infographic PPT Slides)

Image
Many people are visual learners. The population of visual learners is why it helps to use visual infographics to explain an idea in your PowerPoint presentation.  One of the best examples is using a PowerPoint puzzle pieces template.  Each piece of the puzzle can fit together into a singular idea. Templates like the premium  PANDAWA  template have all the PowerPoint puzzle pieces you need to explain a complex idea. Using a PowerPoint puzzle pieces will help you explain concepts more clearly. You can quickly and easily change professionally designed PowerPoint puzzles to fit your needs. This round-up will show you some of the best free PowerPoint puzzle templates. You'll also see top PowerPoint puzzle pieces templates from Envato Elements, and more puzzle pieces templates from GraphicRiver. Download the Best PowerPoint Puzzle Templates From Envato Elements Later in this tutorial, you'll see free template options. But for now, let's focus on the bes...
Read more