There are many questions in SO regarding passing multiple arguments in python multiprocessing Pool's starmap method. But what I want to ask is if I can send a queue object in the method which can be shared between different processes?
I am able to do this using threading and multiprocessing Process method, but not using Pool's starmap method.
from multiprocessing import Process, Queue, Pool def get_data(pageNo,q): q.put(pageNo*pageNo) if __name__ == "__main__": q = Queue() p = {} no_pages = 5 pool_tuple = [(x,q) for x in range(1,no_pages)] with Pool(processes=3) as pool: pool.starmap(get_data, pool_tuple)
What I want is that I should be able to put the result in the queue from different processes. The error I am getting is: Queue objects should only be shared between processes through inheritance
Am I doing something wrong in the way I am passing the queue object? Since if Process supports this, so should the Pool method be able to.
score:1
try using Manager()
like this:
from multiprocessing import Manager, Pool
def get_data(pageNo, q):
q.put(pageNo * pageNo)
if __name__ == "__main__":
m = Manager()
q = m.Queue()
p = {}
no_pages = 5
pool_tuple = [(x, q) for x in range(1, no_pages)]
with Pool(processes=3) as pool:
pool.starmap(get_data, pool_tuple)
for i in range(1, no_pages):
print("result", i, ":", q.get())
Output:
result 1 : 1
result 2 : 4
result 3 : 9
result 4 : 16
Credit To: stackoverflow.com
Related Query
- Can I pass queue object in multiprocessing pool starmap method
- Can I replace an existing method of an object in Python?
- How to pass a sqlite Connection Object through multiprocessing
- Can you pass an object to a celery task by its memory id using ctypes?
- Pass object method as lambda or partial function, replacing self parameter with target object?
- How can I pass an object attribute to lambda function, as function parameter?
- Using python 3.x how can I pass a Tree object from ete3 to DendroPy without writing to file
- Passing function to map or starmap in multiprocessing pool
- How to pass a nested params object for post method of requests library in python3?
- Class object can only call a method if object attribute is between a range
- List method in python returns empty list when we pass magic mock object
- the position between Pool object and Manager object in multiprocessing module
- Python 3.5 multiprocessing pool and queue don't work
- How can I test that a method on an imported module is called during object construction
- What is the best way to pass a method when I initialize an object
- Can I implicitly pass attributes of an object to one of its other attributes in python
- Python multiprocessing - probably failure when method name pass to Process
- How can I pass positional arguments to my function object set as a value in a dictionary?
- How to pass object from main process to child process using multiprocessing
- What's the difference between ThreadPool vs Pool in the multiprocessing module?
- Get defining class of unbound method object in Python 3
- Can we overload behavior of class object
- Mock a method of a mocked object in Python?
- Python 3.4 multiprocessing Queue faster than Pipe, unexpected
- How can I correct the error ' AttributeError: 'dict_keys' object has no attribute 'remove' '?
- Get length of Queue in Python's multiprocessing library
- How can I make an object with an interface like a random number generator, but that actually generates a specified sequence?
- how can i fix AttributeError: 'dict_values' object has no attribute 'count'?
- Why changing start method to 'spawn' from 'fork' in Python multiprocessing does not allow me run my job anymore?
- How can I check if an object is a file with isinstance()?
- multiprocessing Queue subclass issue
- Memory not freed after Python's multiprocessing Pool is finished
- How can I convert bytes object to decimal or binary representation in python?
- Can you set an attribute to a method in python
- Can I change a Python bound method object's __str__() attribute?
- Can I use multiprocessing.Pool in a method of a class?
- Multiprocessing Pool with a for loop
- How can I properly mimic this encryption method to produce the proper value for the encryptedPwd field?
- DeprecationWarning: executable_path has been deprecated, please pass in a Service object
- Python 3: How can object be instance of type?
- How can I type-hint a nested object in Python?
- How can I use functional programming to make a generic method in python?
- Mutliprocessing Queue vs. Pool
- Python Multiprocessing Queue when set to infinite is capped at 32768 (2^15)
- How can I use a static method as a default parameter for the strategy design pattern?
- Can you add attributes to an object dynamically?
- Custom exceptions are not raised properly when used in Multiprocessing Pool
- How can I pass keyword arguments as parameters to a function?
- Cannot subclass multiprocessing Queue in Python 3.5
- How can I copy style of an existing Line2D object to a plot() call? (matplotlib)
More Query from same tag
- Not able to import camelot in Python 3.7(Anaconda) in MacOS Catalina
- Find Integers in list
- Soundex Function for Turkish Language in Python
- How to add an image in pyqt qdock widget
- How to reuse a custom @contextlib.contextmanager?
- How to run Docker with python and Java?
- how to prefix with 0x and separate by , a list of numbers in python
- Autocomplete for gtk3 and pygobject
- XlsxWriter: lock only specific cells
- Sort dictionary by dict value
- Checking method execution order in Python class
- python pdfkit error wkhtmltopdf exited with non-zero code -6
- MyPy type hints for a function defined with named parameters but can take **kwargs?
- Occurrences of a pattern in a string and its indices
- What are the appropriate argument/return types for a function to take binary files/streams/filenames and convert them to readable text format?
- Making a Timer in Python 3
- Python: Invalid argument when renaming folder
- Why does this code to reverse a number give "inf" as output?
- Why timeit.timeit is giving error when using with variable?
- Oversampling multiclass data failing using ADASYN algorithm
- Find list of images(may be .jpg) using python without knowing the path.
- Separate layout and controller in kivy gui
- cx_Freeze ImportError: cannot import name
- How to edit opencv source code for stitching module in python?
- MySQL 'SHOW TABLES' Returns count instead of list (Python)
- map(float for Resampling DICOM images throwing Multivalue error
- extracting the region of interest as a separate image
- Blank line before the return statement in a Python function
- How to remove special characters from a string
- I need to convert a string to a list in python 3, how would I do this?
- "Programmatically" add stuff to a class?
- Disable mouse pointer in QGraphicsView
- Win 10: Cmd can bring windows to foreground, but any other terminal can't(even if elevated). Why is this so?
- Intersection with non unique items in Python
- Dynamic SQLAlchemy ORM relationship generation