Progreso del estado de subprocesamiento de la cola

I'm trying to create a web extractor, I have this code for multithreads, and I need print the status/progress of the scanner :

import time
import threading
import Queue       
import sys

try:
    Lista = open(sys.argv[1], "r").readlines()
except(IOError): 
    print "Error: Check your ip list path\n"
    sys.exit(1)

class WorkerThread(threading.Thread) :

    def __init__(self, queue) :
        threading.Thread.__init__(self)
        self.queue = queue

    def run(self) :
        while True :
            counter = self.queue.get()
            sys.stdout.write("line nr : \r")
            self.queue.task_done()      

queue = Queue.Queue()

for i in range(50) :
    worker = WorkerThread(queue)
    worker.setDaemon(True)
    worker.start()

for line in Lista:
    queue.put(line)

queue.join()

print "All task over!"

How can I print the status/progress when the scanner working, I'm tried len(queue) but it doesn't work?

preguntado el 09 de septiembre de 13 a las 21:09

Can you explain a bit why len(queue) ¿no es correcto? -

@OfirIsrael: Queues don't have lengths, so you get a TypeError or AttributeError depending on your Python version, and presumably he doesn't consider that a correct way to get information for a progress bar… -

1 Respuestas

Queue los objetos no tienen len because, by their very nature, they're being shared across threads, and it would be inaccurate and misleading.

However, they have a qsize method that gives you the approximate size, for exactly this kind of purpose.


Si desea exacto values, you need a second Queue for that, where each task puts something on the out-queue, and some extra thread (or possibly the main thread) loops over it and counts up the tasks done so far. Or, alternatively, something simpler, like a global int counter with a global Lock protecting it.


However, I think it would be a lot simpler to write this in terms of a pool or executor. That would take care of queueing up tasks for you, and returning a value for each one to the main thread, without you needing to manage anything. For example, using futures, the backport of the Python 3.x concurrent.futures module for 2.x, here's your whole program, with progress added:

import sys
import futures 

try:
    Lista = open(sys.argv[1], "r").readlines()
except(IOError): 
    print "Error: Check your ip list path\n"
    sys.exit(1)

def task(line):
    # Do something

with futures.ThreadPoolExecutor(50) as executor:
    fs = [executor.submit(task, line) for line in Lista]
    for i, f in enumerate(futures.as_completed(fs)):
        sys.stdout.write("line nr: {} / {} \r".format(i, len(Lista)))

print "All task over!"

Respondido el 09 de Septiembre de 13 a las 21:09

No es la respuesta que estás buscando? Examinar otras preguntas etiquetadas or haz tu propia pregunta.