Categories
kill multithreading python terminate

Is there any way to kill a Thread?

924

Is it possible to terminate a running thread without setting/checking any flags/semaphores/etc.?

0

    796

    It is generally a bad pattern to kill a thread abruptly, in Python, and in any language. Think of the following cases:

    • the thread is holding a critical resource that must be closed properly
    • the thread has created several other threads that must be killed as well.

    The nice way of handling this, if you can afford it (if you are managing your own threads), is to have an exit_request flag that each thread checks on a regular interval to see if it is time for it to exit.

    For example:

    import threading
    
    class StoppableThread(threading.Thread):
        """Thread class with a stop() method. The thread itself has to check
        regularly for the stopped() condition."""
    
        def __init__(self,  *args, **kwargs):
            super(StoppableThread, self).__init__(*args, **kwargs)
            self._stop_event = threading.Event()
    
        def stop(self):
            self._stop_event.set()
    
        def stopped(self):
            return self._stop_event.is_set()
    

    In this code, you should call stop() on the thread when you want it to exit, and wait for the thread to exit properly using join(). The thread should check the stop flag at regular intervals.

    There are cases, however, when you really need to kill a thread. An example is when you are wrapping an external library that is busy for long calls, and you want to interrupt it.

    The following code allows (with some restrictions) to raise an Exception in a Python thread:

    def _async_raise(tid, exctype):
        '''Raises an exception in the threads with id tid'''
        if not inspect.isclass(exctype):
            raise TypeError("Only types can be raised (not instances)")
        res = ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(tid),
                                                         ctypes.py_object(exctype))
        if res == 0:
            raise ValueError("invalid thread id")
        elif res != 1:
            # "if it returns a number greater than one, you're in trouble,
            # and you should call it again with exc=NULL to revert the effect"
            ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(tid), None)
            raise SystemError("PyThreadState_SetAsyncExc failed")
    
    class ThreadWithExc(threading.Thread):
        '''A thread class that supports raising an exception in the thread from
           another thread.
        '''
        def _get_my_tid(self):
            """determines this (self's) thread id
    
            CAREFUL: this function is executed in the context of the caller
            thread, to get the identity of the thread represented by this
            instance.
            """
            if not self.isAlive():
                raise threading.ThreadError("the thread is not active")
    
            # do we have it cached?
            if hasattr(self, "_thread_id"):
                return self._thread_id
    
            # no, look for it in the _active dict
            for tid, tobj in threading._active.items():
                if tobj is self:
                    self._thread_id = tid
                    return tid
    
            # TODO: in python 2.6, there's a simpler way to do: self.ident
    
            raise AssertionError("could not determine the thread's id")
    
        def raiseExc(self, exctype):
            """Raises the given exception type in the context of this thread.
    
            If the thread is busy in a system call (time.sleep(),
            socket.accept(), ...), the exception is simply ignored.
    
            If you are sure that your exception should terminate the thread,
            one way to ensure that it works is:
    
                t = ThreadWithExc( ... )
                ...
                t.raiseExc( SomeException )
                while t.isAlive():
                    time.sleep( 0.1 )
                    t.raiseExc( SomeException )
    
            If the exception is to be caught by the thread, you need a way to
            check that your thread has caught it.
    
            CAREFUL: this function is executed in the context of the
            caller thread, to raise an exception in the context of the
            thread represented by this instance.
            """
            _async_raise( self._get_my_tid(), exctype )
    

    (Based on Killable Threads by Tomer Filiba. The quote about the return value of PyThreadState_SetAsyncExc appears to be from an old version of Python.)

    As noted in the documentation, this is not a magic bullet because if the thread is busy outside the Python interpreter, it will not catch the interruption.

    A good usage pattern of this code is to have the thread catch a specific exception and perform the cleanup. That way, you can interrupt a task and still have proper cleanup.

    33

    • 120

      @Bluebird75: Furthermore, I’m not sure I get the argument that threads should not be killed abruptly “because the thread might be holding a critical resource that must be closed properly”: this is also true from a main program, and main programs can be killed abruptly by the user (Ctrl-C in Unix, for instance)–in which case they try to handle this possibility as nicely as possible. So, I fail to see what is special with threads, and why they should not receive the same treatment as main programs (namely that they can be killed abruptly). 🙂 Could you elaborate on this?

      Jul 14, 2011 at 16:53


    • 21

      @EOL: On the other hand, if all the resources that the thread is owning are local resources (open files, sockets), Linux is reasonably good at the process cleanup and this does not leak. I had cases though where I created a server using socket, and if I do a brutal interruption with Ctrl-C, I can non longer launch the program because it can not binds the socket. I need to wait 5 minutes. Proper solution was to catch Ctrl-C and do clean socket deconnection.

      Aug 2, 2011 at 9:45


    • 12

      @Bluebird75: btw. you can use SO_REUSEADDR socket option to avoid Address already in use error.

      – Messa

      Sep 11, 2011 at 9:09


    • 13

      Note about this answer: at least for me (py2.6), I had to pass None instead of 0 for the res != 1 case, and I had to call ctypes.c_long(tid) and pass that to any ctypes function rather than the tid directly.

      – Walt W

      Nov 17, 2011 at 21:16

    • 23

      Its worth mentioning that _stop is already occupied in the Python 3 threading library. As such, maybe use a different variable otherwise you will get an error.

      Jan 18, 2013 at 1:51

    152

    A multiprocessing.Process can p.terminate()

    In the cases where I want to kill a thread, but do not want to use flags/locks/signals/semaphores/events/whatever, I promote the threads to full blown processes. For code that makes use of just a few threads the overhead is not that bad.

    E.g. this comes in handy to easily terminate helper “threads” which execute blocking I/O

    The conversion is trivial: In related code replace all threading.Thread with multiprocessing.Process and all queue.Queue with multiprocessing.Queue and add the required calls of p.terminate() to your parent process which wants to kill its child p

    See the Python documentation for multiprocessing.

    Example:

    import multiprocessing
    proc = multiprocessing.Process(target=your_proc_function, args=())
    proc.start()
    # Terminate the process
    proc.terminate()  # sends a SIGTERM
    

    5

    • Thanks. I replaced queue.Queue with multiprocessing.JoinableQueue and followed this answer: stackoverflow.com/a/11984760/911207

      Aug 15, 2014 at 17:30

    • 20

      multiprocessing is nice, but be aware that arguments is pickled to the new process. So if one of the arguments is something not-pickable (like a logging.log) it might not be a good idea to use multiprocessing.

      – Lyager

      Apr 9, 2018 at 9:10

    • 3

      multiprocessing arguments are pickled to the new process on Windows, but Linux uses forking to copy them (Python 3.7, unsure what other versions). So you’ll end up with code that works on Linux but raises pickle errors on Windows.

      Apr 18, 2019 at 22:21


    • multiprocessing with logging is tricky business. Need to use QueueHandler (see this tutorial). I learned it the hard way.

      Dec 9, 2019 at 0:10

    • Shame I can’t spy on functions ran in multiprocessing… Thank you

      – Adonis

      Mar 30 at 14:56

    136

    There is no official API to do that, no.

    You need to use platform API to kill the thread, e.g. pthread_kill, or TerminateThread. You can access such API e.g. through pythonwin, or through ctypes.

    Notice that this is inherently unsafe. It will likely lead to uncollectable garbage (from local variables of the stack frames that become garbage), and may lead to deadlocks, if the thread being killed has the GIL at the point when it is killed.

    1

    • 47

      It will lead to deadlocks if the thread in question holds the GIL.

      Sep 20, 2015 at 15:09