Categories
closures lambda python

What do lambda function closures capture? [duplicate]

333

Recently I started playing around with Python and I came around something peculiar in the way closures work. Consider the following code:

adders=[None, None, None, None]

for i in [0,1,2,3]:
   adders[i]=lambda a: i+a

print adders[1](3)

It builds a simple array of functions that take a single input and return that input added by a number. The functions are constructed in for loop where the iterator i runs from 0 to 3. For each of these numbers a lambda function is created which captures i and adds it to the function’s input. The last line calls the second lambda function with 3 as a parameter. To my surprise the output was 6.

I expected a 4. My reasoning was: in Python everything is an object and thus every variable is essential a pointer to it. When creating the lambda closures for i, I expected it to store a pointer to the integer object currently pointed to by i. That means that when i assigned a new integer object it shouldn’t effect the previously created closures. Sadly, inspecting the adders array within a debugger shows that it does. All lambda functions refer to the last value of i, 3, which results in adders[1](3) returning 6.

Which make me wonder about the following:

  • What do the closures capture exactly?
  • What is the most elegant way to convince the lambda functions to capture the current value of i in a way that will not be affected when i changes its value?

15

  • 50

    I have had this problem in UI code. Drove me nuts. The trick is to remember that loops do not create new scope.

    – detly

    Jun 24, 2010 at 6:36

  • 3

    @TimMB How does i leave the namespace?

    – detly

    May 24, 2013 at 13:46

  • 3

    @detly Well I was going to say that print i wouldn’t work after the loop. But I tested it for myself and now I see what you mean – it does work. I had no idea that loop variables lingered after the loop body in python.

    – Tim MB

    May 24, 2013 at 19:03

  • 25

    This is in the official Python FAQ, under Why do lambdas defined in a loop with different values all return the same result?, with both an explanation and the usual workaround.

    – abarnert

    Nov 6, 2014 at 1:23

  • 2

    @abarnert: so in C++, lambdas with [&] are closures (albeit of limited lifetime) and lambdas with [=] are not closures? That choice of definition isn’t going to confuse anyone 😉 And In Python before nonlocal, assignment didn’t work for captured variables in nested functions, so were they closures then, or is read-only binding by reference to the lexical scope sufficient to be a closure? I wonder if trying to apply lambda calculus in this way to imperative languages is in point of fact a waste of effort, and better would be to invent new terminology…

    Jun 12, 2015 at 9:06


197

Your second question has been answered, but as for your first:

what does the closure capture exactly?

Scoping in Python is dynamic and lexical. A closure will always remember the name and scope of the variable, not the object it’s pointing to. Since all the functions in your example are created in the same scope and use the same variable name, they always refer to the same variable.

Regarding your other question of how to overcome this, there are two ways that come to mind:

  1. The most concise, but not strictly equivalent way is the one recommended by Adrien Plisson. Create a lambda with an extra argument, and set the extra argument’s default value to the object you want preserved.

  2. A little more verbose but less hacky would be to create a new scope each time you create the lambda:

     >>> adders = [0,1,2,3]
     >>> for i in [0,1,2,3]:
     ...     adders[i] = (lambda b: lambda a: b + a)(i)
     ...     
     >>> adders[1](3)
     4
     >>> adders[2](3)
     5
    

The scope here is created using a new function (a lambda, for brevity), which binds its argument, and passing the value you want to bind as the argument. In real code, though, you most likely will have an ordinary function instead of the lambda to create the new scope:

def createAdder(x):
    return lambda y: y + x
adders = [createAdder(i) for i in range(4)]

3

  • 3

    Python has static scoping, not dynamic scoping.. it’s just all variables are references, so when you set a variable to a new object, the variable itself (the reference) has the same location, but it points to something else. the same thing happens in Scheme if you set!. see here for what dynamic scope really is: voidspace.org.uk/python/articles/code_blocks.shtml .

    – Claudiu

    Jun 29, 2010 at 15:21

  • 8

    Option 2 resembles what functional languages would call a “Curried function.”

    Sep 20, 2011 at 2:15

  • Solution 2 is better. I prefer it over the default parameter. It is more logical and less dependent on the specific way Python is designed. A second lambda provides local variables that function like a closure. 👍 👍

    – Sohail Si

    May 22 at 19:51


197

Your second question has been answered, but as for your first:

what does the closure capture exactly?

Scoping in Python is dynamic and lexical. A closure will always remember the name and scope of the variable, not the object it’s pointing to. Since all the functions in your example are created in the same scope and use the same variable name, they always refer to the same variable.

Regarding your other question of how to overcome this, there are two ways that come to mind:

  1. The most concise, but not strictly equivalent way is the one recommended by Adrien Plisson. Create a lambda with an extra argument, and set the extra argument’s default value to the object you want preserved.

  2. A little more verbose but less hacky would be to create a new scope each time you create the lambda:

     >>> adders = [0,1,2,3]
     >>> for i in [0,1,2,3]:
     ...     adders[i] = (lambda b: lambda a: b + a)(i)
     ...     
     >>> adders[1](3)
     4
     >>> adders[2](3)
     5
    

The scope here is created using a new function (a lambda, for brevity), which binds its argument, and passing the value you want to bind as the argument. In real code, though, you most likely will have an ordinary function instead of the lambda to create the new scope:

def createAdder(x):
    return lambda y: y + x
adders = [createAdder(i) for i in range(4)]

3

  • 3

    Python has static scoping, not dynamic scoping.. it’s just all variables are references, so when you set a variable to a new object, the variable itself (the reference) has the same location, but it points to something else. the same thing happens in Scheme if you set!. see here for what dynamic scope really is: voidspace.org.uk/python/articles/code_blocks.shtml .

    – Claudiu

    Jun 29, 2010 at 15:21

  • 8

    Option 2 resembles what functional languages would call a “Curried function.”

    Sep 20, 2011 at 2:15

  • Solution 2 is better. I prefer it over the default parameter. It is more logical and less dependent on the specific way Python is designed. A second lambda provides local variables that function like a closure. 👍 👍

    – Sohail Si

    May 22 at 19:51


48

For completeness another answer to your second question: You could use partial in the functools module.

With importing add from operator as Chris Lutz proposed the example becomes:

from functools import partial
from operator import add   # add(a, b) -- Same as a + b.

adders = [0,1,2,3]
for i in [0,1,2,3]:
    # store callable object with first argument given as (current) i
    adders[i] = partial(add, i) 

print adders[1](3)

1

  • As the years pass I only become more convinced that this is the best way to solve the problem.

    Sep 27, 2021 at 21:14