I'm using generators in list comprehensions, and getting some unexpected behavior with one of the generators ending early. Why does creating the generator outside of the list comprehension cause the behavior to change?

The generator I created is as follows:

def inc_range(a,b):

    for i in range(min(a,b), max(a,b) + 1):

        yield i

The first way of calling is as follows:

[(i,j) for i in inc_range(1,3) for j in inc_range(4,6)]

This gives me the following result:

[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]

The second way of calling it is as follows:

a = inc_range(1,3)

b = inc_range(4,6)



[(i,j) for i in a for j in b]

This gives me the following:

[(1, 4), (1, 5), (1, 6)]

Experimenting around, the following two examples gave me the first result:

a = range(1,4)

b = range(4,7)



[(i,j) for i in a for j in b]

a = (i for i in range(1,4))

b = (i for i in range(4,7))



a = list(a)

b = list(b)



[(i,j) for i in a for j in b]

While the following gave me the second result again.

a = (i for i in range(1,4))

b = (i for i in range(4,7))



[(i,j) for i in a for j in b]

What rule am I violating here regarding generators? Why does it make a difference when I assign the generators to variables before using them in a list comprehension, vs. using them directly?

ANSWERS

Check out the following answers which helped me understand what is occurring here:

Alex Yu
mkrieger1

And the answer is in:

PEP 289 -- Generator Expressions. The details

Only the outermost for-expression is evaluated immediately, the other expressions are deferred until the generator is run:

g = (tgtexp  for var1 in exp1 if exp2 for var2 in exp3 if exp4)

is equivalent to:

def __gen(bound_exp):

    for var1 in bound_exp:

        if exp2:

            for var2 in exp3:

                if exp4:

                    yield tgtexp

g = __gen(iter(exp1))

del __gen

Clarification. Why PEP-289 and not PEP-202?

Because comprehensions on lists works in another way.

This would be example for comprehensions on lists:

a = list(inc_range(1,3))

b = list(inc_range(4,6))



[(i,j) for i in a for j in b]

And there will be no "wonders": everything will be as expected.

A generator is an iterable object hence when you call it outside of a list it returns the next item alone.

a = inc_range(1,3)

b = inc_range(4,6)

c = inc_range(7,9)



[(i,j,k) for i in a for j in b for k in c]

This will only yield the elements for k in c when run

Hence you need to iterate through all its objects as you have when defining it as an array.

[(i,j,k) for i in inc_range(1,3) for j in inc_range(3,6) for k in inc_range(7,9)]

This forces the generator to yield all values in each of the iterations.

To get the desired result, the "inner" generator would have to be run as many times as the "outer" generator yields a value.

But, after the first run, the "inner" generator is exhausted and cannot be run again.

Adding a print illustrates this (simplifying the example):

>>> def inc(a, b):

...    for i in range(a, b):

...        print(i)

...        yield i

...

>>> a = inc(1, 4)

>>> b = inc(4, 7)

>>> [(i, j) for i in a for j in b]

1  # <-- a begins to run

4  # <-- b begins to run

5

6  # <-- b exhausted here

2  # <-- a continued, but not resulting in list item, because lacking value from b

3

[(1, 4), (1, 5), (1, 6)]

The reason why not storing the generators in variables works as expected is because a new "inner" generator is created for each iteration of the "outer" generator. Again, illustrated by some prints:

>>> def inc(a, b):

...    print('started', a, b)

...    for i in range(a, b):

...        yield i

... 

>>> [(i, j) for i in inc(1, 4) for j in inc(4, 7)]

started 1 4

started 4 7

started 4 7

started 4 7

[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]

The reason why using range objects or lists works as expected is because they can be iterated over arbitrarily many times without being exhausted.