Iterators & Iterables¶

All iterators are iterable, but not all iterables are iterators.
Iterators have state. They remember where they left off and can continue.
Lists and tuples are iterable but they are not iterators.
The iter() function can turn an iterable into an iterator.
Iterators have a __next__() method.
The builtin next() function will advance an iterator and return its ‘next’ value.
Iterators and Iterables | YouTube.com Corey Schafer

Itertools Module¶

count()
cycle()
combinations()
combinations_with_replacement()
permutation()
islice()
accumulate()
starmap()

Iterable vs Iterator¶

def is_iterable(obj):
    """ Returns True if the object is iterable. """
    return '__iter__' in dir(obj)


def is_iterator(obj):
    """ Returns True if the object is an iterator. """
    return '__next__' in dir(obj)

print("Iterable:", is_iterable(range(10)))
print("Iterator:", is_iterator(range(10)))

Iterable: True
Iterator: False

Turn an iterable into an iterator with the iter() function.

print("Iterable:", is_iterable(iter(range(10))))
print("Iterator:", is_iterator(iter(range(10))))

Iterable: True
Iterator: True

Above iter(range(start, stop, step)) is similar to itertools.count(start, step) below. Both can take a starting point and a step parameter and both are iterable. However the range function requires a stopping point and count does not. More importantly, count is an iterator, and range is not – unless we make it so.

Iterators Can Continue – this is the key to iterators!¶

Plain iterables can not do this.

import itertools

counter = itertools.count(1)
for _ in range(10):
    print(next(counter))

# The counter knows where it stopped, and can continue
# This is the special sause of being an iterator.
for _ in range(10):
    print(next(counter))

Cycle will continue rotating through an iterable indefinately. Since cycle is an iterator it will remember where it stops and can continue on from there at a later time.

it = itertools.cycle(("Alpha", "Beta", "Gamma"))

for _ in range(4):
    print(next(it))

Alpha
Beta
Gamma
Alpha

Notice below how the second for loop will begin with “Beta” since we last saw “Alpha” and “Beta” is next in the sequence.

for _ in range(4):
    print(next(it))

Beta
Gamma
Alpha
Beta

If we did this again, what would it start with?

my_iter_range = iter(range(10))

for itm in my_iter_range:
    print(itm)

for itm in my_iter_range:
    print(itm)

Nothing is printed because my_iter_range is exhausted!

Unpacking with Star¶

Like sequences, iterators can be unpacked with the *. Be careful not to unpack an infinite iterator! *itertools.count() would like to go on and on forever. If this happens use the control-c command, this will stop Python. If you let it continue, eventually your computer will run out of memory and crash. This will not hurt your computer, but it might be rather embarrassing. In some cases control-c won’t work, and you may need to force quit Python.

it = itertools.combinations((1,2,3,4), 2)
print(*it, sep='\n')

(1, 2)
(1, 3)
(1, 4)
(2, 3)
(2, 4)
(3, 4)

it = itertools.combinations_with_replacement((1,2,3,4), 2)
print(*it, sep='\n')

(1, 1)
(1, 2)
(1, 3)
(1, 4)
(2, 2)
(2, 3)
(2, 4)
(3, 3)
(3, 4)
(4, 4)

it = itertools.permutations((1,2,3,4), 2)
print(*it, sep='\n')

(1, 2)
(1, 3)
(1, 4)
(2, 1)
(2, 3)
(2, 4)
(3, 1)
(3, 2)
(3, 4)
(4, 1)
(4, 2)
(4, 3)

Slice An Iterator¶

itertools.islice() is used to slice an iterator. It’s pronounced “i-slice” not “is-lice” – if you’re curious.

start, stop, step = 50, 101, 5
it = itertools.islice(itertools.count(), start, stop, step)
print(*it, sep=', ')

50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100

This is an amazing ability and saves us from casting the iterator to a list so we can slice it. Casting would take more time and memory – possibly a lot more. It also pretty much defeats the purpose of using an iterator as you can’t call next on a list or continue where it left off. While debugging, however, it can be very useful to cast an iterator to a list just to see its guts.

Partial Sum¶

Partial Sum can be used to transform relative weights into cumulative weights.

import operator

# Relative
rel_weights = list(range(1, 11))
print(rel_weights)

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Cumulative
partial_sum = itertools.accumulate(rel_weights, operator.add)
cum_weights = list(partial_sum)
print(cum_weights)

[1, 3, 6, 10, 15, 21, 28, 36, 45, 55]

Adjacent Difference¶

Adjacent Difference can be used to transform cumulative weights back into relative weights.

# Back to Relative
adjacent_diff = itertools.starmap(
    operator.sub, zip(cum_weights, [0] + cum_weights))
rel_weights = list(adjacent_diff)
print(rel_weights)

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Groupby Example¶

party_of_casters = [
    {
        "name": "Gandolf",
        "class": "Wizard",
        "alignment": "Good",
    }, {
        "name": "Habius",
        "class": "Cleric",
        "alignment": "Evil",
    }, {
        "name": "Merlin",
        "class": "Wizard",
        "alignment": "Good",
    }, {
        "name": "Raven",
        "class": "Druid",
        "alignment": "Neutral",
    }, {
        "name": "Morgause",
        "class": "Sorceress",
        "alignment": "Evil",
    }, {
        "name": "Jinx",
        "class": "Warlock",
        "alignment": "Evil",
    }, {
        "name": "Zoaster",
        "class": "Cleric",
        "alignment": "Good",
    },
]

party_of_casters = sorted(
    party_of_casters, key=lambda p: (p['class'], p['name']))

class_groups = itertools.groupby(
    party_of_casters, lambda p: p['class'])

for class_name, players in class_groups:
    print(f"{class_name}:", end=' ')
    print(*(p['name'] for p in players), sep=', ')

Cleric: Habius, Zoaster
Druid: Raven
Sorceress: Morgause
Warlock: Jinx
Wizard: Gandolf, Merlin

party_of_casters = sorted(
    party_of_casters, key=lambda p: (p['alignment'], p['name']))

align_groups = itertools.groupby(
    party_of_casters, lambda p: p['alignment'])

for align, players in align_groups:
    print(f"{align}:", end=' ')
    print(*(p['name'] for p in players), sep=', ')

Evil: Habius, Jinx, Morgause
Good: Gandolf, Merlin, Zoaster
Neutral: Raven