Let's Get Pythonic -- Generators1848 words. Time to Read: About 18 minutes.
A couple posts ago, I wrote about using generators to efficiently create prime numbers. I think I promised then to go a little more in detail about them, which I am now doing here, thus proving that I am a dependable and trustworthy friend. A little caveat. Before I started writing this, I knew the basics, but I wanted to get a better grasp of the finer details. This article by Obi Ike-Nwosu is where I got most of my extra details. In fact, this article will mostly be for me, to make sure I have a grasp on everything, and will contain a significant amount of the information from his original article. I strongly recommend you check out The Intermediate Pythonista for the abovementioned article and apparently many others as well. For those of you committed to seeing me lay out a review of the same information in my own way, let us begin.
So. Generators. What are they? Generators are a subset of a larger group of objects called iterators, which I’ll explain in a minute. In short, generators (or, generator-iterators) are functions that, instead of simply returning a value at the end, will pause and save their state until the next time they are called to continue running. Lemme ‘splain. Have a gander at the following function:
A pretty common pattern, yes? Create a full list, and then do stuff with that list. Sum it, map it, print it, filter it, count it, graph it, etc. One thing that is maybe a bummer is if you wanted to not use the whole list for some reason. You are forced to create the whole thing and then use it. Another bummer is if you are unsure of your limit! What if you’re not sure how many powers it will take, you just want however many until
2**i > 10000000? That is where generators come in, with the
yield keyword. Check it.
In this newer version, the yield keyword is used. Let’s see it in action:
As you can see, you can grab each of the values as you need them – lazily – rather than producing the entire list first. But what is it?? Let’s see:
It’s a generator! But where are your items!? Do you have to call next a bazillion times? That’s exhausting! No, don’t worry. Most functions that take lists as inputs will also take these generator objects. Look again:
OK, back to the theory. Just to clarify, generators, or generator functions, create generator objects. These are disposable objects that you can call
next() on to get values. These generators are just quick and dirty ways to create generator objects though. You can actually create your own (which is what the function does behind the scenes). All you need to create an iterator object is an
__iter__ method. All that you need to be an iterator object is a
next method. Generally, it is efficient to do both within the same object. You’ll see.
A note of warning! When creating your own iterators, they do not pause execution in the middle of the next function like a
yield would. The important line in the above next function is that we update our current variable before we return anything. Returning kicks us back to the top with whatever state we had at the time, leaving the last line of the while loop ignored!
The other thing of note that happened here (and which you can include in your generator functions as well) is the
StopIteration exception. When you throw that from within an iterator, it signals the for loop to stop looping. If you come upon it by using the
next() function, it will actually throw the StopIteration exception.
Bonus 1: Generator Expressions
You know what are great? List comprehensions. Off topic, but a 15-second explanation.
Even greater is that you can create generators with a similar generator expression. Simply use () instead of  and your sequence will lazy load!
So what’s the difference?? Generator expressions are throwaway versions that can only be used once. They create items on the fly, however, so they use much less memory. You should use them if you only need them once or if your sequence is very large. If you need to loop over the sequence multiple times, stick to list comprehensions.
Bonus 2: Recursive Yielding
One more thing! Generators can delegate to each other! Here is a silly example, and then a useful example.
See? You can kick into a different generator to get more values. A more useful example from my previous post:
Recursive Generation!!!! I guess, if you’ve already read the previous post, you’re not that surprised, but it’s cool, ja? Anyways, there are a lot more things you can do with the new
yield from keywords. I suggest you check out the official documentation for ideas.
Whew! This was a long one! Like I said before, be sure to check out the Intermediate Pythonista’s take on generators for similar information presented slightly differently. Hopefully this article was helpful!Author: Ryan Palo | Tags: python tricks | Buy me a coffee