Twisted Tutorials – VII

This article is going to deal with another aspect of Deferreds, which normally is a bit non-trivial to realise,so we are going to present a complex situation, which will demonstrate the problem, and how deferred solves the problem.

To motivate our discussion we’re going to add a hypothetical feature to our poetry client. Suppose some hard-working Computer Science professor has invented a new poetry-related algorithm, the Byronification Engine. This nifty algorithm takes a single poem as input and produces a new poem like the original, but written in the style of Lord Byron. What’s more, our professor has kindly provided a reference implementation in Python, with this interface:

class IByronificationEngine(Interface):

    def byronificate(poem):
        """
        Return a new poem like the original, but in the style of Lord Byron.

        Raises GibberishError if the input is not a genuine poem.
        """

Like most bleeding-edge software, the implementation has some bugs. This means that in addition to the documented exception, the byronificate method sometimes throws random exceptions when it hits a corner-case the professor forgot to handle.

We’ll also assume the engine runs fast enough that we can just call it in the main thread without worrying about tying up the reactor. This is how we want our program to work:

1. Try to download the poem.
2. If the download fails, tell the user we couldn’t get the poem.
3. If we do get the poem, transform it with the Byronification Engine.
4. If the engine throws a GibberishError, tell the user we couldn’t get the poem.
5. If the engine throws another exception, just keep the original poem.
6. If we have a poem, print it out.
7. End the program.

The idea here is that a GibberishError means we didn’t get an actual poem after all, so we’ll just tell the user the download failed. That’s not so useful for debugging, but our users just want to know whether we got a poem or not. On the other hand, if the engine fails for some other reason then we’ll use the poem we got from the server. After all, some poetry is better than none at all, even if it’s not in the trademark Byron style.

Here’s the synchronous version of our code:

try:
    poem = get_poetry(host, port) # synchronous get_poetry
except:
    print >>sys.stderr, 'The poem download failed.'
else:
    try:
        poem = engine.byronificate(poem)
    except GibberishError:
        print >>sys.stderr, 'The poem download failed.'
    except:
        print poem # handle other exceptions by using the original poem
    else:
        print poem

sys.exit()

This sketch of a program could be make simpler with some refactoring, but it illustrates the flow of logic pretty clearly. We want to update our most recent poetry client (which uses deferreds) to implement this same scheme. But we won’t do that until Part 10. For now, instead, let’s imagine how we might do this with client 3.1, our last client that didn’t use deferreds at all. Suppose we didn’t bother handling exceptions, but instead just changed the got_poem callback like this:

def got_poem(poem):
    poems.append(byron_engine.byronificate(poem))
    poem_done()

What happens when the byronificate method raises a GibberishError or some other exception? Looking at Figure 11 from Part 6, we can see that:

1. The exception will propagate to the poem_finishedcallback in the factory, the method that actually invokes the callback.
2. Since poem_finished doesn’t catch the exception, it will proceed to poemReceived on the protocol.
3. And then on to connectionLost, also on the protocol.
4. And then up into the core of Twisted itself, finally ending up at the reactor.

As we have learned, the reactor will catch and log the exception instead of crashing. But what it certainly won’t do is tell the user we couldn’t download a poem. The reactor doesn’t know anything about poems or GibberishErrors, it’s a general-purpose piece of code used for all kinds of networking, even non-poetry-related networking.

I think the concept is rather simple, and dealing with it just requires plain reading. So I won’t be going after copying the content. You can refer to the website for rest of the stuff (https://krondo.com/a-second-interlude-deferred/)

Twisted Tutorials – VI

This article is the sixth series of the Twisted’s Tutorial. The whole text has been taken from the website(https://krondo.com), and this has been the writer’s interpretation of whatever that he has understood.

We will discuss about the errbacks, the potential problems with factoring errbacks manually and how Twisted provides us with a solution.

Callbacks are a fundamental aspect of asynchronous programming with Twisted. Rather than just a way of interfacing with the reactor, callbacks will be woven into the structure of any Twisted program we write. So using Twisted, or any reactor-based asynchronous system, means organizing our code in a particular way, as a series of “callback chains” invoked by a reactor loop.

Even an API as simple as our get_poetry function required callbacks, two of them in fact: one for normal results and one for errors. Since, as Twisted programmers, we’re going to have to make so much use of them, we should spend a little bit of time thinking about the best ways to use callbacks, and what sort of pitfalls we might encounter.

The basic plan here is clear:

1. If we get the poem, print it out.
2. If we don’t get the poem, print out an Error Haiku.
3. In either case, end the program.

The ‘synchronous analogue’ to the above code might look something like this:

...
try:
    poem = get_poetry(host, port) # the synchronous version of get_poetry
except Exception, err:
    print >>sys.stderr, 'poem download failed'
    print >>sys.stderr, 'I am terribly sorry'
    print >>sys.stderr, 'try again later?'
    sys.exit()
else:
    print poem
    sys.exit()

So the callback is like the else block and the errback is like the except. That means invoking the errback is the asynchronous analogue to raising an exception and invoking the callback corresponds to the normal program flow.

What are some of the differences between the two versions? For one thing, in the synchronous version the Python interpreter will ensure that, as long as get_poetryraises any kind of exception at all, for any reason, the except block will run. If we trust the interpreter to run Python code correctly we can trust that error block to run at the right time.

Contrast that with the asynchronous version: the poem_failed errback is invoked by our code, the clientConnectionFailed method of the PoetryClientFactory. We, not Python, are in charge of making sure the error code runs if something goes wrong. So we have to make sure to handle every possible error case by invoking the errback with a Failure object. Otherwise, our program will become “stuck” waiting for a callback that never comes.

That shows another difference between the synchronous and asynchronous versions. If we didn’t bother catching the exception in the synchronous version (by not using a try/except), the Python interpreter would “catch” it for us and crash to show us the error of our ways. But if we forget to “raise” our asynchronous exception (by calling the errback function in PoetryClientFactory), our program will just run forever, blissfully unaware that anything is amiss.

Clearly, handling errors in an asynchronous program is important, and also somewhat tricky. You might say that handling errors in asynchronous code is actually more important than handling the normal case, as things can go wrong in far more ways than they can go right. Forgetting to handle the error case is a common mistake when programming with Twisted.

Here’s another fact about the synchronous code above: either the else block runs exactly once, or the exceptblock runs exactly once (assuming the synchronous version of get_poetry doesn’t enter an infinite loop). The Python interpreter won’t suddenly decide to run them both or, on a whim, run the else block twenty-seven times. And it would be basically impossible to program in Python if it did!

But again, in the asynchronous case we are in charge of running the callback or the errback. Knowing us, we might make some mistakes. We could call both the callback and the errback, or invoke the callback twenty-seven times. That would be unfortunate for the users of get_poetry. Although the docstring doesn’t explicitly say so, it really goes without saying that, like the else and except blocks in a try/except statement, either the callback will run exactly once or the errback will run exactly once, for each specific call to get_poetry. Either we get the poem or we don’t.

Imagine trying to debug a program that makes three poetry requests and gets seven callback invocations and two errback invocations. Where would you even start? You’d probably end up writing your callbacks and errbacks to detect when they got invoked a second time for the same get_poetry call and throw an exception right back. Take that, get_poetry.

One more observation: both versions have some duplicate code. The asynchronous version has two calls to reactor.stop and the synchronous version has two calls to sys.exit. We might refactor the synchronous version like this:

...
try:
    poem = get_poetry(host, port) # the synchronous version of get_poetry
except Exception, err:
    print >>sys.stderr, 'poem download failed'
    print >>sys.stderr, 'I am terribly sorry'
    print >>sys.stderr, 'try again later?'
else:
    print poem

sys.exit()

Can we refactor the asynchronous version in a similar way? It’s not really clear that we can, since the callback and errback are two different functions. Do we have to go back to a single callback to make this possible?

1. Calling errbacks is very important. Since errbacks take the place of except blocks, users need to be able to count on them. They aren’t an optional feature of our APIs.
2. Not invoking callbacks at the wrong time is just as important as calling them at the right time. For a typical use case, the callback and errback are mutually exclusive and invoked exactly once.
3. Refactoring common code might be harder when using callbacks.

Twisted provides an abstraction called Deferred, which helps us in addressing the pitfalls of the problem. A deferred contains a pair of callback chains, one for normal results and one for errors. A newly-created deferred has two empty chains. We can populate the chains by adding callbacks and errbacks and then fire the deferred with either a normal result (here’s your poem!) or an exception (I couldn’t get the poem, and here’s why). Firing the deferred will invoke the appropriate callbacks or errbacks in the order they were added.

Summary

In this Part we analyzed callback programming and identified some potential problems. We also saw how the Deferred class can help us out:

1. We can’t ignore errbacks, they are required for any asynchronous API. Deferreds have support for errbacks built in.
2. Invoking callbacks multiple times will likely result in subtle, hard-to-debug problems. Deferreds can only be fired once, making them similar to the familiar semantics of try/except statements.
3. Programming with plain callbacks can make refactoring tricky. With deferreds, we can refactor by adding links to the chain and moving code from one link to another.

Twisted Tutorials – V

In this article, we are going to deal with making our poetry code much more usable, API based, so that it can be reused anywhere.

We need a way to send a poem to the code that requested the poem in the first place. In a synchronous program we might make an API like this:

def get_poetry(host, post):
    """Return a poem from the poetry server at the given host and port.""

But of course, we can’t do that here. The above function necessarily blocks until the poem is received in entirety, otherwise it couldn’t work the way the documentation claims. But this is a reactive program so blocking on a network socket is out of the question. We need a way to tell the calling code when the poem is ready, without blocking while the poem is in transit. But this is the same sort of problem that Twisted itself has. Twisted needs to tell our code when a socket is ready for I/O, or when some data has been received, or when a timeout has occurred, etc. We’ve seen that Twisted solves this problem using callbacks, so we can use callbacks too:

def get_poetry(host, port, callback):
    """
    Download a poem from the given host and port and invoke

      callback(poem)

    when the poem is complete.
    """

Now we have an asynchronous API we can use with Twisted, so let’s go ahead and implement it.

Now I am not going to copy paste the code, that I was doing for earlier articles. If you want to see the progress, please refer (https://krondo.com/and-then-we-took-it-higher/).

The main point of discussion here is that, unlike synchronous programming, where we can deal with errors and exceptions, going the same way in asynchronous way won’t help us any time. So we will go about the error handling using callbacks( also known as errbacks). These help us catch the error, and do the specific handling.

However, when we do the above, we are not able to deal with different types of errors that are possible. So Twisted provides us with Failure object, which help us determine the type of errors formed.

Summary –

• The APIs we write for Twisted programs will have to be asynchronous.
• We can’t mix synchronous code with asynchronous code.
• Thus, we have to use callbacks in our own code, just like Twisted does.
• And we have to handle errors with callbacks, too.

Does that mean every API we write with Twisted has to include two extra arguments, a callback and an errback? That doesn’t sound so nice. Fortunately, Twisted has an abstraction we can use to eliminate both those arguments and pick up a few extra features in the bargain.

Twisted Tutorials – IV

In this article we will deal with the abstractions that Twisted provides, and eases our own work.

First of all, the client includes code for mundane details like creating network sockets and receiving data from those sockets. Twisted provides support for these sorts of things so we don’t have to implement them ourselves every time we write a new program. This is especially helpful because asynchronous I/O requires a few tricky bits involving exception handling as you can see in the client code.

Another problem with the current client is error handling. Try running version 1.0 of the Twisted client and tell it to download from a port with no server. It just crashes. We could fix the current client, but error handling is easier with the Twisted APIs we’ll be using today.

We’re going to fix the first and second problems using a higher-level set of APIs and Interfaces. The Twisted framework is loosely composed of layers of abstractions and learning Twisted means learning what those layers provide, i.e, what APIs, Interfaces, and implementations are available for use in each one. Since this is an introduction we’re not going to study each abstraction in complete detail or do an exhaustive survey of every abstraction that Twisted offers. We’re just going to look at the most important pieces to get a better feel for how Twisted is put together. Once you become familiar with the overall style of Twisted’s architecture, learning new parts on your own will be much easier.

In general, each Twisted abstraction is concerned with one particular concept. For example, the 1.0 client from Part 4 uses IReadDescriptor, the abstraction of a “file descriptor you can read bytes from”. A Twisted abstraction is usually defined by an Interface specifying how an object embodying that abstraction should behave. The most important thing to keep in mind when learning a new Twisted abstraction is this:

Most higher-level abstractions in Twisted are built by using lower-level ones, not by replacing them.

So when you are learning a new Twisted abstraction, keep in mind both what it does and what it does not do. In particular, if some earlier abstraction A implements feature F, then F is probably not implemented by any other abstraction. Rather, if another abstraction B needs feature F, it will use A rather than implement F itself.  (In general, an implementation of B will either sub-class an implementation of A or refer to another object that implements A).

When you choose to use Twisted you are also choosing to use the Reactor Pattern, and that means programming in the “reactive style” using callbacks and cooperative multi-tasking. If you want to use Twisted correctly, you have to keep the reactor’s existence (and the way it works) in mind.

Before we dive into the code, there are three new abstractions to introduce: Transports, Protocols, and Protocol Factories.

TRANSPORTS

The Transport abstraction is defined by ITransport in the main Twisted interfaces module. A Twisted Transport represents a single connection that can send and/or receive bytes. For our poetry clients, the Transports are abstracting TCP connections like the ones we have been making ourselves in earlier versions.

If you scan the methods defined for ITransport, you won’t find any for receiving data. That’s because Transports always handle the low-level details of reading data asynchronously from their connections, and give the data to us via callbacks. Along similar lines, the write-related methods of Transport objects may choose not to write the data immediately to avoid blocking. Telling a Transport to write some data means “send this data as soon as you can do so,  subject to the requirement to avoid blocking”. The data will be written in the order we provide it, of course.

We generally don’t implement our own Transport objects or create them in our code. Rather, we use the implementations that Twisted already provides and which are created for us when we tell the reactor to make a connection.

PROTOCOLS

Twisted Protocols are defined by IProtocol in the same interfaces module. As you might expect, Protocol objects implement protocols. That is to say, a particular implementation of a Twisted Protocol should implement one specific networking protocol, like FTP or IMAP or some nameless protocol we invent for our own purposes. Our poetry protocol, such as it is, simply sends all the bytes of the poem as soon as a connection is established, while the close of the connection signifies the end of the poem.

Strictly speaking, each instance of a Twisted Protocol object implements a protocol for one specific connection. So each connection our program makes (or, in the case of servers, accepts) will require one instance of a Protocol. This makes Protocol instances the natural place to store both the state of “stateful” protocols and the accumulated data of partially received messages (since we receive the bytes in arbitrary-sized chunks with asynchronous I/O).

So how do Protocol instances know what connection they are responsible for? If you look at the IProtocoldefinition, you will find a method called makeConnection. This method is a callback and Twisted code calls it with a Transport instance as the only argument. The Transport is the connection the Protocol is going to use.

PROTOCOL FACTORIES

So each connection needs its own Protocol and that Protocol might be an instance of a class we implement ourselves. Since we will let Twisted handle creating the connections, Twisted needs a way to make the appropriate Protocol “on demand” whenever a new connection is made. Making Protocol instances is the job of Protocol Factories.

As you’ve probably guessed, the Protocol Factory API is defined by IProtocolFactory, also in the interfacesmodule. Protocol Factories are an example of the Factorydesign pattern and they work in a straightforward way. The buildProtocol method is supposed to return a new Protocol instance each time it is called. This is the method that Twisted uses to make a new Protocol for each new connection.

Protocol is the set of rules that should be followed when a connection is connected.

For each connections, one set of protocol is generated. By set of each connections, we mean that asynchronously we can have different connections. So after a  connection, we should follow a protocol. Protocol creation is handled by Twisted, using Protocol Factory.

Enough of the theoretical explanations, let us discuss about the code base that we are discussing.

Get Poetry 2.0

In order to start the proceedings using Twisted’s high level abstraction, we first create a Protocol Factory. After that, we create a transport layer, in which we pass the host, port and the protocol factory.

Here, the protocol factory that we create is subclassing the Twisted’s protocol factory. So there is an interface buildProtocol

def buildProtocol(self, address):
    proto = ClientFactory.buildProtocol(self, address)
    proto.task_num = self.task_num
    self.task_num += 1
    return proto

How does the base class know what Protocol to build? Notice we are also setting the class attribute protocol on PoetryClientFactory:

class PoetryClientFactory(ClientFactory):

    task_num = 1

    protocol = PoetryProtocol # tell base class what proto to build

The base Factory class implements buildProtocol by instantiating the class we set on protocol (i.e., PoetryProtocol) and setting the factory attribute on that new instance to be a reference to its “parent” Factory.

As we mentioned above, the factory attribute on Protocol objects allows Protocols created with the same Factory to share state.

The second stage of Protocol construction connects a Protocol with a Transport, using the makeConnectionmethod. We don’t have to implement this method ourselves since the Twisted base class provides a default implementation.

Once initialized in this way, the Protocol can start performing its real job — translating a lower-level stream of data into a higher-level stream of protocol messages (and vice-versa for 2-way connections). The key method for processing incoming data is dataReceived, which our client implements like this:

def dataReceived(self, data):
    self.poem += data
    msg = 'Task %d: got %d bytes of poetry from %s'
    print  msg % (self.task_num, len(data), self.transport.getPeer())

Each time dataReceived is called we get a new sequence of bytes (data) in the form of a string. As always with asynchronous I/O, we don’t know how much data we are going to get so we have to buffer it until we receive a complete protocol message. In our case, the poem isn’t finished until the connection is closed, so we just keep adding the bytes to our .poem attribute.

Note we are using the getPeer method on our Transport to identify which server the data is coming from. We are only doing this to be consistent with earlier clients. Otherwise our code wouldn’t need to use the Transport explicitly at all, since we never send any data to the servers.

Once a poem has finished downloading, the PoetryProtocol object notifies its PoetryClientFactory:

def connectionLost(self, reason):
    self.poemReceived(self.poem)

def poemReceived(self, poem):
    self.factory.poem_finished(self.task_num, poem)

The connectionLost callback is invoked when the transport’s connection is closed. The reason argument is a twisted.python.failure.Failure object with additional information on whether the connection was closed cleanly or due to an error. Our client just ignores this value and assumes we received the entire poem.

The factory shuts down the reactor after all the poems are done. Once again we assume the only thing our program is doing is downloading poems, which makes PoetryClientFactory objects less reusable. We’ll fix that in the next Part, but notice how the poem_finished callback keeps track of the number of poems left to go:

...
    self.poetry_count -= 1

    if self.poetry_count == 0:
        ...

Twisted Tutorials – III

This article deals with the Twisted’s different parts. In this article, we will discuss the client capabilities of Twisted. Before I proceed, I would like to state that this portion( taken from http://www.krondo.com/twisted-poetry/) is one of the most confusing portions I read across. So whatever I might blog at the point of blogging, may be factually incorrect. So I am just blogging my interpretation of whatever I understood. Enough of the context, let us begin –

Our First Twisted Client

Although Twisted is probably more often used to write servers, clients are simpler than servers and we’re starting out as simply as possible.

Let’s take a look at the source code to see how it works. Open up the client in your editor so you can examine the code we are discussing.

Note: As I mentioned in Part 1, we will begin our use of Twisted by using some very low-level APIs. By doing this we bypass some of the layers of Twisted’s abstractions so we can learn Twisted from the “inside out”. But this means a lot of the APIs we will learn in the beginning are not often used when writing real code. Just keep in mind that these early programs are learning exercises, not examples of how to write production software.

The Twisted client starts up by creating a set of PoetrySocket objects. A PoetrySocket initializes itself by creating a real network socket, connecting to a server, and switching to non-blocking mode:

self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect(address)
self.sock.setblocking(0)

Eventually we’ll get to a level of abstraction where we aren’t working with sockets at all, but for now we still need to. After creating the network connection, a PoetrySocket passes itself to the reactor via the addReadermethod:

# tell the Twisted reactor to monitor this socket for reading
from twisted.internet import reactor
reactor.addReader(self)

This method gives Twisted a file descriptor you want to monitor for incoming data. Why are we passing Twisted an object instead of a file descriptor and a callback? And how will Twisted know what to do with our object since Twisted certainly doesn’t contain any poetry-specific code? Trust me, I’ve looked. Open up the twisted.internet.interfaces module and follow along with me.

Uptil this point, you will notice that this portion deals with the part that how the Twisted knows that addReader method knows any specificity about the current code we are using( in this case that we are using Poetry code). In order to answer this, follow along –

The Twisted provides us with interfaces, which helps us to name all the functionality under a same interface. Suppose, we have an interface, engine(). Now three classes , namely car(), bike() and truck() are present. Naturally, all the three of them have engine method among them. Note that engine as an individual entity is unique to each of the bike, car and truck. However, for simplicity and hiding the internal intricacies, we must agree that engine is common to all of them. Thus engine() should be an interface class, partly because that we want each of the units(cars, bikes and trucks) have there own version of engine, and partly because all the units have engine in common. Thus interface helps in addressing the above issue. So Interface helps us in achieving common nomenclature and code structure, and having its own functionality.

Interfaces are implemented using parent interfaces.

Using the above explanation, we will go through the code base –

Skip down the twisted.internet.interfaces source code until you come to the definition of the addReader method. It is declared in the IReactorFDSet Interface and should look something like this:

def addReader(reader):
    """
    I add reader to the set of file descriptors to get read events for.

    @param reader: An L{IReadDescriptor} provider that will be checked for
                   read events until it is removed from the reactor with
                   L{removeReader}.

    @return: C{None}.
    """

IReactorFDSet is one of the Interfaces that Twisted reactors provide. Thus, any Twisted reactor has a method called addReader that works as described by the docstring above. The method declaration does not have a selfargument because it is solely concerned with defining a public interface, and the self argument is part of the implementation (i.e., the caller does not have to pass selfexplicitly). Interface objects are never instantiated or used as base classes for real implementations.

According to the docstring above, the reader argument of addReader should implement the IReadDescriptorinterface. And that means our PoetrySocket objects have to do just that.

Scrolling through the module to find this new interface, we see:

class IReadDescriptor(IFileDescriptor):

    def doRead():
        """
        Some data is available for reading on your descriptor.
        """

And you will find an implementation of doRead on our PoetrySocket class. It reads data from the socket asynchronously, whenever it is called by the Twisted reactor. So doRead is really a callback, but instead of passing it directly to Twisted, we pass in an object with a doRead method. This is a common idiom in the Twisted framework — instead of passing a function you pass an object that must provide a given Interface. This allows us to pass a set of related callbacks (the methods defined by the Interface) with a single argument. It also lets the callbacks communicate with each other through shared state stored on the object.

So what other callbacks are provided on PoetrySocketobjects? Notice that IReadDescriptor is a sub-class of IFileDescriptor. That means any object that provides IReadDescriptor must also provide IFileDescriptor. And if you do some more scrolling, you will find:

class IFileDescriptor(ILoggingContext):
    """
    A file descriptor.
    """

    def fileno():
        ...

    def connectionLost(reason):
        ...

I left out the docstrings above, but the purpose of these callbacks is fairly clear from the names: fileno should return the file descriptor we want to monitor, and connectionLost is called when the connection is closed. And you can see our PoetrySocket objects provide those methods as well.

Finally, IFileDescriptor inherits from ILoggingContext. I won’t bother to show it here, but that’s why we need to include the logPrefix callback. You can find the details in the interfaces module.

Uptil now, if you couldn’t understand the above, I will start explaining them. However, the issue with me lies here: even I also have not been able understand it completely. However, whatever I can understand, I am going to explain –

Firstly, our original question was that we normally pass the reactor, the file description and callback functions. However, we simply passed the PoetrySocket object. Now how does that work?

Firstly, instead of watching how the code moves through to call the callback, let’s just watch the Interface. The IReactorFDSet defines reader as a parameter or argument of addReader. Now reader argument implements the IReadDescriptor interface. So we go through the IReadDescriptor interface. The IReadDescriptor has a method doRead(). So effectively, when going through the documentation, it is revealed that doRead() does not implements any parent class. Note that doRead() is the function which is the callback. Thus instead of calling the doRead() function directly, we are using addReader(), function going through the series of interfaces.

In order to summarise, addReader()–> reader –> doRead().

Using the way of interfaces is beneficial as it helps us to reuse the module, and use all the functionality of Twisted.

Similar to the above pattern, you will observe that the different methods used in the class is named using the Interfaces.(e.g. fileno, logprefix)

 

Twisted Tutorials – II

This article deals with the second part of Twisted. We will be discussing some portions of Twisted, and draw an analogy as to how they help us in implementing asynchronous programming.

Before we dive into the parts of Twisted, let us discuss one of the most paramount concepts of asynchronous programming; event-handling. In asynchronous programming, the whole task is divided into small events, and event handler associated with it. Now what is an event and event handler?

Event is that part when a part of a task is executed. A task can be divided into small portions, such that each part is mutually exclusive from each other. This means, that each task is independent of the execution of another one. Thus each of these tasks can be termed as events. Now comes the event-handling part. As each of the events are executed, we would want to combine them into equivalent single task. That is where event-handling comes into play. Event – handlers are those which handle events and execute the required steps.

Now in Twisted, the whole asynchronous implementation revolves around the concept of reactor. Now what is a reactor? A Reactor is continuous loop of event and event handling. A Reactor can be imagined of having an event generator(not to be confused with the python keyword generator) and event handler, except that the loop, once started, runs forever, and waits( or technically expects an event) and correspondingly executes the required event handler. So you can imagine that reactor is continuous infinite loop that hears for an event, and correspondingly handles the event. Reactor is basically the heart of the Twisted framework.

Now the best part of twisted is that it can handle numerous events, it’s corresponding event handlers, all asynchronously . This is the beauty of Twisted, dealing with asynchronous programming.

Enough of the events, lets deal with the parts of Twisted.

We have discussed about Reactor loop, so here are the characteristics –

1. Twisted’s reactor loop doesn’t start until told to. You start it by calling reactor.run().
2. The reactor loop runs in the same thread it was started in. In this case, it runs in the main (and only) thread.
3. Once the loop starts up, it just keeps going. The reactor is now “in control” of the program (or the specific thread it was started in).
4. If it doesn’t have anything to do, the reactor loop does not consume CPU.
5. The reactor isn’t created explicitly, just imported.

Now the last part is important, as it highlights the important property of Reactor. It is a Singleton, which means there is only one reactor object, and others are implementing by subclassing it.

Twisted actually contains multiple reactor implementations.

To use a specific reactor, you must install it beforeimporting twisted.internet.reactor. Here is how you install the pollreactor:

from twisted.internet import pollreactor

pollreactor.install()

If you import twisted.internet.reactor without first installing a specific reactor implementation, then Twisted will install the default reactor for you. The particular one you get will depend on the operating system and Twisted version you are using. For that reason, it is general practice not to import the reactor at the top level of modules to avoid accidentally installing the default reactor. Instead, import the reactor in the same scope in which you use it.

Here, I will demonstrate my own understanding of the code. So in order to refer the code please use https://krondo.com/our-eye-beams-begin-to-twist/

Hello Twisted

Let us do something, that gets us started with twisted. First of all, we create a function def hello(). This function prints that hello, and we have started our twisted programming. We first import reactor module from twisted.internet and call the function reactor.callWhenRunning(hello). As the name suggests, we are calling the method to execute hello function, when the reactor is running. The parameter contains the function to be called. Then we start the reactor using reactor.run()

After executing the code, the program prints the appropriate message, and keeps running. The program runs forever, as we have not asked it stop.

Now the most important concept that we should discuss, is the callback.We use the term callback to describe the reference to the hello function. A callback is a function reference that we give to Twisted (or any other framework) that Twisted will use to “call us back” at the appropriate time, in this case right after the reactor loop starts up. Since Twisted’s loop is separate from our code, most interactions between the reactor core and our business logic will begin with a callback to a function we gave to Twisted using various APIs.

Why we are using callbacks? Callbacks are only source which helps us in converting asynchronous programming concepts into code. While a reactor is started, it starts hearing for an event, and as soon it reaches the event handling portion, it needs to call a certain event handler. So the event handler is the callback, which responds to the event. The advantage is two fold: First it calls the required code block that we want to execute; and secondly it helps in dealing with asynchronous part.

Before we discuss how Twisted helps in dealing with asynchronous part, let us discuss how reactor and callbacks have relation :

 

Reactor loop and Callback Taken from krondo.com

1. Our callback code runs in the same thread as the Twisted loop.
2. When our callbacks are running, the Twisted loop is not running.
3. And vice versa.
4. The reactor loop resumes when our callback returns.

Goodbye, Twisted

It turns out you can tell the Twisted reactor to stop running by using the reactor’s stop method. But once stopped the reactor cannot be restarted, so it’s generally something you do only when your program needs to exit.

How Twisted reacts to ErrorHandling

Since Twisted often ends up calling our code in the form of callbacks, you might wonder what happens when a callback raises an exception.

During the execution of the code, any exception that is raised is displayed in the stack traceback. However, instead of stopping the reactor, it executes the another callback. This shows that Twisted deals with the Exceptions in such a way that the reactor does not stop.  This is imperative, in spite of this appearing as sloppy, because if the network stops due to a bug, then its robustness goes for a toss. So the above characteristics maintain its robustness.

 

Twisted Tutorials – I

This article series is going to deal with Twisted’s tutorial, the writer’s understanding about the Twisted framework. The entire portion has been taken from http://www.krondo.com, so for the entire series of Twisted tutorials, I will be referring to the above website for understanding.

Introduction to Asynchronous Programming

Asynchronous programming is a concept of programming, in which the program that we have written, would react in an asynchronous way. Now trivially, we write programs, which are executed in a synchronous fashion. Until and unless, a portion of work is executed, the program hangs in balance. As soon it receives the confirmation, the program moves forward, and executes the work.

Imagine downloading 3 poems, using synchronous programming. When we connect to a server, it fetches the poem, one at a time. The rate at which the poem is fetched, depends on the server, as the poem is fetched at a certain bandwidth of speed. The another poem is not downloaded until and unless, the first one is finished. So this is the gist of synchronous programming.

Using asynchronous programming, we can utilise the time slot between the time lag of receiving the data. The bandwidth of receiving data, limits us in receiving data. So the time remaining is utilised in starting another interaction with same or different server. So we are utilising our time in getting data.

How Twisted comes into picture

Twisted is a framework that helps us in implementing asynchronous programming. There are enough tools available in python, which helps us in implementing asynchronous programming. However, instead of worrying about the low-level intricacies, Twisted provides us with high-level abstraction, in order to deal with the problems. Hence Twisted eases our work of diving into the low-level abstraction, and provides us with ready-made modules to apply the concept of asynchronous programming.

 

Libraries used in Scrapy – III

This article deals with the third series of libraries used in Scrapy.

1. from io import BytesIO – A stream implementation using an in-memory bytes buffer.
2. from six.moves.urllib.parse import urljoin – Module to join two urls.
3. from importlib import import_module – Imports a module
4. import inspect – Module that inspects live objects, methods and functions.
5. import webbrowser – This module helps in dealing with functions related with webbrowser.
6. import tempfile – This module creates temporary files and directories. It works on all supported platforms.
7. import hashlib – This module implements a common interface to many different secure hash and message digest algorithms.
8. import gc – This module provides an interface to the optional garbage collector. It provides the ability to disable the collector, tune the collection frequency, and set debugging options.
9. import errno – This module makes available standard errno system symbols. The value of each symbol is the corresponding integer value. The names and descriptions are borrowed from linux/include/errno.h, which should be pretty all-inclusive.
10. from functools import partial, wrap – The functools module is for higher-order functions: functions that act on or return other functions. In general, any callable object can be treated as a function for the purposes of this module.
11. import sys – This module provides access to some variables used or maintained by the interpreter and to functions that interact strongly with the interpreter. It is always available.

12. from six.moves import cPickle as pickle – This module actually converts python objects into serialized values, which are quite easy to transmit. The serialization helps us to transfer the data at a faster rate, rather than using primary methods of transmitting data from one place to another. pickle is a python module that helps us in doing this, and cPickle is a faster form of pickle, written in C, which performs our operations.
13. from pkgutil import iter_modules – This module is useful in dealing with utilities of import system, particularly the package support. The itermodulesYields (module_loader, name, ispkg) for all submodules on path, or, if path is None, all top-level modules on sys.path.path should be either None or a list of paths to look for modules in.

    prefix is a string to output on the front of every module name on output.

14. from w3lib.html import replace_entities – Remove entities from the given `text` by converting them to their corresponding unicode character.
15. import struct – This module performs conversions between Python values and C structs represented as Python strings. This can be used in handling binary data stored in files or from network connections, among other sources.

Libraries used in Scrapy – IV

This article is the fourth of the series of the libraries used in Scrapy.

1. from ftplib import error_perm – This module defines the class FTP and a few related items. The FTP class implements the client side of the FTP protocol. error_perm –  Exception raised when an error code signifying a permanent error (response codes in the range 500–599) is received.
2. import copy – Assignment statements in Python do not copy objects, they create bindings between a target and an object. For collections that are mutable or contain mutable items, a copy is sometimes needed so one can change one copy without changing the other. This module provides generic shallow and deep copy operations (explained below).
3. from collections import OrderedDict, Mapping – First module is used for storing key – value pairs in order. The second module implements mapping.
4.  import numbers – This module provides abstract classes for the numbers.
5. import botocore – This module helps in implementing low-level interface to Amazon Web Services.
6. from abc import ABCMeta – Metaclass for defining Abstract Base Classes (ABCs).
7. from zope.interface.verify import verifyClass, DoesNotImplement – The zope.interface.verify module provides functions that test whether a given interface is implemented by a class or provided by an object, resp.
8. import optparse – optparse is a more convenient, flexible, and powerful library for parsing command-line options, However optparse is Deprecated.
9. import cProfile – cProfile and profile provide deterministic profiling of Python programs. A profile is a set of statistics that describes how often and for how long various parts of the program executed.
10. import pkg_resources.
11. import shutil – The shutil module offers a number of high-level operations on files and collections of files. In particular, functions are provided which support file copying and removal.For operations on individual files, see also the os module.
12. from threading import Thread – A class that represents a thread of control. This class can be safely subclassed in a limited fashion.
13. import json – This module helps in implementing all the features of json.
14. from w3lib.url import is_url – This module helps in returning the url part, stripping off the http:// or https:// part.
15. from unittest import TextTestRunner, TextTestResult – A basic test runner implementation which prints results on standard error. It has a few configurable parameters, but is essentially very simple. Graphical applications which run test suites should provide alternate implementations. (b)A concrete implementation of TestResult used by the TextTestRunner.
16. import subprocess – The subprocess module provides a consistent interface to creating and working with additional processes.
17.  from optparse import OptionGroup – When dealing with many options, it is convenient to group these options for better help output. An OptionParser can contain several option groups, each of which can contain several options.
18. from collections import deque – Deques are a generalization of stacks and queues (the name is pronounced “deck” and is short for “double-ended queue”). Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction.
19.  from w3lib.url import file_uri_to_path – Convert File URI to local filesystem path.
20. from w3lib.url import parse_data_uri – Parse a data: URI, returning a 3-tuple of media type, dictionary of media Parse a data: URI, returning a 3-tuple of media type, dictionary of media.
21.  from six.moves.urllib import robotparser – This module provides a single class, RobotFileParser, which answers questions about whether or not a particular user agent can fetch a URL on the Web site that published the robots.txt file.
22. from w3lib.url import safe_url_string – Convert the given URL into a legal URL by escaping unsafe characters
    according to RFC-3986.
    If a bytes URL is given, it is first converted to `str` using the given
    encoding (which defaults to ‘utf-8’). ‘utf-8’ encoding is used for
    URL path component (unless overriden by path_encoding), and given
    encoding is used for query string or form data.
    When passing an encoding, you should use the encoding of the
    original page (the page from which the URL was extracted from).
    Calling this function on an already “safe” URL will return the URL
    unmodified.
    Always returns a native `str` (bytes in Python2, unicode in Python3).
23. import base64 – The base64 encoding scheme is used to convert arbitrary binary data to plain text.
24. from six.moves.urllib.request import getproxies – This helper function returns a dictionary of scheme to proxy server URL mappings.
25. from six.moves.urllib.parse import unquote – Replace %xx escapes by their single-character equivalent.
26. import zlib – This module helps in compressing the data.
27. from email.utils import formatdate -Returns a date string as per RFC 2822, e.g.:

    Fri, 09 Nov 2001 01:08:47 -0000

28. from w3lib.http import basic_auth_header – Return an `Authorization` header field value for `HTTP Basic Access Authentication (RFC 2617)`
29. import bz2 – This module provides a comprehensive interface for the bz2 compression library. It implements a complete file interface, one-shot (de)compression functions, and types for sequential (de)compression.
30. import gzip – This module provides a simple interface to compress and decompress files just like the GNU programs gzip and gunzip would.
31. import zipfile – The zipfile module can be used to manipulate ZIP archive files.
32. import tarfile – The tarfile module makes it possible to read and write tar archives, including those using gzip or bz2 compression.
33. import mktemp – A CONFUSING MODULE, CURRENTLY NOT UNDERSTOOD BY THE USER.
34. import Guppy – Guppy-PE is a library and programming environment for Python, currently providing in particular the Heapy subsystem, which supports object and heap memory sizing, profiling and debugging. It also includes a prototypical specification language, the Guppy Specification Language (GSL), which can be used to formally specify aspects of Python programs and generate tests and documentation from a common source.
35. import socket – This module provides access to the BSD socket interface. It is available on all modern Unix systems, Windows, Mac OS X, BeOS, OS/2, and probably additional platforms.
36. from tempfile import NamedTemporaryFile – This function operates exactly as TemporaryFile() does, except that the file is guaranteed to have a visible name in the file system (on Unix, the directory entry is not unlinked).
37. import traceback – This module provides a standard interface to extract, format and print stack traces of Python programs. It exactly mimics the behavior of the Python interpreter when it prints a stack trace. This is useful when you want to print stack traces under program control, such as in a “wrapper” around the interpreter.
38. import threading – This module constructs higher-level threading interfaces on top of the lower level thread module.
39. from Pdb import pdb – The module pdb defines an interactive source code debugger for Python programs. It supports setting (conditional) breakpoints and single stepping at the source line level, inspection of stack frames, source code listing, and evaluation of arbitrary Python code in the context of any stack frame. It also supports post-mortem debugging and can be called under program control.
40. from six.moves.http_cookiejar import CookieJar -The http.cookiejar module defines classes for automatic handling of HTTP cookies. It is useful for accessing web sites that require small pieces of data – cookies – to be set on the client machine by an HTTP response from a web server, and then returned to the server in later HTTP requests.
41. from six.moves import xmlrpc_client – XML-RPC is a Remote Procedure Call method that uses XML passed via HTTP(S) as a transport. With it, a client can call methods with parameters on a remote server (the server is named by a URI) and get back structured data. This module supports writing XML-RPC client code; it handles all the details of translating between conformable Python objects and XML on the wire.
42. from sgmllib import SGMLParser  – This module is DEPRECATED IN Python.
43. from functools import partial – Return a new partial object which when called will behave like func called with the positional arguments args and keyword arguments keywords
44. from parsel import Selectors – This module helps in selecting portions using parsel
45. from collections import MutableMapping – This module is an abstract classABCs for read-only and mutable mappings.

Libraries used in Scrapy – II

This article is going to deal with another set of libraries used in scrapy.

1. import warnings – This module is used for issuing warnings to the user.
2. from w3lib.url import any_to_uri – uri module contains functions which are not present generally. If given a path name, return its File URI, otherwise return it unmodified.
3. import platform – This module tells which os one is operating
4. import sys – This module provides some access with the interpreter
5. import cssselect – cssselect parses CSS3 Selectors and translate them to XPath 1.0 expressions. Such expressions can be used in lxml or another XPath engine to find the matching elements in an XML or HTML document.
6. import lxml.etree – The etree module of lxml is used for parsing the xml elements of the xml files.
7. import parsel  – Parsel is a library to extract data from HTML and XML using XPath and CSS selectors.
8. import w3lib – This module is used for parsing (a) encoding (b)html (c)http(d)uri elements of xml.
9. import posix – This module is used for importing posix keywords.
10. import re – Module for regex expressions
11. from six.moves.urllib.parse import ParseResult, urldefrag, urlparse, urlunparse – Module for url operations.
12. from w3lib.url import _safe_chars, _unquotepath  – Scrapy has its own customized module for w3lib. This module  has two paths: CURRENTLY I CANNOT UNDERSTAND THIS MODULE.
13. import weakref – This Module stores the weak references.
14. from operator import itemgetter – Returns the value in the given format: After f = itemgetter(2), the call f(r) returns r[2].
15. from cStringIO import StringIO as BytesIO – imports cStringIO, the fastest version of StringIO.