######################## Developper documentation ######################## End-to-end test suite --------------------- A powerful test suite has been developped to test biboumi’s behaviour in many scenarios. Its goal is to simulate a real-world usage of biboumi, including its interactions with a real IRC server an a real XMPP client. An IRC server is started, with a specific version and configuration, then, for every scenario that we want to test: - Biboumi is started, with a specific configuration - An XMPP “client” starts, communicates with biboumi and checks that biboumi responds in the expected way. The XMPP client is actually not a real client, it’s a python script that uses the slixmpp library to imitate an XMPP server that would transmit the stanzas of one client to its component (biboumi). In real life, the communication to biboumi is done between the XMPP server and biboumi, but since the server just forwards the messages that clients send unmodified, we’ll call that “the clients”. A scenario is a list of functions that will be executed one by one, to verify the behaviour of one specific feature. Ideally, they should be short and test one specific aspect. Run the test suite locally ~~~~~~~~~~~~~~~~~~~~~~~~~~ Since this requires a lot of dependencies (an IRC server with some TLS certificate, slixmpp, many libraries…), it might be cumbersome to get everything on your machine to be able to run them. The simplest solution (as long as you have docker installed and properly configured to be able to run as your developer user…) is to follow these instructions: .. code-block:: bash :caption: Start a docker container with everything installed docker run --name biboumi-e2e -v /home/louiz/biboumi/:/home/tester/biboumi \ --add-host="irc.localhost:127.0.0.1" \ --add-host="biboumi.localhost:127.0.0.1" \ --rm -it docker.louiz.org/louiz/biboumi/test-alpine \ /bin/bash This creates a container where every dependency is already installed. We mount your working directory inside the container: be sure to modify the first path `/home/louiz/biboumi` with your own. The hosts that we add are needed for the test suite to properly work. You can use the test-fedora or test-debian images instead of test-alpine if you want, but it should not change anything (even if your host machine uses debian or fedora), alpine is just the lighter one. .. note:: This container should stay alive as long as you want to run the test suite. For example if you want to run it many times until your code is fine and all tests pass, just leave that shell somewhere without touching it. Then, from an other shell (do NOT run that inside the container we just created): .. code-block:: bash :caption: Configure and build biboumi from inside the container docker exec biboumi-e2e sh -c "cd biboumi && mkdir build && cmake .." This is needed (only once), because if you configure it from your host machine, then the paths generated by cmake will be all wrong when you try to compile from inside the container and nothing will work. .. code-block:: bash :caption: Re-compile and run the test suite inside the container docker exec biboumi-e2e sh -c "cd biboumi/build && make e2e" This should now build everything correctly, and run the test suite. If you want to re-run it again after you edited something in your source tree, just run this last command again. You don’t need to touch anything inside the container again. When you’re done, just close the shell we opened with the first command. Available functions ~~~~~~~~~~~~~~~~~~~ .. py:function:: send_stanza(str) sends one stanza to biboumi. The stanza is written entirely as a string (with a few automatic replacements). The “from” and “to” values have to be specified everytime, because each stanza can come from different clients and be directed to any IRC server/channel .. code-block:: python send_stanza("coucou"), .. py:function:: expect_stanza(xpath[, …]) Waits for a stanza to be received by biboumi, and checks that this stanza matches one or more xpath. If the stanza doesn’t match all the given xpaths, then the scenario ends and we report that as an error. .. code-block:: python expect_stanza("/message[@from='#foo@{biboumi_host}/{nick_one}']/body[text()='coucou']", "/message/delay:delay[@from='#foo@{biboumi_host}']"), This waits for exactly 1 stanza, that is compared against 2 xpaths. Here we check that it is a message, that it has the proper `from` value, the correct body, and a . .. py:function:: expect_unordered(list_of_xpaths[, list_of_xpaths, …]) we wait for more than one stanzas, that could be received in any order. For example, in certain scenario, we wait for two presence stanzas, but it’s perfectly valid to receive them in any order (one is for one client, the other one for an other client). To do that, we pass multiple lists of xpath. Each list can contain one or more xpath (just like `expect_stanza`). When a stanza is received, it is compared with all the xpaths of the first list. If it doesn’t match, it is compared with the xpaths of the second list, and so on. If nothing matchs, it’s an error and we stop this scenario. If the stanza matches with one of the xpath lists, we remove that list, and we wait for the next stanza, until there are no more xpaths. .. code-block:: python expect_unordered( [ "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_two}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']", "/presence/muc_user:x/muc_user:status[@code='303']", ], [ "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_two}/{resource_one}']", ], [ "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_one}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']", "/presence/muc_user:x/muc_user:status[@code='303']", "/presence/muc_user:x/muc_user:status[@code='110']", ], [ "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_one}/{resource_one}']", "/presence/muc_user:x/muc_user:status[@code='110']", ], ), This will wait for 4 stanzas that could be received in any order. To avoid many repetitions between each tests, some helpful sequences are available, `sequences.connection(…)` and `sequences.connection_tls(…)`. They do all the steps that are needed (send and receive stanzas) to connect to the component, or an IRC server. It’s also possible to reuse one simple scenario into an other scenario. The most notable example is to start your own scenario with `scenarios.simple_channel_join.scenario`, if you need your client to be in a channel before you can start your actual scenario. For example if you want to test the behaviour of a topic change, you need to first join a channel. Since this is a very common patern, it’s simpler to just included this very basic scenario at the start of your own scenarios, instead of copy pasting the same thing over and over. Examples of a scenario ~~~~~~~~~~~~~~~~~~~~~~ First example ^^^^^^^^^^^^^ Here we’ll describe how to write your own scenario, from scratch. For this, we will take an existing scenario and explain how it was written, line by line. See for example the scenario tests/end_to_end/scenarios/self_ping_on_real_channel.py .. code-block:: python from scenarios import * All the tests should start with this import. It imports the file tests/end_to_end/scenarios/__init__.py This make all the functions available (send_stanza, expect_stanza…) available, as well as some very common scenarios that you often need to re-use. .. code-block:: python scenario = ( # … ) This is the only required element of your scenario. This object is a tuple of function calls OR other scenarios. .. code-block:: python scenarios.simple_channel_join.scenario, The first line of our scenario is actually including an other existing scenario. You can find it at tests/end_to_end/scenarios/simple_channel_join.py As its name shows, it’s very basic: one client {jid_one}/{resource_one} just joins one room #foo%{irc_server_one} with the nick {nick_one}. Since we want to test the behaviour of a ping to ourself when we are in a room, we just join this room without repeating everything. It is possible to directly insert a scenario inside our scenario without having to extract all the steps: the test suite is smart enough to detect that and extract the inner steps automatically. .. code-block:: python # Send a ping to ourself send_stanza(""), expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='first_ping']"), Here we simple send an iq stanza, properly formatted, using the same JIDs {jid_one}/{resource_one} and #foo%{irc_server_one}/{nick_one} to ping ourself in the room. We them immediately expect one stanza to be received, that is the response to our ping. It only contains one single xpath because everything we need to check can be expressed in one line. Note that it is recommended to explain all the steps of your scenario with comments. This helps understand what is being tested, and why, without having to analyze all the stanza individually. .. code-block:: python # Now join the same room, from the same bare JID, behind the same nick send_stanza(""), expect_stanza("/presence[@to='{jid_one}/{resource_two}'][@from='#foo%{irc_server_one}/{nick_one}']/muc_user:x/muc_user:item[@affiliation='admin'][@role='moderator']", "/presence/muc_user:x/muc_user:status[@code='110']"), expect_stanza("/message[@from='#foo%{irc_server_one}'][@type='groupchat'][@to='{jid_one}/{resource_two}']/subject[not(text())]"), Here we send a presence stanza to join the same channel with an other resource (note the {resource_two}). As a result, we expect two stanzas: The first stanza (our self-presence) is checked against two xpaths, and the second stanza (the empty subject of the room) against only one. .. code-block:: python # And re-send a self ping send_stanza(""), expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='second_ping']"), ## And re-do exactly the same thing, just change the resource initiating the self ping send_stanza(""), expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_two}'][@id='third_ping']"), And finally, we test a second ping, and check that the behaviour is correct that we now have two resources in that channel. Second example ^^^^^^^^^^^^^^ Sometimes we want to do more with the received stanzas. For example we need to extract some values from the received stanzas, to reuse them in future stanzas we send. The most obvious example is iq IDs, that we need to extract, to reuse them in our response. Let’s use for example the tests/end_to_end/scenarios/execute_incomplete_hello_adhoc_command.py scenario: .. code-block:: python from scenarios import * scenario = ( send_stanza(""), expect_stanza("/iq[@type='result']/commands:command[@node='hello'][@sessionid][@status='executing']", "/iq/commands:command/commands:actions/commands:complete", after = save_value("sessionid", extract_attribute("/iq[@type='result']/commands:command[@node='hello']", "sessionid")) ), Here is where the magic happens: as an additional argument to the expect_stanza function, we pass an other function (callback) with the “after=” keyword argument. This “after” callback gets called once the expected stanza has been received and validated. Here we use `save_value(key, value)`. This function just saves a value in our global values that can be used with “send_stanza”, associated with the given “key”. For example if you do `save_value("something_important", "blah")` then you can use `{something_important}` in any future stanza that you send and it will be replaced with “blah”. But this is only useful if we can save some value that we extract from the stanza. That’s where `extract_attribute(xpath, attribute_name)` comes into play. As the first argument, you pass an xpath corresponding to one specific node of the XML that is received, and the second argument is just the name of the attribute whose value you want. Here, we extract the value of the “sessionid=” in the node ``, and we save that value, globally, with the name “sessionid”. .. code-block:: python send_stanza(""), Here we send a second iq, to continue our ad-hoc command, and we use {sessionid} to indicate that we are continuing the session we started before.