diff options
Diffstat (limited to 'doc/developer.rst')
| -rw-r--r-- | doc/developer.rst | 302 |
1 files changed, 302 insertions, 0 deletions
diff --git a/doc/developer.rst b/doc/developer.rst new file mode 100644 index 0000000..b3ef158 --- /dev/null +++ b/doc/developer.rst @@ -0,0 +1,302 @@ +######################## +Developer documentation +######################## + +End-to-end test suite +--------------------- + +A powerful test suite has been developped to test biboumi’s behaviour in +many scenarios. Its goal is to simulate a real-world usage of biboumi, +including its interactions with a real IRC server an a real XMPP client. + +An IRC server is started, with a specific version and configuration, then, +for every scenario that we want to test: + +- Biboumi is started, with a specific configuration +- An XMPP “client” starts, communicates with biboumi and checks that + biboumi responds in the expected way. + +The XMPP client is actually not a real client, it’s a python script that +uses the slixmpp library to imitate an XMPP server that would transmit the +stanzas of one client to its component (biboumi). In real life, the +communication to biboumi is done between the XMPP server and biboumi, but +since the server just forwards the messages that clients send unmodified, +we’ll call that “the clients”. + +A scenario is a list of functions that will be executed one by one, to +verify the behaviour of one specific feature. Ideally, they should be +short and test one specific aspect. + +Run the test suite locally +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Since this requires a lot of dependencies (an IRC server with some TLS +certificate, slixmpp, many libraries…), it might be cumbersome to get +everything on your machine to be able to run them. + +The simplest solution (as long as you have docker installed and properly +configured to be able to run as your developer user… It’s as simple as +“dnf install docker” and “chmod o+rw /var/run/docker.sock”, but that’s not +recommended, because this lets anybody on the system use docker, and +docker is very unsecure) is to follow these instructions: + +.. code-block:: bash + :caption: Start a docker container with everything installed + + docker run --name biboumi-e2e -v /home/louiz/biboumi/:/home/tester/biboumi \ + --add-host="irc.localhost:127.0.0.1" \ + --add-host="biboumi.localhost:127.0.0.1" \ + --rm -it docker.louiz.org/louiz/biboumi/test-alpine \ + /bin/bash + +This creates a container where every dependency is already installed. We +mount your working directory inside the container: be sure to modify the +first path `/home/louiz/biboumi` with your own. The hosts that we add are +needed for the test suite to properly work. + +You can use the test-fedora or test-debian images instead of test-alpine +if you want, but it should not change anything (even if your host machine +uses debian or fedora), alpine is just the lighter one. + +.. note:: + + This container should stay alive as long as you want to run the test + suite. For example if you want to run it many times until your code is + fine and all tests pass, just leave that shell somewhere without + touching it. + +Then, from an other shell (do NOT run that inside the container we just +created): + +.. code-block:: bash + :caption: Configure and build biboumi from inside the container + + docker exec biboumi-e2e sh -c "cd biboumi && mkdir docker-build/ && cd docker-build/ && cmake .." + +This is needed (only once), because if you configure it from your host +machine, then the paths generated by cmake will be all wrong when you try +to compile from inside the container and nothing will work. + +.. code-block:: bash + :caption: Re-compile and run the test suite inside the container + + docker exec biboumi-e2e sh -c "cd biboumi/docker-build && make e2e" + +This should now build everything correctly, and run the test suite. If you +want to re-run it again after you edited something in your source tree, +just run this last command again. You don’t need to touch anything inside +the container again. + +When you’re done, just close the shell we opened with the first command. + +Available functions +~~~~~~~~~~~~~~~~~~~ + +.. py:function:: send_stanza(str) + + sends one stanza to biboumi. The stanza is written entirely + as a string (with a few automatic replacements). The “from” and “to” + values have to be specified everytime, because each stanza can come from + different clients and be directed to any IRC server/channel + + .. code-block:: python + + send_stanza("<message from='{jid_one}/{resource_one}' to='#foo@{biboumi_host}' type='groupchat'><body>coucou</body></message>"), + +.. py:function:: expect_stanza(xpath[, …]) + + Waits for a stanza to be received by biboumi, and checks that this + stanza matches one or more xpath. If the stanza doesn’t match all the + given xpaths, then the scenario ends and we report that as an error. + + .. code-block:: python + + expect_stanza("/message[@from='#foo@{biboumi_host}/{nick_one}']/body[text()='coucou']", + "/message/delay:delay[@from='#foo@{biboumi_host}']"), + + This waits for exactly 1 stanza, that is compared against 2 xpaths. Here + we check that it is a message, that it has the proper `from` value, the + correct body, and a <delay/>. + +.. py:function:: expect_unordered(list_of_xpaths[, list_of_xpaths, …]) + + we wait for more than one stanzas, that could be received in any order. + For example, in certain scenario, we wait for two presence stanzas, but + it’s perfectly valid to receive them in any order (one is for one + client, the other one for an other client). To do that, we pass multiple + lists of xpath. Each list can contain one or more xpath (just like + `expect_stanza`). When a stanza is received, it is compared with all the + xpaths of the first list. If it doesn’t match, it is compared with the + xpaths of the second list, and so on. If nothing matchs, it’s an error + and we stop this scenario. If the stanza matches with one of the xpath + lists, we remove that list, and we wait for the next stanza, until there + are no more xpaths. + + .. code-block:: python + + expect_unordered( + [ + "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_two}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']", + "/presence/muc_user:x/muc_user:status[@code='303']", + ], + [ + "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_two}/{resource_one}']", + ], + [ + "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_one}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']", + "/presence/muc_user:x/muc_user:status[@code='303']", + "/presence/muc_user:x/muc_user:status[@code='110']", + ], + [ + "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_one}/{resource_one}']", + "/presence/muc_user:x/muc_user:status[@code='110']", + ], + ), + + This will wait for 4 stanzas that could be received in any order. + +To avoid many repetitions between each tests, some helpful sequences are +available, `sequences.connection(…)` and `sequences.connection_tls(…)`. +They do all the steps that are needed (send and receive stanzas) to +connect to the component, or an IRC server. + +It’s also possible to reuse one simple scenario into an other scenario. +The most notable example is to start your own scenario with +`scenarios.simple_channel_join.scenario`, if you need your client to be in +a channel before you can start your actual scenario. For example if you +want to test the behaviour of a topic change, you need to first join a +channel. Since this is a very common patern, it’s simpler to just included +this very basic scenario at the start of your own scenarios, instead of +copy pasting the same thing over and over. + +Examples of a scenario +~~~~~~~~~~~~~~~~~~~~~~ + +First example +^^^^^^^^^^^^^ + +Here we’ll describe how to write your own scenario, from scratch. For this, we will take an existing scenario and explain how it was written, line by line. + +See for example the scenario tests/end_to_end/scenarios/self_ping_on_real_channel.py + +.. code-block:: python + + from scenarios import * + +All the tests should start with this import. It imports the file +tests/end_to_end/scenarios/__init__.py This make all the functions +available (send_stanza, expect_stanza…) available, as well as some very +common scenarios that you often need to re-use. + +.. code-block:: python + + scenario = ( + # … + ) + +This is the only required element of your scenario. This object is a tuple of function calls OR other scenarios. + +.. code-block:: python + + scenarios.simple_channel_join.scenario, + +The first line of our scenario is actually including an other existing +scenario. You can find it at +tests/end_to_end/scenarios/simple_channel_join.py As its name shows, it’s +very basic: one client {jid_one}/{resource_one} just joins one room +#foo%{irc_server_one} with the nick {nick_one}. + +Since we want to test the behaviour of a ping to ourself when we are in a +room, we just join this room without repeating everything. + +It is possible to directly insert a scenario inside our scenario without +having to extract all the steps: the test suite is smart enough to detect +that and extract the inner steps automatically. + +.. code-block:: python + + # Send a ping to ourself + send_stanza("<iq type='get' from='{jid_one}/{resource_one}' id='first_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"), + expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='first_ping']"), + +Here we simple send an iq stanza, properly formatted, using the same JIDs +{jid_one}/{resource_one} and #foo%{irc_server_one}/{nick_one} to ping +ourself in the room. We them immediately expect one stanza to be received, +that is the response to our ping. It only contains one single xpath +because everything we need to check can be expressed in one line. + +Note that it is recommended to explain all the steps of your scenario with +comments. This helps understand what is being tested, and why, without +having to analyze all the stanza individually. + +.. code-block:: python + + # Now join the same room, from the same bare JID, behind the same nick + send_stanza("<presence from='{jid_one}/{resource_two}' to='#foo%{irc_server_one}/{nick_one}' />"), + expect_stanza("/presence[@to='{jid_one}/{resource_two}'][@from='#foo%{irc_server_one}/{nick_one}']/muc_user:x/muc_user:item[@affiliation='admin'][@role='moderator']", + "/presence/muc_user:x/muc_user:status[@code='110']"), + + expect_stanza("/message[@from='#foo%{irc_server_one}'][@type='groupchat'][@to='{jid_one}/{resource_two}']/subject[not(text())]"), + +Here we send a presence stanza to join the same channel with an other +resource (note the {resource_two}). As a result, we expect two stanzas: +The first stanza (our self-presence) is checked against two xpaths, and +the second stanza (the empty subject of the room) against only one. + +.. code-block:: python + + # And re-send a self ping + send_stanza("<iq type='get' from='{jid_one}/{resource_one}' id='second_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"), + expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='second_ping']"), + ## And re-do exactly the same thing, just change the resource initiating the self ping + send_stanza("<iq type='get' from='{jid_one}/{resource_two}' id='third_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"), + expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_two}'][@id='third_ping']"), + +And finally, we test a second ping, and check that the behaviour is correct that we now have two resources in that channel. + +Second example +^^^^^^^^^^^^^^ + +Sometimes we want to do more with the received stanzas. For example we +need to extract some values from the received stanzas, to reuse them in +future stanzas we send. The most obvious example is iq IDs, that we need +to extract, to reuse them in our response. + +Let’s use for example the tests/end_to_end/scenarios/execute_incomplete_hello_adhoc_command.py scenario: + +.. code-block:: python + + from scenarios import * + + scenario = ( + send_stanza("<iq type='set' id='hello-command1' from='{jid_one}/{resource_one}' to='{biboumi_host}'><command xmlns='http://jabber.org/protocol/commands' node='hello' action='execute' /></iq>"), + expect_stanza("/iq[@type='result']/commands:command[@node='hello'][@sessionid][@status='executing']", + "/iq/commands:command/commands:actions/commands:complete", + after = save_value("sessionid", extract_attribute("/iq[@type='result']/commands:command[@node='hello']", "sessionid")) + ), + +Here is where the magic happens: as an additional argument to the +expect_stanza function, we pass an other function (callback) with the +“after=” keyword argument. This “after” callback gets called once the +expected stanza has been received and validated. Here we use +`save_value(key, value)`. This function just saves a value in our global +values that can be used with “send_stanza”, associated with the given +“key”. For example if you do `save_value("something_important", "blah")` +then you can use `{something_important}` in any future stanza that you +send and it will be replaced with “blah”. + +But this is only useful if we can save some value that we extract from the +stanza. That’s where `extract_attribute(xpath, attribute_name)` comes into +play. As the first argument, you pass an xpath corresponding to one +specific node of the XML that is received, and the second argument is just +the name of the attribute whose value you want. + +Here, we extract the value of the “sessionid=” in the node `<iq +type='result'><commands:command node='hello' sessionid='…' /></iq>`, and +we save that value, globally, with the name “sessionid”. + +.. code-block:: python + + send_stanza("<iq type='set' id='hello-command2' from='{jid_one}/{resource_one}' to='{biboumi_host}'><command xmlns='http://jabber.org/protocol/commands' node='hello' sessionid='{sessionid}' action='complete'><x xmlns='jabber:x:data' type='submit'></x></command></iq>"), + +Here we send a second iq, to continue our ad-hoc command, and we use {sessionid} to indicate that we are continuing the session we started before. |