summaryrefslogtreecommitdiff
path: root/doc/developer.rst
diff options
context:
space:
mode:
Diffstat (limited to 'doc/developer.rst')
-rw-r--r--doc/developer.rst302
1 files changed, 302 insertions, 0 deletions
diff --git a/doc/developer.rst b/doc/developer.rst
new file mode 100644
index 0000000..b3ef158
--- /dev/null
+++ b/doc/developer.rst
@@ -0,0 +1,302 @@
+########################
+Developer documentation
+########################
+
+End-to-end test suite
+---------------------
+
+A powerful test suite has been developped to test biboumi’s behaviour in
+many scenarios. Its goal is to simulate a real-world usage of biboumi,
+including its interactions with a real IRC server an a real XMPP client.
+
+An IRC server is started, with a specific version and configuration, then,
+for every scenario that we want to test:
+
+- Biboumi is started, with a specific configuration
+- An XMPP “client” starts, communicates with biboumi and checks that
+ biboumi responds in the expected way.
+
+The XMPP client is actually not a real client, it’s a python script that
+uses the slixmpp library to imitate an XMPP server that would transmit the
+stanzas of one client to its component (biboumi). In real life, the
+communication to biboumi is done between the XMPP server and biboumi, but
+since the server just forwards the messages that clients send unmodified,
+we’ll call that “the clients”.
+
+A scenario is a list of functions that will be executed one by one, to
+verify the behaviour of one specific feature. Ideally, they should be
+short and test one specific aspect.
+
+Run the test suite locally
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since this requires a lot of dependencies (an IRC server with some TLS
+certificate, slixmpp, many libraries…), it might be cumbersome to get
+everything on your machine to be able to run them.
+
+The simplest solution (as long as you have docker installed and properly
+configured to be able to run as your developer user… It’s as simple as
+“dnf install docker” and “chmod o+rw /var/run/docker.sock”, but that’s not
+recommended, because this lets anybody on the system use docker, and
+docker is very unsecure) is to follow these instructions:
+
+.. code-block:: bash
+ :caption: Start a docker container with everything installed
+
+ docker run --name biboumi-e2e -v /home/louiz/biboumi/:/home/tester/biboumi \
+ --add-host="irc.localhost:127.0.0.1" \
+ --add-host="biboumi.localhost:127.0.0.1" \
+ --rm -it docker.louiz.org/louiz/biboumi/test-alpine \
+ /bin/bash
+
+This creates a container where every dependency is already installed. We
+mount your working directory inside the container: be sure to modify the
+first path `/home/louiz/biboumi` with your own. The hosts that we add are
+needed for the test suite to properly work.
+
+You can use the test-fedora or test-debian images instead of test-alpine
+if you want, but it should not change anything (even if your host machine
+uses debian or fedora), alpine is just the lighter one.
+
+.. note::
+
+ This container should stay alive as long as you want to run the test
+ suite. For example if you want to run it many times until your code is
+ fine and all tests pass, just leave that shell somewhere without
+ touching it.
+
+Then, from an other shell (do NOT run that inside the container we just
+created):
+
+.. code-block:: bash
+ :caption: Configure and build biboumi from inside the container
+
+ docker exec biboumi-e2e sh -c "cd biboumi && mkdir docker-build/ && cd docker-build/ && cmake .."
+
+This is needed (only once), because if you configure it from your host
+machine, then the paths generated by cmake will be all wrong when you try
+to compile from inside the container and nothing will work.
+
+.. code-block:: bash
+ :caption: Re-compile and run the test suite inside the container
+
+ docker exec biboumi-e2e sh -c "cd biboumi/docker-build && make e2e"
+
+This should now build everything correctly, and run the test suite. If you
+want to re-run it again after you edited something in your source tree,
+just run this last command again. You don’t need to touch anything inside
+the container again.
+
+When you’re done, just close the shell we opened with the first command.
+
+Available functions
+~~~~~~~~~~~~~~~~~~~
+
+.. py:function:: send_stanza(str)
+
+ sends one stanza to biboumi. The stanza is written entirely
+ as a string (with a few automatic replacements). The “from” and “to”
+ values have to be specified everytime, because each stanza can come from
+ different clients and be directed to any IRC server/channel
+
+ .. code-block:: python
+
+ send_stanza("<message from='{jid_one}/{resource_one}' to='#foo@{biboumi_host}' type='groupchat'><body>coucou</body></message>"),
+
+.. py:function:: expect_stanza(xpath[, …])
+
+ Waits for a stanza to be received by biboumi, and checks that this
+ stanza matches one or more xpath. If the stanza doesn’t match all the
+ given xpaths, then the scenario ends and we report that as an error.
+
+ .. code-block:: python
+
+ expect_stanza("/message[@from='#foo@{biboumi_host}/{nick_one}']/body[text()='coucou']",
+ "/message/delay:delay[@from='#foo@{biboumi_host}']"),
+
+ This waits for exactly 1 stanza, that is compared against 2 xpaths. Here
+ we check that it is a message, that it has the proper `from` value, the
+ correct body, and a <delay/>.
+
+.. py:function:: expect_unordered(list_of_xpaths[, list_of_xpaths, …])
+
+ we wait for more than one stanzas, that could be received in any order.
+ For example, in certain scenario, we wait for two presence stanzas, but
+ it’s perfectly valid to receive them in any order (one is for one
+ client, the other one for an other client). To do that, we pass multiple
+ lists of xpath. Each list can contain one or more xpath (just like
+ `expect_stanza`). When a stanza is received, it is compared with all the
+ xpaths of the first list. If it doesn’t match, it is compared with the
+ xpaths of the second list, and so on. If nothing matchs, it’s an error
+ and we stop this scenario. If the stanza matches with one of the xpath
+ lists, we remove that list, and we wait for the next stanza, until there
+ are no more xpaths.
+
+ .. code-block:: python
+
+ expect_unordered(
+ [
+ "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_two}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']",
+ "/presence/muc_user:x/muc_user:status[@code='303']",
+ ],
+ [
+ "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_two}/{resource_one}']",
+ ],
+ [
+ "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_one}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']",
+ "/presence/muc_user:x/muc_user:status[@code='303']",
+ "/presence/muc_user:x/muc_user:status[@code='110']",
+ ],
+ [
+ "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_one}/{resource_one}']",
+ "/presence/muc_user:x/muc_user:status[@code='110']",
+ ],
+ ),
+
+ This will wait for 4 stanzas that could be received in any order.
+
+To avoid many repetitions between each tests, some helpful sequences are
+available, `sequences.connection(…)` and `sequences.connection_tls(…)`.
+They do all the steps that are needed (send and receive stanzas) to
+connect to the component, or an IRC server.
+
+It’s also possible to reuse one simple scenario into an other scenario.
+The most notable example is to start your own scenario with
+`scenarios.simple_channel_join.scenario`, if you need your client to be in
+a channel before you can start your actual scenario. For example if you
+want to test the behaviour of a topic change, you need to first join a
+channel. Since this is a very common patern, it’s simpler to just included
+this very basic scenario at the start of your own scenarios, instead of
+copy pasting the same thing over and over.
+
+Examples of a scenario
+~~~~~~~~~~~~~~~~~~~~~~
+
+First example
+^^^^^^^^^^^^^
+
+Here we’ll describe how to write your own scenario, from scratch. For this, we will take an existing scenario and explain how it was written, line by line.
+
+See for example the scenario tests/end_to_end/scenarios/self_ping_on_real_channel.py
+
+.. code-block:: python
+
+ from scenarios import *
+
+All the tests should start with this import. It imports the file
+tests/end_to_end/scenarios/__init__.py This make all the functions
+available (send_stanza, expect_stanza…) available, as well as some very
+common scenarios that you often need to re-use.
+
+.. code-block:: python
+
+ scenario = (
+ # …
+ )
+
+This is the only required element of your scenario. This object is a tuple of function calls OR other scenarios.
+
+.. code-block:: python
+
+ scenarios.simple_channel_join.scenario,
+
+The first line of our scenario is actually including an other existing
+scenario. You can find it at
+tests/end_to_end/scenarios/simple_channel_join.py As its name shows, it’s
+very basic: one client {jid_one}/{resource_one} just joins one room
+#foo%{irc_server_one} with the nick {nick_one}.
+
+Since we want to test the behaviour of a ping to ourself when we are in a
+room, we just join this room without repeating everything.
+
+It is possible to directly insert a scenario inside our scenario without
+having to extract all the steps: the test suite is smart enough to detect
+that and extract the inner steps automatically.
+
+.. code-block:: python
+
+ # Send a ping to ourself
+ send_stanza("<iq type='get' from='{jid_one}/{resource_one}' id='first_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"),
+ expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='first_ping']"),
+
+Here we simple send an iq stanza, properly formatted, using the same JIDs
+{jid_one}/{resource_one} and #foo%{irc_server_one}/{nick_one} to ping
+ourself in the room. We them immediately expect one stanza to be received,
+that is the response to our ping. It only contains one single xpath
+because everything we need to check can be expressed in one line.
+
+Note that it is recommended to explain all the steps of your scenario with
+comments. This helps understand what is being tested, and why, without
+having to analyze all the stanza individually.
+
+.. code-block:: python
+
+ # Now join the same room, from the same bare JID, behind the same nick
+ send_stanza("<presence from='{jid_one}/{resource_two}' to='#foo%{irc_server_one}/{nick_one}' />"),
+ expect_stanza("/presence[@to='{jid_one}/{resource_two}'][@from='#foo%{irc_server_one}/{nick_one}']/muc_user:x/muc_user:item[@affiliation='admin'][@role='moderator']",
+ "/presence/muc_user:x/muc_user:status[@code='110']"),
+
+ expect_stanza("/message[@from='#foo%{irc_server_one}'][@type='groupchat'][@to='{jid_one}/{resource_two}']/subject[not(text())]"),
+
+Here we send a presence stanza to join the same channel with an other
+resource (note the {resource_two}). As a result, we expect two stanzas:
+The first stanza (our self-presence) is checked against two xpaths, and
+the second stanza (the empty subject of the room) against only one.
+
+.. code-block:: python
+
+ # And re-send a self ping
+ send_stanza("<iq type='get' from='{jid_one}/{resource_one}' id='second_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"),
+ expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='second_ping']"),
+ ## And re-do exactly the same thing, just change the resource initiating the self ping
+ send_stanza("<iq type='get' from='{jid_one}/{resource_two}' id='third_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"),
+ expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_two}'][@id='third_ping']"),
+
+And finally, we test a second ping, and check that the behaviour is correct that we now have two resources in that channel.
+
+Second example
+^^^^^^^^^^^^^^
+
+Sometimes we want to do more with the received stanzas. For example we
+need to extract some values from the received stanzas, to reuse them in
+future stanzas we send. The most obvious example is iq IDs, that we need
+to extract, to reuse them in our response.
+
+Let’s use for example the tests/end_to_end/scenarios/execute_incomplete_hello_adhoc_command.py scenario:
+
+.. code-block:: python
+
+ from scenarios import *
+
+ scenario = (
+ send_stanza("<iq type='set' id='hello-command1' from='{jid_one}/{resource_one}' to='{biboumi_host}'><command xmlns='http://jabber.org/protocol/commands' node='hello' action='execute' /></iq>"),
+ expect_stanza("/iq[@type='result']/commands:command[@node='hello'][@sessionid][@status='executing']",
+ "/iq/commands:command/commands:actions/commands:complete",
+ after = save_value("sessionid", extract_attribute("/iq[@type='result']/commands:command[@node='hello']", "sessionid"))
+ ),
+
+Here is where the magic happens: as an additional argument to the
+expect_stanza function, we pass an other function (callback) with the
+“after=” keyword argument. This “after” callback gets called once the
+expected stanza has been received and validated. Here we use
+`save_value(key, value)`. This function just saves a value in our global
+values that can be used with “send_stanza”, associated with the given
+“key”. For example if you do `save_value("something_important", "blah")`
+then you can use `{something_important}` in any future stanza that you
+send and it will be replaced with “blah”.
+
+But this is only useful if we can save some value that we extract from the
+stanza. That’s where `extract_attribute(xpath, attribute_name)` comes into
+play. As the first argument, you pass an xpath corresponding to one
+specific node of the XML that is received, and the second argument is just
+the name of the attribute whose value you want.
+
+Here, we extract the value of the “sessionid=” in the node `<iq
+type='result'><commands:command node='hello' sessionid='…' /></iq>`, and
+we save that value, globally, with the name “sessionid”.
+
+.. code-block:: python
+
+ send_stanza("<iq type='set' id='hello-command2' from='{jid_one}/{resource_one}' to='{biboumi_host}'><command xmlns='http://jabber.org/protocol/commands' node='hello' sessionid='{sessionid}' action='complete'><x xmlns='jabber:x:data' type='submit'></x></command></iq>"),
+
+Here we send a second iq, to continue our ad-hoc command, and we use {sessionid} to indicate that we are continuing the session we started before.