summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorlouiz’ <louiz@louiz.org>2019-11-07 18:59:43 +0100
committerlouiz’ <louiz@louiz.org>2019-11-09 21:06:02 +0100
commit7f2e74c09161737ca0b84995c5017b0fd043f32b (patch)
treed79f13d0f08dd6c6447d273404136d179ee2d8be
parent7571020243889db72bb0182dbacc84e506afa279 (diff)
downloadbiboumi-7f2e74c09161737ca0b84995c5017b0fd043f32b.tar.gz
biboumi-7f2e74c09161737ca0b84995c5017b0fd043f32b.tar.bz2
biboumi-7f2e74c09161737ca0b84995c5017b0fd043f32b.tar.xz
biboumi-7f2e74c09161737ca0b84995c5017b0fd043f32b.zip
Add a doc describing how to write e2e tests
-rw-r--r--CHANGELOG.rst6
-rw-r--r--doc/contributing.rst28
-rw-r--r--doc/developper.rst240
-rw-r--r--doc/index.rst1
4 files changed, 250 insertions, 25 deletions
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 7e838e4..172bb7c 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -25,6 +25,12 @@ For admins
still be set at configure time by passing the option "-DWATCHDOG_SEC=20”
to cmake, if you want to enable the systemd watchdog.
+For developers
+--------------
+- The end-to-end tests have been refactored, cleaned and simplified a lot.
+ A tutorial and a documentation have been written. It should now be easy
+ to write a test that demonstrates a bug or a missing feature.
+
Version 8.3 - 2018-06-01
========================
diff --git a/doc/contributing.rst b/doc/contributing.rst
index 26572fa..8f01c82 100644
--- a/doc/contributing.rst
+++ b/doc/contributing.rst
@@ -50,35 +50,13 @@ Tests
There are two test suites for biboumi:
- unit tests that can be run simply using `make check`.
- These tests use the Catch test framework, are written in pure C++
+ These tests use the Catch2 test framework, are written in pure C++
and they should always succeed, in all possible build configuration.
- a more complex end-to-end test suite. This test suite is written in python3,
uses a specific IRC server (`charybdis`_), and only tests the most complete
- biboumi configuration (when all dependencies are used). To run it, you need
- to install various dependencies: refer to fedora’s `Dockerfile.base`_ and
- `Dockerfile`_ to see how to install charybdis, slixmpp, botan, a ssl
- certificate, etc.
-
- Once all the dependencies are correctly installed, the tests are run with
-
-.. code-block:: sh
-
- make e2e
-
-To run one or more specific tests, you can do something like this:
-
-.. code-block:: sh
-
- make biboumi && python3 ../tests/end_to_end self_ping basic_handshake_success
-
-This will run two tests, self_ping and basic_handshake_success.
-
-To write additional tests, you need to add a Scenario
-into `the __main__.py file`_. If you have problem running this end-to-end
-test suite, or if you struggle with this weird code (that would be
-completely normal…), don’t hesitate to ask for help.
-
+ biboumi configuration (when all dependencies are used).
+ Read more about these tests in the specific documentation TODO.
All these tests automatically run with various configurations, on various
platforms, using gitlab CI.
diff --git a/doc/developper.rst b/doc/developper.rst
new file mode 100644
index 0000000..1df9826
--- /dev/null
+++ b/doc/developper.rst
@@ -0,0 +1,240 @@
+########################
+Developper documentation
+########################
+
+End-to-end test suite
+---------------------
+
+A powerful test suite has been developped to test biboumi’s behaviour in
+many scenarios. Its goal is to simulate a real-world usage of biboumi,
+including its interactions with a real IRC server an a real XMPP client.
+
+An IRC server is started, with a specific version and configuration, then,
+for every scenario that we want to test:
+
+- Biboumi is started, with a specific configuration
+- An XMPP “client” starts, communicates with biboumi and checks that
+ biboumi responds in the expected way.
+
+The XMPP client is actually not a real client, it’s a python script that
+uses the slixmpp library to imitate an XMPP server that would transmit the
+stanzas of one client to its component (biboumi). In real life, the
+communication to biboumi is done between the XMPP server and biboumi, but
+since the server just forwards the messages that clients send unmodified,
+we’ll call that “the clients”.
+
+A scenario is a list of functions that will be executed one by one, to
+verify the behaviour of one specific feature. Ideally, they should be
+short and test one specific aspect.
+
+Available functions
+~~~~~~~~~~~~~~~~~~~
+
+.. py:function:: send_stanza(str)
+
+ sends one stanza to biboumi. The stanza is written entirely
+ as a string (with a few automatic replacements). The “from” and “to”
+ values have to be specified everytime, because each stanza can come from
+ different clients and be directed to any IRC server/channel
+
+ .. code-block:: python
+
+ send_stanza("<message from='{jid_one}/{resource_one}' to='#foo@{biboumi_host}' type='groupchat'><body>coucou</body></message>"),
+
+.. py:function:: expect_stanza(xpath[, …])
+
+ Waits for a stanza to be received by biboumi, and checks that this
+ stanza matches one or more xpath. If the stanza doesn’t match all the
+ given xpaths, then the scenario ends and we report that as an error.
+
+ .. code-block:: python
+
+ expect_stanza("/message[@from='#foo@{biboumi_host}/{nick_one}']/body[text()='coucou']",
+ "/message/delay:delay[@from='#foo@{biboumi_host}']"),
+
+ This waits for exactly 1 stanza, that is compared against 2 xpaths. Here
+ we check that it is a message, that it has the proper `from` value, the
+ correct body, and a <delay/>.
+
+.. py:function:: expect_unordered(list_of_xpaths[, list_of_xpaths, …])
+
+ we wait for more than one stanzas, that could be received in any order.
+ For example, in certain scenario, we wait for two presence stanzas, but
+ it’s perfectly valid to receive them in any order (one is for one
+ client, the other one for an other client). To do that, we pass multiple
+ lists of xpath. Each list can contain one or more xpath (just like
+ `expect_stanza`). When a stanza is received, it is compared with all the
+ xpaths of the first list. If it doesn’t match, it is compared with the
+ xpaths of the second list, and so on. If nothing matchs, it’s an error
+ and we stop this scenario. If the stanza matches with one of the xpath
+ lists, we remove that list, and we wait for the next stanza, until there
+ are no more xpaths.
+
+ .. code-block:: python
+
+ expect_unordered(
+ [
+ "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_two}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']",
+ "/presence/muc_user:x/muc_user:status[@code='303']",
+ ],
+ [
+ "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_two}/{resource_one}']",
+ ],
+ [
+ "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_one}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']",
+ "/presence/muc_user:x/muc_user:status[@code='303']",
+ "/presence/muc_user:x/muc_user:status[@code='110']",
+ ],
+ [
+ "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_one}/{resource_one}']",
+ "/presence/muc_user:x/muc_user:status[@code='110']",
+ ],
+ ),
+
+ This will wait for 4 stanzas that could be received in any order.
+
+To avoid many repetitions between each tests, some helpful sequences are
+available, `sequences.connection(…)` and `sequences.connection_tls(…)`.
+They do all the steps that are needed (send and receive stanzas) to
+connect to the component, or an IRC server.
+
+It’s also possible to reuse one simple scenario into an other scenario.
+The most notable example is to start your own scenario with
+`scenarios.simple_channel_join.scenario`, if you need your client to be in
+a channel before you can start your actual scenario. For example if you
+want to test the behaviour of a topic change, you need to first join a
+channel. Since this is a very common patern, it’s simpler to just included
+this very basic scenario at the start of your own scenarios, instead of
+copy pasting the same thing over and over.
+
+Examples of a scenario
+~~~~~~~~~~~~~~~~~~~~~~
+
+First example
+^^^^^^^^^^^^^
+
+Here we’ll describe how to write your own scenario, from scratch. For this, we will take an existing scenario and explain how it was written, line by line.
+
+See for example the scenario tests/end_to_end/scenarios/self_ping_on_real_channel.py
+
+.. code-block:: python
+
+ from scenarios import *
+
+All the tests should start with this import. It imports the file
+tests/end_to_end/scenarios/__init__.py This make all the functions
+available (send_stanza, expect_stanza…) available, as well as some very
+common scenarios that you often need to re-use.
+
+.. code-block:: python
+
+ scenario = (
+ # …
+ )
+
+This is the only required element of your scenario. This object is a tuple of funcion calls OR other scenarios.
+
+.. code-block:: python
+
+ scenarios.simple_channel_join.scenario,
+
+The first line of our scenario is actually including an other existing
+scenario. You can find it at
+tests/end_to_end/scenarios/simple_channel_join.py As its name shows, it’s
+very basic: one client {jid_one}/{resource_one} just joins one room
+#foo%{irc_server_one} with the nick {nick_one}.
+
+Since we want to test the behaviour of a ping to ourself when we are in a
+room, we just join this room without repeating everything.
+
+It is possible to directly insert a scenario inside our scenario without
+having to extract all the steps: the test suite is smart enough to detect
+that and extract the inner steps automatically.
+
+.. code-block:: python
+
+ # Send a ping to ourself
+ send_stanza("<iq type='get' from='{jid_one}/{resource_one}' id='first_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"),
+ expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='first_ping']"),
+
+Here we simple send an iq stanza, properly formatted, using the same JIDs
+{jid_one}/{resource_one} and #foo%{irc_server_one}/{nick_one} to ping
+ourself in the room. We them immediately expect one stanza to be received,
+that is the response to our ping. It only contains one single xpath
+because everything we need to check can be expressed in one line.
+
+Note that it is recommended to explain all the steps of your scenario with
+comments. This helps understand what is being tested, and why, without
+having to analyze all the stanza individually.
+
+.. code-block:: python
+
+ # Now join the same room, from the same bare JID, behind the same nick
+ send_stanza("<presence from='{jid_one}/{resource_two}' to='#foo%{irc_server_one}/{nick_one}' />"),
+ expect_stanza("/presence[@to='{jid_one}/{resource_two}'][@from='#foo%{irc_server_one}/{nick_one}']/muc_user:x/muc_user:item[@affiliation='admin'][@role='moderator']",
+ "/presence/muc_user:x/muc_user:status[@code='110']"),
+
+ expect_stanza("/message[@from='#foo%{irc_server_one}'][@type='groupchat'][@to='{jid_one}/{resource_two}']/subject[not(text())]"),
+
+Here we send a presence stanza to join the same channel with an other
+resource (note the {resource_two}). As a result, we expect two stanzas:
+The first stanza (our self-presence) is checked against two xpaths, and
+the second stanza (the empty subject of the room) against only one.
+
+.. code-block:: python
+
+ # And re-send a self ping
+ send_stanza("<iq type='get' from='{jid_one}/{resource_one}' id='second_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"),
+ expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='second_ping']"),
+ ## And re-do exactly the same thing, just change the resource initiating the self ping
+ send_stanza("<iq type='get' from='{jid_one}/{resource_two}' id='third_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"),
+ expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_two}'][@id='third_ping']"),
+
+And finally, we test a second ping, and check that the behaviour is correct that we now have two resources in that channel.
+
+Second example
+^^^^^^^^^^^^^^
+
+Sometimes we want to do more with the received stanzas. For example we
+need to extract some values from the received stanzas, to reuse them in
+future stanzas we send. The most obvious example is iq IDs, that we need
+to extract, to reuse them in our response.
+
+Let’s use for example the tests/end_to_end/scenarios/execute_incomplete_hello_adhoc_command.py scenario:
+
+.. code-block:: python
+
+ from scenarios import *
+
+ scenario = (
+ send_stanza("<iq type='set' id='hello-command1' from='{jid_one}/{resource_one}' to='{biboumi_host}'><command xmlns='http://jabber.org/protocol/commands' node='hello' action='execute' /></iq>"),
+ expect_stanza("/iq[@type='result']/commands:command[@node='hello'][@sessionid][@status='executing']",
+ "/iq/commands:command/commands:actions/commands:complete",
+ after = save_value("sessionid", extract_attribute("/iq[@type='result']/commands:command[@node='hello']", "sessionid"))
+ ),
+
+Here is where the magic happens: as an additional argument to the
+expect_stanza function, we pass an other function (callback) with the
+“after=” keyword argument. This “after” callback gets called once the
+expected stanza has been received and validated. Here we use
+`save_value(key, value)`. This function just saves a value in our global
+values that can be used with “send_stanza”, associated with the given
+“key”. For example if you do `save_value("something_important", "blah")`
+then you can use `{something_important}` in any future stanza that you
+send and it will be replaced with “blah”.
+
+But this is only useful if we can save some value that we extract from the
+stanza. That’s where `extract_attribute(xpath, attribute_name)` comes into
+play. As the first argument, you pass an xpath corresponding to one
+specific node of the XML that is received, and the second argument is just
+the name of the attribute whose value you want.
+
+Here, we extract the value of the “sessionid=” in the node `<iq
+type='result'><commands:command node='hello' sessionid='…' /></iq>`, and
+we save that value, globally, with the name “sessionid”.
+
+.. code-block:: python
+
+ send_stanza("<iq type='set' id='hello-command2' from='{jid_one}/{resource_one}' to='{biboumi_host}'><command xmlns='http://jabber.org/protocol/commands' node='hello' sessionid='{sessionid}' action='complete'><x xmlns='jabber:x:data' type='submit'></x></command></iq>"),
+
+Here we send a second iq, to continue our ad-hoc command, and we use {sessionid} to indicate that we are continuing the session we started before.
diff --git a/doc/index.rst b/doc/index.rst
index b97c8fd..e77d898 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -21,3 +21,4 @@ XMPP client as if these channels were XMPP MUCs.
admin
user
contributing
+ developper