diff options
| -rw-r--r-- | CHANGELOG.rst | 6 | ||||
| -rw-r--r-- | doc/contributing.rst | 28 | ||||
| -rw-r--r-- | doc/developper.rst | 240 | ||||
| -rw-r--r-- | doc/index.rst | 1 |
4 files changed, 250 insertions, 25 deletions
diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 7e838e4..172bb7c 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -25,6 +25,12 @@ For admins still be set at configure time by passing the option "-DWATCHDOG_SEC=20” to cmake, if you want to enable the systemd watchdog. +For developers +-------------- +- The end-to-end tests have been refactored, cleaned and simplified a lot. + A tutorial and a documentation have been written. It should now be easy + to write a test that demonstrates a bug or a missing feature. + Version 8.3 - 2018-06-01 ======================== diff --git a/doc/contributing.rst b/doc/contributing.rst index 26572fa..8f01c82 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -50,35 +50,13 @@ Tests There are two test suites for biboumi: - unit tests that can be run simply using `make check`. - These tests use the Catch test framework, are written in pure C++ + These tests use the Catch2 test framework, are written in pure C++ and they should always succeed, in all possible build configuration. - a more complex end-to-end test suite. This test suite is written in python3, uses a specific IRC server (`charybdis`_), and only tests the most complete - biboumi configuration (when all dependencies are used). To run it, you need - to install various dependencies: refer to fedora’s `Dockerfile.base`_ and - `Dockerfile`_ to see how to install charybdis, slixmpp, botan, a ssl - certificate, etc. - - Once all the dependencies are correctly installed, the tests are run with - -.. code-block:: sh - - make e2e - -To run one or more specific tests, you can do something like this: - -.. code-block:: sh - - make biboumi && python3 ../tests/end_to_end self_ping basic_handshake_success - -This will run two tests, self_ping and basic_handshake_success. - -To write additional tests, you need to add a Scenario -into `the __main__.py file`_. If you have problem running this end-to-end -test suite, or if you struggle with this weird code (that would be -completely normal…), don’t hesitate to ask for help. - + biboumi configuration (when all dependencies are used). + Read more about these tests in the specific documentation TODO. All these tests automatically run with various configurations, on various platforms, using gitlab CI. diff --git a/doc/developper.rst b/doc/developper.rst new file mode 100644 index 0000000..1df9826 --- /dev/null +++ b/doc/developper.rst @@ -0,0 +1,240 @@ +######################## +Developper documentation +######################## + +End-to-end test suite +--------------------- + +A powerful test suite has been developped to test biboumi’s behaviour in +many scenarios. Its goal is to simulate a real-world usage of biboumi, +including its interactions with a real IRC server an a real XMPP client. + +An IRC server is started, with a specific version and configuration, then, +for every scenario that we want to test: + +- Biboumi is started, with a specific configuration +- An XMPP “client” starts, communicates with biboumi and checks that + biboumi responds in the expected way. + +The XMPP client is actually not a real client, it’s a python script that +uses the slixmpp library to imitate an XMPP server that would transmit the +stanzas of one client to its component (biboumi). In real life, the +communication to biboumi is done between the XMPP server and biboumi, but +since the server just forwards the messages that clients send unmodified, +we’ll call that “the clients”. + +A scenario is a list of functions that will be executed one by one, to +verify the behaviour of one specific feature. Ideally, they should be +short and test one specific aspect. + +Available functions +~~~~~~~~~~~~~~~~~~~ + +.. py:function:: send_stanza(str) + + sends one stanza to biboumi. The stanza is written entirely + as a string (with a few automatic replacements). The “from” and “to” + values have to be specified everytime, because each stanza can come from + different clients and be directed to any IRC server/channel + + .. code-block:: python + + send_stanza("<message from='{jid_one}/{resource_one}' to='#foo@{biboumi_host}' type='groupchat'><body>coucou</body></message>"), + +.. py:function:: expect_stanza(xpath[, …]) + + Waits for a stanza to be received by biboumi, and checks that this + stanza matches one or more xpath. If the stanza doesn’t match all the + given xpaths, then the scenario ends and we report that as an error. + + .. code-block:: python + + expect_stanza("/message[@from='#foo@{biboumi_host}/{nick_one}']/body[text()='coucou']", + "/message/delay:delay[@from='#foo@{biboumi_host}']"), + + This waits for exactly 1 stanza, that is compared against 2 xpaths. Here + we check that it is a message, that it has the proper `from` value, the + correct body, and a <delay/>. + +.. py:function:: expect_unordered(list_of_xpaths[, list_of_xpaths, …]) + + we wait for more than one stanzas, that could be received in any order. + For example, in certain scenario, we wait for two presence stanzas, but + it’s perfectly valid to receive them in any order (one is for one + client, the other one for an other client). To do that, we pass multiple + lists of xpath. Each list can contain one or more xpath (just like + `expect_stanza`). When a stanza is received, it is compared with all the + xpaths of the first list. If it doesn’t match, it is compared with the + xpaths of the second list, and so on. If nothing matchs, it’s an error + and we stop this scenario. If the stanza matches with one of the xpath + lists, we remove that list, and we wait for the next stanza, until there + are no more xpaths. + + .. code-block:: python + + expect_unordered( + [ + "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_two}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']", + "/presence/muc_user:x/muc_user:status[@code='303']", + ], + [ + "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_two}/{resource_one}']", + ], + [ + "/presence[@from='#foo%{irc_server_one}/{nick_one}'][@to='{jid_one}/{resource_one}'][@type='unavailable']/muc_user:x/muc_user:item[@nick='Bernard']", + "/presence/muc_user:x/muc_user:status[@code='303']", + "/presence/muc_user:x/muc_user:status[@code='110']", + ], + [ + "/presence[@from='#foo%{irc_server_one}/{nick_three}'][@to='{jid_one}/{resource_one}']", + "/presence/muc_user:x/muc_user:status[@code='110']", + ], + ), + + This will wait for 4 stanzas that could be received in any order. + +To avoid many repetitions between each tests, some helpful sequences are +available, `sequences.connection(…)` and `sequences.connection_tls(…)`. +They do all the steps that are needed (send and receive stanzas) to +connect to the component, or an IRC server. + +It’s also possible to reuse one simple scenario into an other scenario. +The most notable example is to start your own scenario with +`scenarios.simple_channel_join.scenario`, if you need your client to be in +a channel before you can start your actual scenario. For example if you +want to test the behaviour of a topic change, you need to first join a +channel. Since this is a very common patern, it’s simpler to just included +this very basic scenario at the start of your own scenarios, instead of +copy pasting the same thing over and over. + +Examples of a scenario +~~~~~~~~~~~~~~~~~~~~~~ + +First example +^^^^^^^^^^^^^ + +Here we’ll describe how to write your own scenario, from scratch. For this, we will take an existing scenario and explain how it was written, line by line. + +See for example the scenario tests/end_to_end/scenarios/self_ping_on_real_channel.py + +.. code-block:: python + + from scenarios import * + +All the tests should start with this import. It imports the file +tests/end_to_end/scenarios/__init__.py This make all the functions +available (send_stanza, expect_stanza…) available, as well as some very +common scenarios that you often need to re-use. + +.. code-block:: python + + scenario = ( + # … + ) + +This is the only required element of your scenario. This object is a tuple of funcion calls OR other scenarios. + +.. code-block:: python + + scenarios.simple_channel_join.scenario, + +The first line of our scenario is actually including an other existing +scenario. You can find it at +tests/end_to_end/scenarios/simple_channel_join.py As its name shows, it’s +very basic: one client {jid_one}/{resource_one} just joins one room +#foo%{irc_server_one} with the nick {nick_one}. + +Since we want to test the behaviour of a ping to ourself when we are in a +room, we just join this room without repeating everything. + +It is possible to directly insert a scenario inside our scenario without +having to extract all the steps: the test suite is smart enough to detect +that and extract the inner steps automatically. + +.. code-block:: python + + # Send a ping to ourself + send_stanza("<iq type='get' from='{jid_one}/{resource_one}' id='first_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"), + expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='first_ping']"), + +Here we simple send an iq stanza, properly formatted, using the same JIDs +{jid_one}/{resource_one} and #foo%{irc_server_one}/{nick_one} to ping +ourself in the room. We them immediately expect one stanza to be received, +that is the response to our ping. It only contains one single xpath +because everything we need to check can be expressed in one line. + +Note that it is recommended to explain all the steps of your scenario with +comments. This helps understand what is being tested, and why, without +having to analyze all the stanza individually. + +.. code-block:: python + + # Now join the same room, from the same bare JID, behind the same nick + send_stanza("<presence from='{jid_one}/{resource_two}' to='#foo%{irc_server_one}/{nick_one}' />"), + expect_stanza("/presence[@to='{jid_one}/{resource_two}'][@from='#foo%{irc_server_one}/{nick_one}']/muc_user:x/muc_user:item[@affiliation='admin'][@role='moderator']", + "/presence/muc_user:x/muc_user:status[@code='110']"), + + expect_stanza("/message[@from='#foo%{irc_server_one}'][@type='groupchat'][@to='{jid_one}/{resource_two}']/subject[not(text())]"), + +Here we send a presence stanza to join the same channel with an other +resource (note the {resource_two}). As a result, we expect two stanzas: +The first stanza (our self-presence) is checked against two xpaths, and +the second stanza (the empty subject of the room) against only one. + +.. code-block:: python + + # And re-send a self ping + send_stanza("<iq type='get' from='{jid_one}/{resource_one}' id='second_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"), + expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_one}'][@id='second_ping']"), + ## And re-do exactly the same thing, just change the resource initiating the self ping + send_stanza("<iq type='get' from='{jid_one}/{resource_two}' id='third_ping' to='#foo%{irc_server_one}/{nick_one}'><ping xmlns='urn:xmpp:ping' /></iq>"), + expect_stanza("/iq[@from='#foo%{irc_server_one}/{nick_one}'][@type='result'][@to='{jid_one}/{resource_two}'][@id='third_ping']"), + +And finally, we test a second ping, and check that the behaviour is correct that we now have two resources in that channel. + +Second example +^^^^^^^^^^^^^^ + +Sometimes we want to do more with the received stanzas. For example we +need to extract some values from the received stanzas, to reuse them in +future stanzas we send. The most obvious example is iq IDs, that we need +to extract, to reuse them in our response. + +Let’s use for example the tests/end_to_end/scenarios/execute_incomplete_hello_adhoc_command.py scenario: + +.. code-block:: python + + from scenarios import * + + scenario = ( + send_stanza("<iq type='set' id='hello-command1' from='{jid_one}/{resource_one}' to='{biboumi_host}'><command xmlns='http://jabber.org/protocol/commands' node='hello' action='execute' /></iq>"), + expect_stanza("/iq[@type='result']/commands:command[@node='hello'][@sessionid][@status='executing']", + "/iq/commands:command/commands:actions/commands:complete", + after = save_value("sessionid", extract_attribute("/iq[@type='result']/commands:command[@node='hello']", "sessionid")) + ), + +Here is where the magic happens: as an additional argument to the +expect_stanza function, we pass an other function (callback) with the +“after=” keyword argument. This “after” callback gets called once the +expected stanza has been received and validated. Here we use +`save_value(key, value)`. This function just saves a value in our global +values that can be used with “send_stanza”, associated with the given +“key”. For example if you do `save_value("something_important", "blah")` +then you can use `{something_important}` in any future stanza that you +send and it will be replaced with “blah”. + +But this is only useful if we can save some value that we extract from the +stanza. That’s where `extract_attribute(xpath, attribute_name)` comes into +play. As the first argument, you pass an xpath corresponding to one +specific node of the XML that is received, and the second argument is just +the name of the attribute whose value you want. + +Here, we extract the value of the “sessionid=” in the node `<iq +type='result'><commands:command node='hello' sessionid='…' /></iq>`, and +we save that value, globally, with the name “sessionid”. + +.. code-block:: python + + send_stanza("<iq type='set' id='hello-command2' from='{jid_one}/{resource_one}' to='{biboumi_host}'><command xmlns='http://jabber.org/protocol/commands' node='hello' sessionid='{sessionid}' action='complete'><x xmlns='jabber:x:data' type='submit'></x></command></iq>"), + +Here we send a second iq, to continue our ad-hoc command, and we use {sessionid} to indicate that we are continuing the session we started before. diff --git a/doc/index.rst b/doc/index.rst index b97c8fd..e77d898 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -21,3 +21,4 @@ XMPP client as if these channels were XMPP MUCs. admin user contributing + developper |