# Reporting The Magento Functional Testing Framework provides two types of reporting: - Inline reporting that you can view in the terminal as you run [`mftf`][mftf] or [`codecept`][codecept] CLI commands. - HTML reports that you can view using the [Allure Framework][] after a test run completes. When you run a test, MFTF copies all reporting artifacts to the `dev/tests/acceptance/tests/_output` subdirectory in the Magento root directory. The directory contains: - `allure-results/` that is a directory generated and served by the Allure Framework. - `failed` that is a text file containing relative paths to failed tests after the last test run. The paths are relative to `dev/tests/acceptance/`. - `.html` and `.png` files that are screenshots of fails in HTML and PNG formats. To cleanup the `_output/` directory, remove them manually. The `mftf` tool logs output continuously to the `dev/tests/acceptance/mftf.log` file. ## Command line The MFTF reports about its progress during test run when you run the `mftf` CLI tool with [`run:test`][] or [`run:group`][] commands. The report can contain three main parts: - Pre-run checks: - Environment check, such as PHP warnings, etc. - XML test validation like deprecation warnings such as missing required components in XML tests. - Codeception report which is the progress report for each test. - Total results of the test run such as number of tests, assertions, and failures. To manage the level of verbosity, use `-v` or `--verbose` flag in the `mftf` commands. To enable verbosity using the `codecept` commands, refer to the Codeception [Console Commands][codeception]. The following sections demonstrate an example interpretation of a complete log separated into chunks. ### Pre-run check report First, the MFTF reports about issues with environment. In our case, there is an issue with PHP library loading. ```terminal PHP Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/pecl/20160303/php_intl.dll' - dlopen(/usr/local/lib/php/pecl/20160303/php_intl.dll, 9): image not found in Unknown on line 0 ``` Next, the MFTF returns `DEPRECATION` reports alerting you that required test components are missing in XML test definitions. ```terminal DEPRECATION: Test AdminFilteringCategoryProductsUsingScopeSelectorTest is missing required annotations.{"testName":"AdminFilteringCategoryProductsUsingScopeSelectorTest","missingAnnotations":"stories"} DEPRECATION: Test AdminAbleToShipPartiallyInvoicedItemsTest is missing required annotations.{"testName":"AdminAbleToShipPartiallyInvoicedItemsTest","missingAnnotations":"stories"} DEPRECATION: Test AdminRemoveProductWeeeAttributeOptionTest is missing required annotations.{"testName":"AdminRemoveProductWeeeAttributeOptionTest","missingAnnotations":"stories"} Generate Tests Command Run ``` `Generate Tests Command Run` indicates the moment when the MFTF has run the tests generation command actually. ### Test execution report A test execution report is generated by Codeception and includes configuration information, scenario execution steps, and PASSED/FAIL verdict after each test completion. #### General information The general information can be useful for MFTF contributors, but can be ignored by a test writer. Let's consider the general part of the following test execution report: ```terminal ==== Redirecting to Composer-installed version in vendor/codeception ==== Codeception PHP Testing Framework v2.3.9 Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors. Magento\FunctionalTestingFramework.functional Tests (2) ------------------------ Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoRestDriver, PhpBrowser, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTes ``` After the test generation command (mentioned in the previous section), the MFTF delegates control to the `vendor/codeception` tool, which is the `Codeception PHP Testing Framework` of version `2.3.9` that uses `PHPUnit` of version `6.5.13`. The tool runs `2 Tests` using the configuration defined in the `functional` suite under the `Magento\FunctionalTestingFramework` namespace. The corresponding configuration file is `acceptance/tests/functional.suite.yml`. It enables `Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoRestDriver, PhpBrowser, \Magento\FunctionalTestingFramework\Module\MagentoSequence, ...` #### Passed tests The next chunk of the log reports about test execution of the first test: ```terminal AdminLoginTestCest: Admin login test Signature: Magento\AcceptanceTest\_default\Backend\AdminLoginTestCest:AdminLoginTest Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminLoginTestCest.php:AdminLoginTest Scenario -- I am on page "/admin/admin" I fill field "#username","admin" I fill field "#login","123123q" I click ".actions .action-primary" I wait for page load 30 I close admin notification I see in current url "/admin/admin" PASSED ``` The running test is `AdminLoginTestCest`, which is `Admin login test` (this text is generated from the test name but with the `Cest` part excluded). Its test signature is `Magento\AcceptanceTest\_default\Backend\AdminLoginTestCest:AdminLoginTest` that matches a `className:methodName` format using namespaces. A path to the corresponding `Test` is `tests/functional/Magento/FunctionalTest/_generated/default/AdminLoginTestCest.php:AdminLoginTest` (relative to the `acceptance/` directory). `Scenario` lists the tests steps as they run during test execution, ending with the successful test verdict `PASSED`. It means that all test steps were processed as expected. #### Failed tests The second test fails with the following report: ```terminal AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test Signature: Magento\AcceptanceTest\_default\Backend\AdminMenuNavigationWithSecretKeysTestCest:AdminMenuNavigationWithSecretKeysTest Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest Scenario -- I magento cli "config:set admin/security/use_form_key 1" Value was saved. I magento cli "cache:clean config full_page" Cleaned cache types: config full_page I am on page "/admin/admin" I wait for page load I fill field "#username","admin" I fill field "#login","123123q" I click ".actions .action-primary" I wait for page load 30 I close admin notification I click "//li[@id='menu-magento-backend-stores']" I wait for loading mask to disappear I click "#nav li[data-ui-id='menu-magento-config-system-config']" I wait for page load I see current url matches "~\/admin\/system_config\/~" I see "#something" I save screenshot FAIL I magento cli "config:set admin/security/use_form_key 0" Value was saved. I magento cli "cache:clean config full_page" Cleaned cache types: config full_page I am on page "/admin/admin/auth/logout/" -------------------------------------------------------------------------------- ``` The general test details and scenario has the same format as in the Passed test. The interesting part starts near the `FAIL` line. ```terminal I see "#something" I save screenshot FAIL ``` When a test step fails, the MFTF always saves a screenshot of the web page with the failing state immediately after the failure occurs. `I save screenshot` follows the failing test step `I see "#something"` in our case. A screenshot of the fail goes at the `acceptance/tests/_output` directory in both PNG and HTML formats: - `Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.html` - `Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.png` The file name encodes: - `Magento` namespace - with the `AcceptanceTest` test type - generated as a part of the `_default` suite - defined at the `Magento_Backend` module - implemented in the `AdminMenuNavigationWithSecretKeysTestCest` PHP class - with the `AdminMenuNavigationWithSecretKeysTest` test name - and execution status `fail` Actions after `FAIL` are run as a part of the [`after`][] hook of the test. ### Test result report After the MFTF completed test execution, it generates a general report about test results along with detailed information about each fail. ```terminal -------------------------------------------------------------------------------- DEPRECATION: Calling the "Symfony\Component\BrowserKit\Client::getInternalResponse()" method before the "request()" one is deprecated since Symfony 4.1 and will throw an exception in 5.0. /Users/.../magento2ce/vendor/symfony/browser-kit/Client.php:208 Time: 52.43 seconds, Memory: 16.00MB There was 1 failure: --------- ``` First you see warnings and deprecations. The `DEPRECATION` here is thrown by an MFTF dependency (Symfony) that is out of the scope for test writers and should be considered by MFTF contributors. If you encounter this type of reporting, [report an issue][]. Then, the MFTF reports that the test run took 52.43 seconds using 16 MB of system RAM. And, finally, that there was `1 failure`. Next, the report provides details about the test failure. ```terminal --------- 1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test Test tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest Step See "#something" Fail Failed asserting that on page /admin/admin/system_config/index/key/678b7ba922c.../ --> DASHBOARD SALES CATALOG CUSTOMERS MARKETING CONTENT REPORTS STORES SYSTEM FIND PARTNERS & EXTENSIONS Configuration admin 1 Store View: Default Config What is this? Save Config Country Options State Options Locale Options Store Information Store Name Store Phone Number Store Hours of Operation Countr [Content too long to display. See complete response in '/Users/dmytroshevtsov/Projects/vagrant/vagrant-magento/magento2ce/dev/tests/acceptance/tests/_output/' directory] --> contains "#something". Scenario Steps: 23. $I->amOnPage("/admin/admin/auth/logout/") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:54 22. // Cleaned cache types: config full_page 21. $I->magentoCLI("cache:clean config full_page") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:52 20. // Value was saved. 19. $I->magentoCLI("config:set admin/security/use_form_key 0") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:50 18. $I->saveScreenshot() at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:63 ``` - `1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test` - the failed Codeception test is *AdminMenuNavigationWithSecretKeysTestCest*. It references to the PHP class that implemented the failed test. - `Test tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest` - the test is implemented in the *AdminMenuNavigationWithSecretKeysTest* test method of the *tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php* file under `/dev/tests/acceptance/`. It matches the corresponding test defined in XML that is *AdminMenuNavigationWithSecretKeysTest* defined in `...` - `Step See "#something"` - the failing test step is the *see* action with the *#something* selector. It would correspond the `` test step in the XML defined tests. - `Fail Failed asserting that on page /admin/admin/system_config/index/key/678b7ba922c.../` - the fail occurred on the web page `/admin/admin/system_config/index/key/678b7ba922c.../`. ```terminal --> ... [Content too long to display. See complete response in '/../../magento2/dev/tests/acceptance/tests/_output/' directory] --> contains "#something". ``` The web page is too long to be reported in the CLI, and it is stored at *'/../../magento2/dev/tests/acceptance/tests/_output/'*. Search the web page by test name *AdminMenuNavigationWithSecretKeysTest*. The failing test assertion is that the web page contains *contains* a CSS locator *#something*. Finally, the report finishes with fairly self-descriptive lines. ```terminal FAILURES! Tests: 2, Assertions: 3, Failures: 1. ``` The MFTF encountered failures due to the last test run, that included *2* tests with *3* assertions. *1* assertion fails. ## Allure Each time you run tests, the MFTF appends an XML file with results at the `tests/_output/allure-results/` directory. The official [Allure Test Report][] documentation is well-covered, so we'll list only the CLI commands that you would need for your day-to-day work.
The following commands are relative to the Magento installation directory.
To generate a report to the `allure-report/` at the current directory: ```bash allure generate dev/tests/acceptance/tests/_output/allure-result ``` To generate a report to a particular directory, use the `-o` option: ```bash allure generate dev/tests/acceptance/tests/_output/allure-result -o dev/tests/acceptance/tests/_output/allure-report ``` To launch the generated report in a web browser: ```bash allure open dev/tests/acceptance/tests/_output/allure-report ```
By default, Allure generates reports in the `allure-report/` at the current directory. For example, if you run the command without `-o` flag while you are in the `magento2/` directory, Allure will generate a report at the `magento2/allure-report/` directory.
```bash allure generate dev/tests/acceptance/tests/_output/allure-result ``` Example of file structure after the command run: ```terminal magento2 ├── allure-report ├── app ├── bin ├── dev ├── ... ``` And if you run the `open` command with no arguments while you are in the same directory (`magento2/`): ```bash allure open ``` Allure would attempt to open a generated report at the `magento2/allure-report/` directory.' %} To clean up existing reports before generation (for example after getting new results), use the `--clean` flag: ```bash allure generate dev/tests/acceptance/tests/_output/allure-result --clean ``` To generate the HTML Allure report in a temporary folder and open the report in your default web browser: ```bash allure serve dev/tests/acceptance/tests/_output/allure-results/ ``` Refer to the [Reporting section][] for more Allure CLI details. [`after`]: test.md#after-tag [`run:group`]: commands/mftf.md#rungroup [`run:test`]: commands/mftf.md#runtest [Allure Framework]: https://docs.qameta.io/allure/ [Allure Test Report]: http://allure.qatools.ru/ [codecept]: commands/codeception.md [codeception]: https://codeception.com/docs/reference/Commands [mftf]: commands/mftf.md [report an issue]: https://github.com/magento/magento2-functional-testing-framework/blob/master/.github/CONTRIBUTING.md#report-an-issue [Reporting section]: https://docs.qameta.io/allure/#_reporting