Testing

This page contains information useful for testing of CATcher.

---

Dev commands

Given below are different commands you can use to run the tests.

---

Setting up custom CATcher sessions

Sometimes, it may be useful to create and use your own custom CATcher session for manual testing, instead of using the default session on CATcher-org that is shared among the CATcher developers.

Setting up GitHub repositories

A repository containing the settings for the custom session must be set up for CATcher to store and retrieve the issues properly. The easiest way to do this is to fork the CATcher-org/public_data repository and to edit the settings.json and data.csv files accordingly.

Populating issues in CATcher repositories

In CATcher, submitting a bug in the Bug Reporting phase will create a GitHub issue in the corresponding repository with the appropriate format and labels. For subsequent phases, the responses are stored as comments of that particular issue, and these comments must be of a certain format for CATcher to parse them correctly.

Loading the custom session

The custom session will not be present in the default dropdown list on CATcher's startup page, so it can be loaded by clicking on the file icon beside the session dropdown. Following which, submit a file with the .json file extension, where the format is specified below.

---

Testing with Jasmine

Jasmine is a behavior-driven development framework specific for testing JavaScript code. We follow the Jasmine Style Guide loosely for our tests (Link under Resources for Jasmine). One main guideline is that a describe block should be created for each method / scenario under test, and an it block should be created for each property being verified.

Resources:

1. Jasmine Style Guide
2. Official Jasmine documentation : This is the official Jasmine documentation for Jasmine 3.6
3. Introduction to Jasmine 2.0 : This is a good summary / introduction of Jasmine test features

---

Angular testBed utility

Because the above Jasmine framework does not test the DOM, we require the Angular TestBed Utility functions to set up component tests for testing HTML / view changes of components in CATcher.

Steps to set up component tests:

1. Configure the testing module through TestBed function configureTestingModule with the corresponding component's settings
2. Use TestBed function to create the component (fixture) to be tested
3. Observe HTML changes in the fixture during testing of functions by querying HTML elements of the fixture

You can refer to the AssigneeComponent test under our main repository for more details on how to set up a component test in CATcher.

Resources:

1. Angular Guide - Basics of testing components : Official Angular developer guide for the basics of component tests
2. Angular Guide - Component Fixture : Official Angular developer guide on ComponentFixture
3. Introduction to Unit Testing in Angular : Useful article on how to test component fixtures

---

E2E testing

Running E2E tests

can be executed using npm run e2e. You should see CATcher launch in an instance of Chromium, with some automated actions occurring on it. Note: Google Chrome needs to be installed on the machine. You can also run e2e tests against Firefox,Chromium,Webkit in headless mode if you run npx playwright test

Alternatively, it is highly recommended that you install Playwright Test for VSCode, which would allow you to run specific tests, utilize the pick locator when writing tests, etc.

E2E tests simulate how an user will interact with our application. However, to avoid hitting the GitHub API in our tests, we currently do not perform E2E tests on the production version of CATcher. Instead, a test version of our application where mock data is used to simulate the GitHub API is used in our E2E tests. Under the hood, Playwright first activates this test version of our server using npm run ng:serve:test before running tests.

To cut down the time running Playwright tests, you can first start the testing version manually using npm run ng:serve:test. Subsequent runs of Playwright will see that a test server is live and run the E2E tests again them. You can also run npm run ng:serve:test to further develop or debug the E2E tests.

Playwright Configuration

E2E Tests are run using Playwright testing framework.

• Playwright primarily requires the playwright.config.ts file located at the project root to define E2E Testing environments. This includes the list of browsers (or "projects" as they call it), base URL, number of workers etc.
• E2E Tests are typically split into Page-Objects Files and Test Files in accordance with the Page Object Model
• E2E Tests are also grouped into files based on the Application's Phase (i.e. Login, Bug-Reporting).

How the E2E tests work

E2E Tests are run with the following stages:

1. Build CATcher using test architecture
   • Using test build configuration located in angular.json under projects.catcher.architect.configurations we build a version of CATcher within a test environment that replaces src/environments/environment.ts with src/environments/environment.test.ts on runtime. This file provides data that allows CATcher to switch into "E2E test" mode.
2. Provide Test Environment Information
   • The Test Environment (in src/environments/environment.test.ts) provides information such as,
     • Login Credentials (Username).
     • User Role and Team Information.
     • A test flag that is set to true, so that CATcher switches into "E2E test mode"
3. Mock Service Injections
   • Once CATcher switches to E2E test mode, it creates mocks of some services, in order to simulate behaviour that is outside the scope of E2E Testing. This includes authentication, and communication with GitHub (via its APIs).
   • These Service Injections are carried out in the respective *-module.ts files with the help of Factories (located in /src/app/core/services/factories) that check the current build environment and make the Service Replacements accordingly.
4. Browser Action Automation using Playwright
   • With the application ready for testing, we then utilize Playwright to run test cases that are located in the /e2e directory.