User workflow

Overview

The diagram below shows what happens in various phases of a CATcher-assisted .

Here is a brief explanation of each phase:

  1. Bug reporting phase
    1. Student testers test the software they have been assigned and report bugs through CATcher. These bugs are stored in a repo in the tester's GitHub account. e.g., tester/pe.
    2. After the testing session is over, a script S1a transfers the bugs to an interim private repository (e.g., module-org/pe-interim), and another script S1b transfers the bugs from the interim repo to a central repo e.g., module-org/pe-dev-response.
      Bugs are channelled through an interim repo to hide the identity of the tester.
      Scripts used to transfer/process bugs between phases are not part of CATcher codebase at the moment. They are managed separately by teaching staff members.

During the PE dry run (which has only one phase and the anonymity is not required), the script S1c transfers bugs directly to the receiving team's issue tracker.

  1. Dev response phase
    1. Dev teams (i.e., teams that developed the target products) use CATcher to respond to bugs they received.
    2. At the end, a script S2 transfers dev team's response back to the original bug reports in the tester's GitHub account e.g., tester/pe.
  2. Tester response phase
    1. Testers use CATcher to provide their own response to the dev team's response.
    2. At the end, a script S3 transfers the tester's response to another private repository e.g., module-org/pe-moderation that tutors can access.
  3. Moderation phase
    1. Tutors give their own inputs on bugs that caused disputes between the tester and the dev team.
    2. At the end, a script S4 generates a final report and marks.

The settings repo: CATcher behavior for the above phases are dictated by config data stored in a public repository we call the settings repo e.g., module-org/pe.

More detail of the things mentioned above are given in the sections below.

The settings repo

  • Each PE must have its own settings repo, which is a public repository containing config data that govern the behavior of CATcher e.g., which phase is open, which students are taking part, names used for various repos etc.
    For example, the module instructor can open a phase (i.e., allow students to do that phase through CATcher) by modifying the settings.json file in the settings repo.
  • An example https://github.com/nus-cs2103-AY2021S2/pe
  • The settings repo is also used to hold files uploaded by students (e.g., screenshots). CATcher initially uploads those files to the same repo that holds the bug. But our scripts copy those files to the settings repo so that the identity of the uploader cannot be discovered using the file URL.

Bug Reporting Phase

[CATcher] Collect Bug Reports

  • Title and body as the tester entered.
  • Labels: severity.*, type.* (both compulsory)

[Script S1a] Tester-Repo → Interim-Repo

Issue title remains the same as the original issue, throughout the workflow

Formats used

Notes:

  • Only open issues are to be transferred.
  • Copy the severity.* label. If no severity label, apply severity.Low
  • Copy the type.* label.
  • Transfer image files to the settings repo (public) and update the link in the issue body.
  • Check the timestamp. Only bugs reported within the PE period should be transferred. Add a comment to issues falling outside the accepted time window.

    Bug report not accepted as it was modified outside the time window {start time}-{end time}

[Script S1b] Interim-Repo → Dev-Repo

Formats used

Notes:

  • Copy all labels
  • Apply the correct tutorial.* label and team.* label to indicate the receiving team

[Script S1c] In the dry run: Tester-Repo → Team-Repo

  • Transfer the bug report to the respective team repo.
  • Labels are to be omitted (as the script doesn't have permission to add labels in team's repo) but mentioned in the body of the issue (for reference).
  • Image links can be kept as they are, as tester anonymity is not required.

Dev Response Phase

[CATcher] Collect Dev Response

Formats used

  • Labels: severity.*, type.*, response.*, duplicate

[Script S2] Dev-Repo → Tester-Repo

Add a comment to the original issue in tester/repo-name, in the following format:

Formats used

Notes:

  • If the issue is a duplicate, it should be considered as using severity, type, and response from the "original" issue.
  • If the response.* is missing, add response.Accepted to the destination issue (not the source issue).
  • If the dev response contains images, they need to be transferred to the settings repo as well

Tester Response Phase

[CATcher] Collect Tester Response

Update the comment.

Formats used

[Script S3] Tester-Repo → Tutor-Repo

Formats used

  • Labels to be copied from the dev-response repo: type, severity, response, tutorial, team
  • The script will also generate a label based on the number of Disputes (e.g pending.2 for issue with two disputes)

Moderation Phase

This phase is not currently handled by CATcher. Instead, tutors do this phase by editing issues through the GitHub Web interface directly.

[CATcher] Collect Tutor Response

  • Show ticks for each dispute
  • Add a comment to record tutor response

Formats used

  • Allow tutor to change other labels
  • The pending.x label will be removed if all tasks are done
  • Allow adding an Unsure label, in case the tutor is unsure about the decision