E2E Testing
Overview
There are currently no requirements from the MDO team as to what this stage needs to do, as long as it passes. The number and scope of tests are up to the application team to decide, but they should cover a reasonable amount of functionality to verify application integrity. However, the CtF review may have certain requirements that must be met prior to CtF authorization.
E2E CtF Requirements
- All projects/pipelines must perform E2E tests.
Per Project Requirements
All pipelines/projects should perform these tests. The Party Bus team understands that this can create confusion about testing back-end vs front-end pipelines. Our current guideline is, at a bare minimum, to define one suite of tests per front-end and then run them in all pipelines. We are working to make this process more efficient and will keep all teams posted on whether this changes.
E2E tests must perform basic functionality tests and also cover any requirements in SD Elements that are not met by other stages of the pipeline. If you are not familiar with SD Elements or have not gone through the CtF Process, please submit a CtF Request to start the process.
All tests are performed on the front-end, but should encompass any back-end functionality as well.
a. Essentially, you are testing your "Application" as a whole and should keep this in mind for your tests. See notes for more on what this looks like in a microservice/pipeline environment.
Party Bus supports Cypress testing by default, but this is not a requirement.
Functionality and Technical Guidelines
Currently, for Cypress tests, we define a location where screenshots/videos from your pipeline will be archived. This location is cypress/screenshots. Please use this as an output directory to be able to look at the results from your E2E tests in a visual format. This will better help you diagnose any issues with your E2E testing in the P1 environment.
E2E testing will need to be performed against your staging environment. We use the variable ZAP_URL to define your staging address. Use this as your baseURL in Cypress.
- If your application parses the P1 auth header/JWT to provide extra levels of authorization/RBAC for your app, then you will need to define a test JWT for E2E testing, and you will need that JWT to be able to authenticate properly. Refer to our SSO, Authentication, and Authorization User Guide for more information.
E2E Gateway
Party Bus uses a separate gateway to bypass Keycloak in this stage. At no point should your app receive any Keycloak errors or redirects back, or be blocked by Keycloak in general. Please open a ticket with the Help Desk if this is an issue.
As stated above, Party Bus supports Cypress testing by default, but if your project requires anything other than Cypress, you will need to create a Help Desk ticket. Note that Party Bus does not provide support for anything other than Cypress and Pytest, so if your application requires another testing method, you will need to have fully fleshed-out tests that are functional in a staging environment, using the guidelines above.
E2E Support
The Party Bus team does not provide guidance on how to define Cypress tests or any other type of E2E testing. It is each product team's responsibility to learn Cypress or any other E2E testing suite and define those tests on their own. We can assist if you are having issues testing from the pipelines, but you will need to provide proof that it is a platform issue, not a code issue.
Cross-IL Setups
E2E testing across ILs is not currently supported. The staging environment must be in the same IL as the source code. In this situation, E2E testing will need to be done locally, and results captured and passed along to Cyber.
Cypress
You will need a script in package.json called test:e2e-ci that defines how the E2E test is invoked. Generally, the script in package.json will look as follows:
"test:e2e-ci": "npx cypress run --browser chrome --config-file cypress/config/pipeline.config.js"Learn more about writing your first end-to-end test.
Customers manage their own Cypress version. In your GitLab repository, you may create a CI/CD variable titled "CYPRESS_VERSION" and set the value to the desired Cypress image tag (i.e., set the value to 14.3.0). The Cypress image used in the pipeline is from Iron Bank and called "cypress-included". You may find the available versions in Iron Bank via Registry1.
Pytest
You will need to set E2E_PYTEST to your E2E entrypoint. We assume pytests/e2e_pytest.py by default.
Data Isolation
After the migration from the use of separate E2E and staging clusters, there is no longer any data isolation for E2E. All data is the same since the E2E and staging clusterS are one and the same.
WARNING
At this time, unsupported databases and their isolation are the responsibility of the product team.
FAQs
How does the JWT work for E2E tests?
Since auth-service and Keycloak are not present in the E2E gateway, requests will hit your app without a JWT. It is recommended that your tests add a mock JWT header.
View an example implementation in our SSO, Authentication, and Authorization User Guide.
What URL should my pipeline E2E tests use?
Your E2E tests in the staging pipelines should use the staging URL of your application.