MDO Announcements - 2021
17 Nov 2021: Pipeline Changes - Compliance Check
Hi everybody, we have renamed the "Dockerfile Scan" stage of the pipeline to "Compliance Check" and we have added a new job under it called project-configs.
This job will verify that your repository's Gitlab CI/CD configuration is set up according to our project settings requirements. Additionally in the coming weeks we will be auditing all repositories in Party Bus Gitlab to verify that pipelines are configured according to these requirements.
A couple notes about our CI/CD requirements:
.gitlab-ci.yml files are not permitted in repositories in Party Bus All pipelines in Party Bus Gitlab must be managed by the MDO Team.
Relevant documentation:
16 Nov 2021: Introducing Gradle Lint Support
Hello everyone, Gradle pipelines now have a lint job. This job works by running a Gradle task ./gradlew lintGradle. This job will be allowed to fail at a global level for two weeks as we test it out.
We have documented this in these two articles:
25 Oct 2021: Docker Image Requirements Changes
Hello everyone! We are making a change to the "Find Unauthorized" job under the "Dockerfile Scan" stage of the pipeline.
This change adds a new enforcement check to verify that your Dockerfiles are only using approved base images:
- Images in the pipeline-templates repository container registry
- Ironbank
If you are not using images from one of these two sources please update your Dockerfile to reference them.
We have documented these requirements in the following places:
As a final note - we are not enforcing this job at this time and it is allowed to fail. In the future this job will not be allowed to fail; we will provide at least two weeks of notice before that time.
28 Sep 2021: Twistlock Scan Changes
Hi all, we are slightly changing the way the Twistlock scan stage of the pipeline works:
- Exceptions for CVE findings in base images will now be granted automatically
- CVE findings should now only occur for items added to the image in your Dockerfile
What this means for you:
- If you see Twistlock findings and they are NOT from your app, try using a newer image
- If you see Twistlock findings from your app and you believe they are false-positives, please open a help desk ticket.
Relevant links:
16 Sep 2021: Python Template Changes
Swapping Pipeline Image
We are switching out the pipeline image. When this happens you will need to clear runner caches, (CI/CD > Pipelines > Clear Runner Caches).
- From registry.il2.dso.mil/platform-one/devops/pipeline-templates/ubi8-python3.8:8.1.
- To an image based on registry1.dso.mil/ironbank/opensource/python/python39:v3.9.6.
Dependency Check
We changed how python dependencies are scanned in dependency check. Instead of scanning your python packages directory, we will scan a generated requirements.txt that will include all dependencies (main + transient) with versions specified for each. We have found that this method yields less false-positives and more accurate findings. The generated requirements.txt will be created in the build job and saved into artifacts for you to download if you want.
Lint
We upgraded pylint to 2.10.2. With that, a new feature has been added that allows you to ignore directories. E.g. ignore-paths=^./.cache/.*$ would ignore .cache in scans using a .pylintrc or a switch.
- Note, in the near future we plan to create new python template using pipenv. With that change, we plan to stop using pylint environment variables like PYLINT_EXCLUSIONS and PYLINT_EXTRA_SWITCHES. The new pipenv template will follow the npm template and let you choose your linter. If plan to keep using pylint, I suggest switching to using a .pylintrc as soon as possible and move away from the environment variables.
Unit Tests
We introduced a UNITTEST_EXCLUSIONS environment variable to exclude certain directories from running tests. By default, this is set to .cache/,/usr/lib/_,test*.py.
- The pipenv changes that were mentioned for lint also apply to unit tests.
Build
The build job will only run if there are changes to your requirements.txt (change in your dependencies) similar to the npm template.
We plan to merge this change on Friday EOD. I plan to announce a reminder beforehand.
03 Sep 2021: Build Job and Cache
If you do not see the build job when you trigger a pipeline it is because build is meant to download your dependencies and store them in cache to be used for subsequent pipeline runs. The cache does not expire. However, it can be cleared manually through CI/CD > Pipelines > Clear Runner Caches. If it is cleared you will run into issues in the pipeline if build does not run again. The build job triggers only when there are changes to your package-lock.json or if the pipeline is triggered manually through CI/CD > Pipelines > Run Pipeline. TL;DR: if you clear the cache, run the pipeline manually to rebuild your dependency cache; CI/CD > Pipelines > Clear Runner Caches then CI/CD > Pipelines > Run Pipeline.
This is only applicable to the npm and yarn template, but will eventually reflect all pipelines.
19 Aug 2021: Nodejs Pipeline Image Update
TL;DR
Platform One is changing the Nodejs docker image used in the pipeline's build, lint, and unit test jobs from Docker Hub's node:slim image to Iron Bank's Nodejs16 image.
When will this Occur? MON 23 AUG 2021
What to Expect? Platform One has successfully tested the new image on multiple projects and we do not expect there to be a lot of fallout from the change. However, if you do experience an issue please create a Jira Service Desk ticket and we'll take care of it as quickly as we can.
Who is Affected? This change is for any node projects built with either npm or yarn.
Why is this Happening? Platform One is committed to constantly improving its security posture. That is the motivation behind the switch from the public Docker Hub image to one from the Iron Bank, which manages a repository of digitally signed, binary container images that have been hardened and accredited for DoD-wide use across classifications.
05 Aug 2021: Dockerfile Linting Change Justification
Hey everyone, I wanted to add a note in here to address our reasoning behind the recent Dockerfile pipeline changes.
We added the "dockerfile lint" job for the following reasons:
Our C-ATO requires that product teams use Ironbank images. These images have been vetted for vulnerabilities and hardened by the IB team to reduce potential attack surface. Enforcing Dockerfile best practices leads to more efficient use of P1 resources and reduces chances of bugs. Read more here: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
We added the "find unauthorized" job for these reasons:
Pipelines already compile code and download dependencies during the build stage, reusing these artifacts saves runner resources and time and will speed up pipelines across Party Bus. Downloading and installing external packages breaks the hardnening work done by Ironbank, potentially exposing the cluster to new vulnerabilities. Code and dependencies are scanned for security during the SCA stage of the pipeline. Reinstalling or rebuilding these during the Docker build means we can no longer guarantee the code and artifacts have been scanned. While we scan the images with Twistlock, it is not a static code analysis tool. Identifying unauthorized commands early allows teams time to update their Dockerfiles before the CTF process. 02 Aug 2021: New "Find Unauthorized" Pipeline Job
We are introducing a new Dockerfile Scan stage into the pipeline which will have the dockerfile lint job that is currently in .pre and a new find unauthorized job.
Allow failure has been removed from the dockerfile lint job as of this morning. This runs hadolint and will set to fail for issues that are a warning or error. It will also check if you are pulling images from an approved registry. Approved registries are:
*registry.il2.dso.mil *registry.il4.dso.mil *registry.il5.dso.mil
The new pipeline job, find unauthorized will scan your Dockerfile for any unauthorized commands. If it finds any of the commands or keywords listed within a command it will flag it as an issue. We will allow failure for now to give you all a chance to address any issues found before we start to enforce it.
Please note that the job will fail if the Dockerfile is not in the root of your repo.
Please visit Pipeline Basics - Find Unauthorized Stage for the list of commands/keywords, how to resolve these type of issues, and instructions on how to run this job locally. 
29 Jul 2021: Hadolint Guidance
The hadolint warning warning: Last USER should not be root is a valid finding. You can resolve the warning by switching back from root at the end of every stage. There is a note about this in the - Pipeline Basics. Also a reminder that this stage will be enforced starting Monday.
22 Jul 2021: Notice of Docker Lint Stage Enforcement
Starting on August 1st we will be enforcing the Pipeline Basics - Docker Lint Stageof the pipeline. This means that we will not allow failures in this stage for all applications in all impact levels, including both staging-only apps and production apps.
This stage currently is set to enforce that your Dockerfiles only pull from the Project Settings Requirements as well as other basic Dockerfile best practices.
What this means for you:
- If your application is currently passing the Docker lint stage and uses approved images, there are no changes for you.
- If you are using an unapproved image, you will need to update your Dockerfile to use an approved image (review the Project Settings Requirements in Preparing for the Pipeline Queue).
- If you have previously been given an exception to use a different image than in the approved list, please request an exception for this enforcement through a help desk ticket .
16 Jul 2021: IL2 E2E Stage Functional
The E2E stage is now functional in IL2 for apps that run cypress tests. There is also new functionality in the Deploy Staging pipeline stage such that the job will wait until ArgoCD has deployed the new version of your app before completing. This will ensure the E2E stage runs tests against the new version of your app.
What this means for you:
- If your pipelines and staging application are in IL2 and your repository has cypress tests you can expect these to now run correctly.
- If your app uses a different e2e testing framework than cypress, please submit a feature request .
- If your pipelines are in IL2 but your staging application is in IL4 or above, E2E tests will not work for you. Let us know and we can disable the argo wait condition for your app.
- If you are not in IL2, there are no changes for you yet. We are still working on IL4 and IL5 implementations.
If you have any questions or concerns feel free to ask in this channel or through a help desk ticket .