Party Bus Frequently Asked Questions

Repos

1. How much access does the product team get to the repo?

Your organization will make the determination as to how access is given for a repo/project. Each team is designated a PM/Onboarding supervisor who can onboard and set role levels as needed.

2. Is there any documentation for importing an existing project?

We do not have P1-specific instructions. Read the official GitLab instructions on migrating GitLab data using file exports.

3. How can product teams set up repo mirroring?

We do not allow repo mirroring from P1-external sources. We have security controls, like signed commits, so that only approved users can add code. The only scenario where mirroring is allowed is from repo1 to Party Bus IL2, in which case all mirroring still has to be configured by a GitLab Admin, which is Party Bus personnel only.

4. How to invite members to a GitLab group?

Users must already be registered users in GitLab. E-mail notifications are disabled. The OS was designated as an Owner. They will be able to manage membership of the group.

5. Are there any app size limits that repos can host?

We do not have a hard limit, but we do monitor pipelines for excessive resource usage and may shut down pipelines/repos if so. If a repo has many files/lines of code to process, some jobs may struggle to complete successfully within job timeout periods. We recommend breaking large repos into multiple microservices.

Above baseline resource requirements may require additional funding to support. Repos support GitLab LFS, and large files can be committed.

Pipelines

Review our Pipeline Basics article to better understand how our pipelines work.

1. What logs are available to review?

You can review the GitLab CI/CD pipeline log output as well as Argo CD logs (if access has been granted).

2. Where does code reside when deploying to IL6?

If you deployed your staging environment to IL2/IL4/IL5, your code base will remain there while the IL6/AFSCI deployment is passed over to the proper POC's at DAF Cloudworks.

3. What e2e frameworks are supported?

We primarily use Cypress, but can invoke integration tests for pytest as well. Please open a ticket for any other E2E framework support.

4. Can we run a parallel test in Cypress?

Yes, invoke cypress-parallel.

5. Can we have a development deployment environment or another environment before staging?

For most use cases, we only provide two deployment environments: staging and production (which is only available after the CtF has been obtained). However, recently we have introduced the concept of Auxiliary staging and production deployments, which can be set up as variations of your existing deployments. Note that there are important considerations when it comes to pricing and resources for auxiliary deployments.

6. We have a repo using multiple languages. Is there any support for that?

No, our pipelines are technology/language-specific; the repos need to be separated.

7. Is there any chance to support "new" languages?

While everything is in the realm of technical possibility, we must take into consideration resources to develop support for new languages. Additionally, our scanning tools do not support languages such as Fortran, COBOL, etc, so we would be unable to check for any vulnerabilities. If a new language/package manager is desired, you can submit a feature request to the help desk , but if there is low demand for a specific technology, we do not have the resources to design, implement, and maintain the pipelines needed to support it.

8. Can we have repos that are dependent on other custom libraries?

Yes, we can set up package pipelines, which will result in an artifact that can then be imported into another repo/project. You must be able to import the code for said libraries into another GitLab project and utilize the Party Bus package pipelines to build them. You will not be able to import compiled packages unless you are given cyber approval to do so. Considerations will need to be made for funding another pipeline.

9. What is the typical turnaround to create a pipeline?

It varies widely from product team to product team. Other teams also need to be involved, e.g., CNAP, DNS, Onboarding. Once an MDO engineer grabs your pipeline out of the queue, it can take anywhere from 1 day to 2 weeks, depending on how compliant it is with our standards. Updates to the repo to address SCA findings or set up unit testing are on your product team, but can be worked on your own timeline, but will block progress until they are resolved.

12. My anchore/twistlock scans seem to be failing because of Red Hat/operating system-level vulnerabilities

This is common due to base image issues and whitelisting. Vulnerabilities that are "inherited" from the Iron Bank base images are automatically whitelisted; this only applies to tags that have been built more recently. Please ensure you are using the most recent tag for your image, which can be found at the Iron Bank repo. If you are still having issues, re-open a ticket. Additionally, please refer to the Docker Renovate Guide for more information on how to utilize Renovate in order to keep your base image up-to-date.

13. Can someone on the Party Bus/MDO team assist with Cypress/e2e testing?

We cannot help you design your E2E tests or troubleshoot any Cypress/E2E code, but we do ensure that your E2E test is functional from a platform perspective (i.e, we make sure you can reach your staging deployment from the pipeline and it returns a non-4xx/5xx HTTP code). Read more information on end-to-end testing.

14. What are some common whitelists that are granted?

• Vulnerabilities for parsing and executing untrusted data
• Transient dependencies
• Misattributions to other dependencies
  • Dependency Check uses fuzzy string matching and will sometimes associate it with similarly-named dependencies

15. What recourse does the product team have for vulnerabilities that cannot be resolved?

Any whitelist requests require justification. If we deem the justification insufficient or the risk too large, we may decline to whitelist the vulnerability.

16. Is it possible to downgrade the severity of SonarQube findings?

This is typically not offered. Depending on the number of requests, it may be granted.

17. What type of tests are required for the pipelines?

We require unit tests, E2E tests, and penetration tests. The product team is responsible for developing its own unit and E2E tests. Penetration tests are performed within the pipeline and by the CAT.

18. My pipeline job is running out of resources. Can I get more memory and/or CPU?

All pipeline jobs are given 10Gi of memory and 4 dedicated vCPU cores maximum. We do not have the ability to increase memory on a per-job basis. If this is not sufficient, you will have to work with CST and PB Ops to allocate a dedicated node to run high-resource jobs. Please email the BAMS at AFLCMC.HNCX.BAM@us.af.mil to start this process.

19. My unit tests are taking an abnormal amount of time to finish, especially when compared to when I run them locally. What's going on?

We have seen, especially in the case of jest/npm, that the underlying code attempts to utilize every CPU possible when running. This is mostly fine; however, our GitLab CI pipelines run on Kubernetes jobs, and the CPU resource counts are abstracted away, so Jest ends up reading the underlying EC2 Kubernetes node resources, which are significantly higher, thereby causing long response times. We have found that adding "–maxWorkers=4" to your unit test scripts results in some reduced test time.

20. Why is my namespace-health-check job failing?

There are many answers to this question, but the best approach is to think of this job as more of an "e2e/pen test readiness" stage. The deployment for the microservice you are pushing out may have actually succeeded, but there could be other microservices failing in your environment. In most cases, you will need to check Argo CD to see what this issue is.

21. Do pipelines support multiple branches?

Only the default branch (initially master) is allowed to deploy or generate a package. Any other "dev" branches will run the pipeline up to (but not including) the deploy-staging/publish-package stage.

22. Can product teams use tags to manage releases?

Product teams cannot tag releases directly. The pipeline release-xxx stage automates the process and manages the release tags in correlation with the CtF version. This feature is enabled after receiving a CtF.

23. How can we exclude certain files from deployment?

We scan all code files and persist them as pipeline artifacts. Any of those files can selectively be deployed to the Docker image.

24. We use a DAF Cloudworks version of GitLab on NIPR. Can we use that?

No, product teams are required to use our instance of GitLab.

25. If a product team is onboarded without pipeline support (GitLab seats only), are they able to utilize the pipeline-templates?

The pipeline templates repo is publicly available, so they can use it as a reference. However, we do not allow product teams to run pipelines if they have not purchased the support to do so.

26. My Twistlock scans are coming back with findings that start with PRISMA-. What does this mean?

PRISMA-* findings are those that Palo Alto detects but do not yet (or ever) have a CVE ID. However, they can still be valid findings. Please address these as you would any other finding or create a whitelisting ticket if you feel they are false positives. Read more information on Prisma Cloud.

Containers/Deployments

1. Is Docker Compose supported?

No, we use the Kubernetes container orchestration framework.

2. Do product teams need to set up pipelines for common container services like Kafka, Redis, etc.?

No, pipelines do not need to be set up. Containers using Iron Bank images can be added easily if no special configuration or additional packages/code is required.

3. Are Helm charts supported?

We use Kustomize to manage Kubernetes manifests, and while Argo CD may support Helm in some capacity, Party Bus does not support this means of deployment.

4. What is Argo CD?

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes, which will allow you to monitor your application and its Kubernetes configuration. Permitted users will be able to view the standard output for deployments and restart pods. For access to Argo CD, please have your OS submit a Add/Remove Argo CD Access request to the help desk .

5. Are generated logs accessible through Argo CD?

No, Argo CD will display stdout/stderr from your deployments, but output files cannot be accessed directly through it. Any output files can be persisted into S3 (if purchased).

6. Is there a way to get extended logging for Argo CD? There appear to be issues getting the previous logs.

Argo CD will display the output from the currently running containers (and the previous if a crash occurred at the container level). If the pod resets, the logs are gone; this is a limitation of Kubernetes.

This is a common request, but it is not something we have been able to implement yet.

7. Does P1 offer W3C log (or other) data aggregation/monitoring?

There is currently no service available as we are trying to implement a multi-tenant solution with the proper boundaries between customers.

8. My deployment is failing/I can't reach my application, and I am getting "connection reset"/503 errors. What is wrong?

For any deployment issues or web page problems, Argo CD is going to be your best friend. Below are some common error codes and what they mean.

Code/Response	Issue	Response
503	Microservice is crashing due to various issues	Check Argo for any problems in your logs. If logs are not indicating issues, check the status tab for your pod in Argo and look for the status OOMKilled. This means that the pod does not have enough memory. You can provision more, up to 2G per pod. If all else fails, please open a ticket with the help desk.
301	Redirect	Check your application for redirects. Additionally, if you are trying to reach the external address application from a non-browser-based session, you will get redirected to Keycloak, which will dead-end your session.
RBAC: Access Denied	Virtual Service missing or authconfig missing	Please open a ticket

9. Does Party Bus Kubernetes support node affinity for containers that need persistent storage, e.g., MongoDB containers?

Yes, product teams are allowed to make updates to the app-manifest (which contains their Kubernetes configuration) as long as it does not violate any Party Bus requirements. Read our Manifests guide to learn more.

10 Is there any way to dynamically spin up resources as workers in the cluster?

This would be configured in the app-manifest. Product teams are allowed to make updates to deploy pods and have them autoscale.

11. Are pods/clusters restarted periodically?

Yes, Party Bus ingests Big Bang, which will sometimes necessitate cycling pods to deploy upgrades, e.g., to Istio. Upgrades are enforced to stay in parity with our security posture.

12. Is there any way to deploy components utilizing unsupported technologies? Can ATO still be achieved?

Since we cannot scan repos utilizing the unsupported technologies list, we cannot set up pipelines for them. The alternative is to have a vendor, or someone, sponsor that image into Iron Bank.

13. Does P1 support AWS QuickSight?

Apache Superset is in Iron Bank, but AWS QuickSight is not.

14. What are the resource limitations?

There is no hard cutoff. We allow product teams to configure resource allocations for their containers, but it should be within reason. In the multi-tenant cluster (t3.medium to t3.2xl), that's usually about 100 Gigs of storage, .5 Gigs of memory, and .25 CPU (peak). For single-tenant solutions, limits are determined by AWS and what you pay for.

15. Is it possible to deploy multiple apps on a container in Kubernetes?

Each app is its own cluster. Each repo/project should be a microservice that results in a single deployment container. Traffic between pods within an app/cluster is open.

16. What is the difference between the Iron Bank Catalog and Harbor?

The Iron Bank Catalog and Harbor are different views of the same Docker registry. Harbor is the open-source implementation, whereas the Iron Bank Catalog is the P1-built solution (which we recommend over Harbor).

17. What is the difference between Kubernetes nodes and pods?

A node is a worker machine (can be physical or virtual) in Kubernetes that serves as a computing resource.

A pod is the smallest deployable unit in Kubernetes and represents one or more containers that are tightly coupled and share certain resources.

18. How are IPs for the apps set?

Statically. We use Amazon Elastic IPs, which forward to Elastic load balancers in our account, which then forward to Istio gateways, which listen on NodePorts to then route traffic to pods.

19. Is it possible for an app at a higher IL to send data to an app in a lower IL?

It is technically feasible, but not approved by Cyber at this time.

20. How do I request new Iron Bank images?

Review How To Pull an Iron Bank Image for guidance on how to pull an image.

Using New Iron Bank–Approved Images: A new image is an image that does not exist in the Iron Bank hardened images repository. The Iron Bank team is the approval authority for custom images.

21. Could a team develop at IL4 staging and deploy their application to IL2 production?

Staging deployments must be done at the same IL where the code resides, and production deployments can be in the same IL or a higher IL.

22. Can I or should I manage my own HTTPS/TLS within my application?

Party Bus utilizes Istio for all things TLS. This means that any HTTPS you require for serving out your application will be done utilizing Istio ingress gateways and virtual services. Additionally, Istio does mTLS encapsulation for traffic, so connections within the cluster or microservice-to-microservice connections, although they may be configured as cleartext/HTTP, will be protected by mTLS. Therefore, there is generally no need to generate or manage certificates, unless there are very specific cases where this is necessary.

Databases

1. What tools are provided for managing a team's database?

Currently, Party Bus teams have the option of managing Postgres through the use of PGAdmin. Read our pgadmin documentation for more information.

2. Can product teams initialize their own databases and create tables, users, etc?

Use tools like Flyway that contain migration scripts, creation scripts, etc. Databases are initialized at the deploy stage with your own schema scripts.

3. How is database migration handled?

Product teams should handle database migration through their application or app-manifest, but a help desk ticket can be opened if an administrator is needed to dump or seed a database.

4. What is the maintenance schedule for databases?

There are automated daily backups for the last 7 days. Additional manual snapshots are taken at critical points.

5. Can I connect to my RDS from my pipeline?

We do not allow connections from GitLab to the RDS due to the separation of boundaries between the clusters/environments. If you need to do any database operations during unit testing or E2E testing, you can utilize GitLab Services (ticket needed). Otherwise, you can programmatically add database procedures in your deployed application that can be accessed via api calls or otherwise. We will also note that in some cases, doing database connections during a unit test can be considered an anti-pattern and also require extra effort from the MDO team to configure and help troubleshoot them. Therefore, they are highly discouraged.

Checkstyle

1. Are check styles required for lint?

Yes.

2. Can check styles for lint be customized?

Linting is customizable by product teams. Usually, linting tools will have a config file associated with it that it automatically picks up, i.e., pylint looks for a .pytlintrc file. This is tool-dependent.

Miscellaneous

1. Best way for product teams to do notifications?

P1PS is a new offering from Party Bus that allows mission applications to send emails via Amazon SES. Please reach out to the Bullhorn team in the P1PS Help channel or submit a General Help request to P1PS . Note that this notification service is not available for collaboration apps like GitLab.

2. What AWS services are available to product teams?

We provide integration with RDS (PostgreSQL and MySQL) and S3. Other services may be supportable, but are not currently drop-in ready; a feature request will need to be submitted by the product team.

3. Where can I ask more questions?

The Party Bus Value Stream Support Channel is a good starting place for assistance. Simple, general questions can be asked there, and there are links to the various support desks and documentation. We'll also be able to direct you to the right people for Party Bus requests from there.

• For pipeline issues and requests, a help desk ticket should be opened at the Party Bus Pipeline Help Desk ; an MDO Engineer will be assigned to your ticket and work through your issue.
• If you would like to talk with other app teams, go to the App Team Guild Mattermost channel .
• Cyber related questions can be directed to the Cyber Value Stream Support channel .

4. Does P1 provide an artifact/package repository like Nexus?

We use GitLab's package registry to host artifacts and packages created by pipelines. GitLab's container registry is used to host containers created by pipelines.

5. Can I use my own Keycloak server, or can I use direct integration into login.dso.mil?

P1 does not allow or recommend the use of a per-team Keycloak instantiation. Party Bus uses the P1 Keycloak for all operations. This means that you will need to use the JWT/Authorization header to authorize users and do any sort of RBAC.

6. Do P1 and ODIN share a Keycloak instance?

No, each platform utilizes its own Keycloak instance.

7. What is defined in a CtF letter?

A CtF letter authorizes a single application and its specific services/deployments for one (or more) production environments. Not all services deployed to staging need to be deployed to production. The CtF letter will specify what ILs, the effective time period, and the major software version release in which it is valid.

8. How can we set up a working session with MDO to resolve an issue?

A help desk ticket should be opened if you have an issue that needs MDO attention. The assigned engineer will use the Jira ticket to communicate progress and relevant information. If a meeting needs to be held, it will be arranged.

9. What necessitates a new CtF?

• Adding a database
• Adding a service/pipeline
• Adding another external connection

10. What kind of access is provided for AWS S3?

We expose connection details for the S3 bucket via environment variables that are available in deployed applications. IRSA is used to broker the connection from your pods to AWS. Read more information on S3 We grant permissions to ListBucket, GetObject, PutObject, DeleteObject.

11. How can we escalate an issue if a ticket is not being addressed?

Reach out on the Party Bus Value Stream Support Channel .

12. What Kubernetes distribution is Party Bus using?

Currently, Party Bus is using RKE2 but has plans to migrate to AWS EKS.

13. What are the SLAs for support tickets?

View our Party Bus SLAs.

14. What is the difference between IL2, IL4, IL5, and beyond?

DoW ILs are determined by a combination of:

• The sensitivity of the information to be stored and/or processed in the cloud
• The potential impact of an event that results in the loss of confidentiality, integrity, or availability of that information

The ILs:

• IL2 accommodates DoW information that has been approved for public release (Low Confidentiality & Moderate Integrity)
• IL4 accommodates DoW CUI (e.g., FOUO)
• IL5 accommodates DoW CUI & NSS
• IL6 accommodates DoW Classified Information up to SECRET

Party Bus

1. How do I configure outbound traffic to an external HTTPS endpoint?

Submit a staging egress request or production egress request .

Please note in your requests if and when you are trying to reach a service within NIPR, as this will take extra time to work through the DISA egress request process. If you are unsure, check with the endpoint owner or your program office.

2. I have an external service that needs to connect to Party Bus. Is this possible?

Party Bus now has an NPE workflow that uses mTLS to broker a zero-trust connection from an external entity to within the Party Bus boundary. This does not go through the normal means of authentication, i.e., Keycloak, but instead relies on special Istio configuration. Please use the General Support request to open a ticket to get an NPE connection set up. This will require the "client" to be able to broker an mTLS connection using a certificate and key generated and provided by P1.