We would be highly delighted if there will be a Unit-Test Option for Release Templates in the future.
At the moment, we are testing our release processes/Templates by running test releases as e2e tests in a CD Pipeline before those new Template version will be rollout out to production. We do this at the earliest possible moment for Quality assurance, to provide early feedback to our release managers of ongoing template changes before they can merge their changes to be rolled out.
Having said that, there needs to be a digital.ai release runtime available at any time to run those E2E test. Startup Time is quite long to fire up a dedicated testing instance on demand. Scripts in the release templates mostly implement release APIs that require a runtime for IDs and so on at the moment.
The E2E Tests, generally take very long and are very slow in comparison to real unit-tests in continuous integration. We would strongly suggest providing a solution like exposing necessary APIs or whatever comes to your mind to just avoid the usage of the whole release-runtime server/database for testing purpose of one or many groovy or python scripts of one or multiple releases. Dependency Injection/inversion could be a way but is very awkward large implementation for simple unit test that is required to be specific to each script.
Having said that, this would be a great benefit compared to other competitors of cd/devops products.
by: Daniel R. | 3 months ago | Integrations
Comments
Thank you for the idea. We have a few follow up questions.
Scope of testing: could you clarify if your primary goal is to test the logic within your Groovy (Jython?) scripts, or are you also looking to validate the full release process, including task execution and outcomes?
API behavior: uou mentioned the need for real behavior from the APIs. Are you looking for a solution that simulates the entire environment, or would a lightweight API simulator that mimics expected responses be sufficient for your testing purposes?
Runtime: if we were to provide a more streamlined testing environment, such as a Docker-based on demand instance or a simplified runtime, would this address your concerns about the startup time? How important is it for this environment to mirror the production runtime exactly?