This is the direction I would like to go with the tests. Instead of mocking out the ResultsDB and WaiverDB services, the tests will start up a copy of the real services and set up results and waivers in them, and then ask Greenwave for a decision.
In the diff, it looks like these test cases are all new, but it's actually just the exact same cases we already had but converted to call the testdatabuilder fixture instead of using mocks.
It makes the test setup a bit more complicated, but it has a number of advantages:
Tests don't need to know all the details of how exactly ResultsDB and WaiverDB respond to a particular request
Tests don't need to know the details of what requests Greenwave will send
We avoid bugs where our mock requests do not represent how the other services actually behave (see for example #4)
If ResultsDB or WaiverDB change behaviour in a way that affects Greenwave's answers, we will find out from these tests
When Jenkins runs the functional tests we would have it set up a test environment in OpenShift (with real ResultsDB + WaiverDB + Greenwave services) and then run these same test cases. That gives even more realism since we will be running the same containers in a production-like way. Similar to what WaiverDB already has in its Jenkinsfile.
This is the direction I would like to go with the tests. Instead of mocking out the ResultsDB and WaiverDB services, the tests will start up a copy of the real services and set up results and waivers in them, and then ask Greenwave for a decision.
In the diff, it looks like these test cases are all new, but it's actually just the exact same cases we already had but converted to call the
testdatabuilderfixture instead of using mocks.It makes the test setup a bit more complicated, but it has a number of advantages:
When Jenkins runs the functional tests we would have it set up a test environment in OpenShift (with real ResultsDB + WaiverDB + Greenwave services) and then run these same test cases. That gives even more realism since we will be running the same containers in a production-like way. Similar to what WaiverDB already has in its Jenkinsfile.