|Who present in meeting|
|Root Cause Analysis|
|Who present in meeting||Jay, Rob, Isaac, Pete R, Mark L, Luna, Phill C|
|Details||Failure in the functiona-tests stage test classes: PushAuthenticationSenderNode, PushResultVerifierTest|
|Root Cause Analysis|
Looks like a Miranda timeout as the mock push service is not indicating that it has received a message. The commit that triggered this does not appear to be directly related to the failure we are seeing.
PushAuthenticationSenderNode and PushResultVerifierTest both timed out while awaiting a state change to be witnessed by Miranda. The build that indicated this failure was not displayed to us in the dashboard due to failures in bitbucket. Instead, investigation focused on PRs that had been merged to master recently, and it was noted that the commit o3ceae saw similar failures in the functional-test stages when run by Jenkins.
Reverted the offending commit: https://stash.forgerock.org/projects/OPENAM/repos/openam/pull-requests/12352/overview
Master has been left locked while we await this commit to be built and verify the issue is resolved.
|Takeaways||Additional scrutiny should be applied to checking that Jenkins has approved a given commit. This is especially important in situations where build failures have been frequent and it has taken some time to get the Jenkins approval, and when multiple other individuals have approved the PR.|
|Who present in meeting||Jay, Rob, Isaac, Pete R, Mark L, Luna|
|Details||Failure in the image-smoke-tests stage test class JwtBearerTokenEndpoint|
|Root Cause Analysis||See RCA on incident report "07/09/2020 15:49"|
Raised the JIRA that was previously created and attempted mitigation back into process.
This failure has been seen three times in the last 48 hours and needs addressing.
An attempt to mitigate this issue was committed in 161065b9785 which does not appear to have worked. Further investigation may be required, a potential solution would be to replace the hard-coded 'alias.localtest.me' with '<random string>.localtest.me' though this may simply hide the underlying issue of realms not being deleted before subsequent tests are run.
|Who present in meeting||Jay, Isaac, Rob, Rich, Andy|
14 functional tests marked as failed.
Test class: TestSessionResourceV2V3AndV4
|Root Cause Analysis||Believed to be a repeating flakey test. Failure observed on master at 03/09/2020 09:30. Also observed recently on a pull request.|
Tracked under AME-20214.
|Takeaways||Given the test is flaky, the recommendation is to allow pipeline to continue.|