Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Who present in meeting
Commit
Details
Root Cause Analysis

  

Actions


Takeaways

...

30/09/2020 20:19

Who present in meeting
Commit7ea2b9d661f19e40e81148fc439188f41f17dedf
DetailsExceptions in the functional-tests-1 & functional-tests-3 stages
Root Cause Analysis

Both stages completed successfully, but then exceptions were thrown while trying to push data to Elasticsearch.

[gcp-gce-jenkins-agent-5] json: {"doc":{"endTime":"2020-09-30T17:43:28.392Z","status":"SUCCESS", (rest of json removed)
[Pipeline] [gcp-gce-jenkins-agent-5] echo
[Pipeline] [gcp-gce-jenkins-agent-5] echo
[gcp-gce-jenkins-agent-5] Exception occurred:
[gcp-gce-jenkins-agent-5] java.lang.InterruptedException


Upon checking the Elasticsearch cluster health, it was in a red state, with a new container starting within the last hour. The cluster health had improved to yellow during this post mortem.

Actions

Master to remain locked while elasticsearch recovers to a green state.

Takeaways

30/09/2020 16:30

Who present in meeting
Commit

Error seen on commit 00cd223e55ec1384168cbf4d147c37fec71087e1 however likely caused by an earlier commit.

Details

Stage 00cd is showing the test failures for the SocialAuthNodeTest. This stage however is not changing code that can affect this commit. Instead it is addressing an issue with the after-hook processing logic for the pipeline. This error was likely preventing stages from uploading the test results to the QA bucket.

Root Cause Analysis

The failures present on the functional-test run are a repeated flaky test which was originally captured 07/09/2020 11:00 in commit 37d4aa2551b41c45f7719202fefe2b16e84fd122.

Test reports for comparison:

There is no further action to take at this stage as this is already captured by a ticket for the DRIVE team to investigate.

Actions

Action AME-20294 has already been raised to track this issue.

Master will now be unlocked.

Takeaways


...