Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Added 9e89805d62c postmortem

...

Who present in meeting
Commit
Details
Root Cause Analysis

Actions


Takeaways

...

08/09/2020 10:35

Who present in meetingIsaac, Luna, Kevin, Jay, Rob, Andy F, Richard
Commit9e89805d62c
Details

InterruptedException in JavaDoc build.

Code Block
themeConfluence
collapsetrue
java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadGroupSynchronously(CpsStepContext.java:248) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.getThreadSynchronously(CpsStepContext.java:237) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:298) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67) at org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:264) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:247) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:180) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20) at com.forgerock.pipeline.reporting.ElasticSearchPipelineRunStorage.updateRunStage(file:/var/lib/jenkins/jobs/OpenAM-master/jobs/OpenAM-Pipeline/branches/master/builds/4756/libs/java-pipeline-libs/src/com/forgerock/pipeline/reporting/ElasticSearchPipelineRunStorage.groovy:94) at com.forgerock.pipeline.reporting.PipelineRun.updateStatus(/forgerock/jenkins/jobs/OpenAM-master/jobs/OpenAM-Pipeline/branches/master/builds/4756/libs/java-pipeline-libs/src/com/forgerock/pipeline/reporting/PipelineRun.groovy:205) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2030) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2015) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2056) at com.forgerock.pipeline.reporting.PipelineRun.updateStatus(/forgerock/jenkins/jobs/OpenAM-master/jobs/OpenAM-Pipeline/branches/master/builds/4756/libs/java-pipeline-libs/src/com/forgerock/pipeline/reporting/PipelineRun.groovy:191) at com.forgerock.pipeline.reporting.PipelineRun.updateStageStatusAsInProgress(/forgerock/jenkins/jobs/OpenAM-master/jobs/OpenAM-Pipeline/branches/master/builds/4756/libs/java-pipeline-libs/src/com/forgerock/pipeline/reporting/PipelineRun.groovy:164) at WorkflowScript.runPipeline(WorkflowScript:148) at com.forgerock.pipeline.stage.StageRunner.accept(file:/var/lib/jenkins/jobs/OpenAM-master/jobs/OpenAM-Pipeline/branches/master/builds/4756/libs/java-pipeline-libs/src/com/forgerock/pipeline/stage/StageRunner.groovy:81) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2030) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2015) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2056) at com.forgerock.pipeline.stage.StageRunner.accept(file:/var/lib/jenkins/jobs/OpenAM-master/jobs/OpenAM-Pipeline/branches/master/builds/4756/libs/java-pipeline-libs/src/com/forgerock/pipeline/stage/StageRunner.groovy:80) at com.forgerock.pipeline.stage.StageRunner.runStage(file:/var/lib/jenkins/jobs/OpenAM-master/jobs/OpenAM-Pipeline/branches/master/builds/4756/libs/java-pipeline-libs/src/com/forgerock/pipeline/stage/StageRunner.groovy:136) at ___cps.transform___(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor650.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor650.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor650.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:237) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Root Cause AnalysisAppears to be caused by JENKINS-46507.

Actions

Jenkins was restarted twice to prevent further builds from falling over. Robin suggests the root cause can only be fixed in GCP.

TakeawaysLook out for InterruptedException and flag ASAP for Jenkins restart.

...

07/09/2020 15:49

Who present in meetingIsaac, Luna, Pete, Rob
Commit311cb771cee
DetailsRepetition of the ExternalDjTest test failure seen at 047da8872ad.
Root Cause AnalysisSame as original.

Actions

We expect the original fix which is in PR to resolve this issue. If it reoccurs then we can take subsequent action.

Takeaways

07/09/2020 15:49

Who present in meetingIsaac, Luna, Pete, Rob
Commithttps://stash.forgerock.org/projects/OPENAM/repos/openam/commits/fac4299620899d8e1b4ab1f83890ffabbc55f1e0
DetailsOauth2 ImageSmokeTestFailure in RFC7523
Root Cause Analysis

5 failures were located in the image smoke test run https://qa.forgerock.com/am/master/fac4299620899d8e1b4ab1f83890ffabbc55f1e0/e167c118-a9f4-464c-868a-c654fad8b0c8/image-smoke-tests/ImageSmokeTestsReport/class-com.forgerock.openam.functionaltest.oauth2.rfc7523.JwtBearerTokenEndpoint.html


The commit that triggered the failure does not appear to contain information which should have resulted in this failure on its own:

https://stash.forgerock.org/projects/OPENAM/repos/openam/commits/fac4299620899d8e1b4ab1f83890ffabbc55f1e0


It appears the test expects for a specific realm to not exist at the start of each test. This realm name appears to be hardcoded ("alias.localtest.me"). The test then fails when it attempts to create this realm at the start of the test.

Actions

A JIRA task has been created

Jira
serverForgeRock JIRA
serverId45b4e541-fe03-30e2-b999-cf4e643b93ec
keyAME-20296
to investigate the cause of the realms not being cleanup correctly during the run.

TakeawaysIt is important to not hardcode significant values (realm names, user names, etc.) in tests. Instead, randomly generate them on each test run, so that issues caused by cleanup failing are mitigated. However, doing so may mask failures in our cleanup code or potential concurrency issues.

...